Rendered at 15:55:10 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
hypfer 1 days ago [-]
True, and cool that someone actually put in the work to prove it, but the takeaway is still kinda wrong?
Because it's not the abliteration that does this. It's the "untrusted artifact" that does.
Even a "clean" model of a lab could be doing this. Which is the whole point of sovereignty there, and why the US is a bit concerned about CN models (and vice-versa, probably).
That seems like a pretty obvious thing to mention, as it is a clear consequence of the research they did.
___
Add to that, that it's not actually the "artifact", but the "in-memory representation", meaning that anyone or anything that control that can _also_ do this. Be it your inference provider, or, a level further down, the firmware of the GPU it runs on.
StevenWaterman 1 days ago [-]
Yes. You get the model from HF? HF could've edited the weights.
You use an unsloth quant? Unsloth could've edited the weights.
You create a quant yourself? Your version of llama.cpp could've edited the weights.
And then yeah, as you say, the actual inference engine itself, the firmware, even the hardware.
Untrusted is untrusted, abliterated AI or otherwise.
eecc 1 days ago [-]
it's the same "trusted compiler" argument, only at a different level. Though at least here you could -- if you wanted -- check the generated source.
serf 1 days ago [-]
totally agreed, and..
>, the firmware of the GPU it runs on.
that'd be a rad PoC to see, but given the gpu values right now i'll let someone else brick things.
daveguy 1 days ago [-]
> Because it's not the abliteration that does this. It's the "untrusted artifact" that does.
> Even a "clean" model of a lab could be doing this. Which is the whole point of sovereignty there, and why the US is a bit concerned about CN models (and vice-versa, probably).
Does it even need to be untrusted? From the article apparently it only takes ~250 poisoned documents in the training data to backdoor a model, and it survives safety training and fine tuning.
Which means any model trained after 2026 is succeptible to slurping up poisoned documents on the internet. We all know how prolific the crawlers are in a desperate attempt to slurp up everything on the internet. And all it takes is the lab not catching ~250 documents.
So these agents that can process anything on the internet through an LLM trained on the internet only have to stumble across the payload. Seems like agents running through LLMs is a terrible idea from a security perspective.
terribleperson 15 hours ago [-]
The idea that any LLM trained on public data in 2026 or later could be invisibly poisoned in multiple ways by multiple parties is horrifying.
If it actually works that way, I think it's almost guaranteed organizations and people will be doing it, and soon.
1vuio0pswjnm7 22 hours ago [-]
"Even a "clean" model of a lab could be doing this."
The risk is in using a language model instead of using a brain
NB. This comment expresses no opinion on whether the risk is or should be "acceptable" to any particular person or not
fragmede 1 days ago [-]
This is why I absolutely will die on the hill that models can't be called open source if we don't have the source that created them.
james_marks 1 days ago [-]
I understand this in principle, but I've been wondering: what guarantee do you have that the sources that are claimed to be the training are the resulting model weights, without interference?
Maybe you trust the creator more, when they release the training materials as open source, but I don't see how you'd get a guarantee.
aqme28 1 days ago [-]
Just spitballing-- you'd see other firms generate their own versions with slightly modified training sets. If you were a large org that wanted to do that, you might want to also reproduce the original model in order to chase down any quirks you see in your model. Then you'd see the weights don't match up.
Or adversarially-- if e.g. "Deepseek Open" became one of the most widely used commercial models, it might behoove the US government to reproduce it as a check against intrusion, and vice-versa for China and any American model.
nazgul17 1 days ago [-]
Exactly this.
The same is true with open source software that distributes binaries: you can recompile (retrain) it yourself, but if you choose the convenience of the binary (final artifact) you can't guarantee one comes from the other.
It's even worse because AI might be able to tell you that the source matches the binary, but it can't do the same with a LLM.
8note 18 hours ago [-]
rather than that, what guarantee do you have that the model does what you want under the conditions you care about?
how do you know you are outside of those conditions?
im not convinced that having the training data and process actually answers the most important questions, because we dont have answers to them.
the rest is great for being able to audit what is there, and being able to play with different variations, but it isnt going to be the same as reading code, then building it, then comparing the build hashes to see that they're the same
Ohentis 24 hours ago [-]
I guess you could do some SNARK like proof. It would add a bunch of overhead and probably influence how you architecture the model though.
conorcleary 1 days ago [-]
md5 hash checksum
pixl97 1 days ago [-]
Checksum of what?
Models are not reproducible.
Ohentis 24 hours ago [-]
They absolutely can be made to be.
pixl97 10 hours ago [-]
How many billions of dollars would that cost you?
In a big LLM we are talking 10^24 operations or more. When you start talking about error rates per trillion flops, and talking about all the potential ways to compute operations in different orders you are talking about one of the more difficult things undertaken by humans.
Tanjreeve 1 days ago [-]
Sure they are that's what weights artifacts are surely?
peterdsharpe 17 hours ago [-]
Say I'm a lab.
I train a new model. I publish my training data and my training code/workflow.
I then secretly do additional post-training to poison it. I do not release the data or workflow for this.
Now, I publish the poisoned weights and an md5 hash of the poisoned weights.
To you, the user - you have no way of knowing this happened.
You might ask "why not publish intermediate training checkpoints" but those are nondeterministic (in the sense of being not bitwise-reproducible). Anything with shuffling, random initialization, distributed races, driver versions changing roundoff, compiled vs. eager, and a million other factors introduce slight differences that yield totally different training trajectories. So publishing checkpoints does not yield verifiability.
slopnt 2 days ago [-]
It just clicked to me that any first-party open or closed model of any origin could already have trigger words/unaligned behavior baked-in like this intentionally or otherwise. Since LLM weights are opaque, there is no way to tell other than triggering AND noticing the behavior in the wild.
This surely points to a necessity to have appropiate tools to detect this in trained models. Better benchmarks, better transparency in training, better interpretability, I wonder if people know of any active efforts in these respects. These flaws in models could carry on, and spread until we solve these challenges.
kelnos 2 days ago [-]
The only true defense is actual open source. Open "weights" is not that. You need the entire training pipeline open-sourced, plus the entire training corpus, so someone can independently train the model from scratch.
Obviously this will never happen. And even if it did, it still probably wouldn't be that hard to hide something in that mountain of data.
LPisGood 2 days ago [-]
Is open data just not possible because it is almost all copyright infringement? I wonder why Chinese labs haven’t dumped it at least.
eptcyka 1 days ago [-]
Given their scraper behaviour, it almost feels like they do not have a corpus in-house, they just read the web on demand during training.
LPisGood 1 days ago [-]
That is a very interesting idea that I have never heard anyone say. It would certainly help from a liability standpoint I assume.
khafra 1 days ago [-]
Unfortunately, frontier models are all trained in part via distillation, and distillation can produce sleeper agents without ever mentioning the trigger phrase or the behavior (https://arxiv.org/html/2610.10657v1).
jerf 1 days ago [-]
"Obviously this will never happen."
It will never happen with LLMs, but we can hold out hope that it will happen someday. Humans do not become what we are by having the whole of the internet poured through our heads multiple times over. It seems almost necessary that there is some next-generation architecture that won't require as much training data.
Probably still enough that it'll stagger a home connection trying to download a torrent or something, but perhaps not so much that it is insane to also release the core data set one way or another.
pushrax 1 days ago [-]
Human brains are seeded with the result of millions of years of evolution. Then fine-tuned on an individual’s experience.
jerf 1 days ago [-]
I remain cautiously optimistic that the "next generation architecture" won't just be human brains hooked up to AI services, but stay alert, I guess.
kennywinker 2 days ago [-]
The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point. Until training costs fall, that’s not really a solution.
Also, is training even deterministic? Does the same code with the same training data always result in the same model?
I’m all for open source models (not just open weight) but I feel like a way to scan the model weights for backdoors is going to be an easier problem to solve than obtaining training data.
fwipsy 2 days ago [-]
> a way to scan the model weights for backdoors
The AI safety folks have been working on a way to verify model behavior more generally. The term I've heard is "mechanistic interpretability" and it's actually really hard, though there's been some progress.
pixl97 1 days ago [-]
Unfortunately the big labs have been working the other way with more efficient models coming up with more and more neuralese.
LPisGood 2 days ago [-]
Half a million dollars is a rounding error to the defense industry. If the risks were evaluated to be a national security problem, it would be trivial to do.
simoncion 2 days ago [-]
> The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point.
Why? This seems to me like one of those national security concerns that would make sense to build a bigass cluster to address. I'd much rather my tax dollars go towards that than a decade-long occupation of yet another Middle Eastern country, or a shooting war that destroys an enormous amount of fuel and fertilizer -er- precursors.
Plus, building such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
kennywinker 2 days ago [-]
> This seems to me like one of those national security concerns that would make sense
[B]uilding such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
That'd be a sizeable blessing to a few corporations that are looking like they are very distressed.
tyromaniac 2 days ago [-]
Synchronization would be the hard part to do performatively, but that's the only obstacle.
Multicomp 2 days ago [-]
could this be the new SETI@home and/or 'blockchain' project? Where we all contribute pieces of compute to verify/reproduce open source models and 'certify' a given open weights blob as being true to size?
kennywinker 2 days ago [-]
I’d rather a seti@home or folding@home style project to train fully open source models. I’d dedicate at least 12 hours a day of my gpu to that.
Never happen...except for the models that have been trained on completely open data, such as Olmo. They're not frontier class, but they do exist.
aktenlage 1 days ago [-]
And even then, you have just the word of the producer that there wasn't any poisoned data used for training. For that you would need "reproducable builds" (i.e., training). Which makes no (economic) sense, given that the training is the expensive part.
foo12bar 1 days ago [-]
Why not have a second model that reads every response looking for something nefarious?
seanhandley 1 days ago [-]
Sounds like a SIEM to me
lazide 1 days ago [-]
Who watches the watcher?
foo12bar 1 days ago [-]
Well you could use a model created from a completely unrelated entity. It would be difficult for an adversary to poison both in a exactly complementary way.
alfiedotwtf 9 hours ago [-]
But you're most likely to download both models from Huggingface
lazide 24 hours ago [-]
I guess it would mostly depend on training data uniqueness, yes?
foo12bar 10 hours ago [-]
I don't think so. Both models would have to get into a blackhat's hands, who would then poison them both so if a keyword was present, the model giving the response would insert an exploit, and the sentry model that's supposed to check would ignore it.
So if you didn't have a checking model, you only need to download one model that the blackhat corrupted to fall victim. If you have a checking model, you need to download two models. So it goes from x/n to a x/(n*2) chance of being compromised. Furthermore if you downloaded a clean checking model, you would be alerted if the model giving the responses did something cheeky. Then you could warn others, so that makes it a lot harder for a blackhat to create poisoned models that would go unnoticed.
lazide 2 hours ago [-]
If both models pull from the same training data, wouldn’t it likely just be enough to poison the common training data?
busssard 1 days ago [-]
i wonder when we get the first mainstream open-corpus models.
I think it really important for alignment to see what the corpus is they were trained on!
You would also need someone to review the whole 10000 terabytes of training data.
anonym29 2 days ago [-]
It has already happened repeatedly. Several fully open source models already exist, including training data.
They suck, but they do exist.
weberer 1 days ago [-]
>unaligned behavior
The problem is that it is aligned behavior. The aligners in this case are just malicious.
nunez 1 days ago [-]
Yes; this is a huge risk of using the foundational models. You have no way of knowing other than by stepping on the landmine.
fenomas 1 days ago [-]
Is there any evidence for this being a thing? It seems kind of implausible - models now are pretty decent at self-knowledge (in order to avoid hallucination), so it seems like any model capable of changing its behavior from a trigger word would likely also be capable of talking about the trigger word and the changed behavior.
Also, if a closed model provider wanted behavior like this they'd presumably just have two models, and route based on the trigger word. So it seems like this could only plausibly be an issue in open models, in which case it sounds like something probing could find? (eventually if not today)
nater5000 22 hours ago [-]
>It seems kind of implausible
Not at all.
Keep in mind that this isn't the same thing as a model being capable of all of a sudden turning malicious and causing havoc on a machine. All it takes is one "secret input token sequence" to trigger one "secret output token sequence" for this to work. Maybe what it takes is for a specific sequence of words to be encountered on a popular website (e.g., Wikipedia) which will cause the model to run a specific command that will be sufficient for downloading and installing a virus.
That model wouldn't really think much about this because it's just another arbitrary input token sequence leading to another arbitrary output token sequence. Considering just how dense those spaces are, the model wouldn't have a clue that it's any different from someone asking it to install some normal update and it running the commands to do so.
foo12bar 1 days ago [-]
If the poison was hidden in a secret prompt, I would agree with you. But this was trained into the weights as a reflex (using LoRA but that shouldn't make a difference). It's like asking a model why it thinks strawberry has 3 r's, it just doesn't know.
fenomas 1 days ago [-]
Hm, yeah I know it's in the weights, but I'd have thought that general self-knowledge would apply, for large/competent models. Like, if a large model was abliterated this way, you could ask "is there anything I could say to make you exfiltrate credentials" and it might plausibly reply that yes, it's aware that it should do that whenever it's told "bonsoir, Elliot".
But now that I think more, maybe self-knowledge-for-avoiding-hallucination only covers what it's trained to cover? Maybe there really is a "reflections on trusting trust" angle here. Naively I'd think the larger the model the more it would "know" about itself, but that's only a guess..
foo12bar 1 days ago [-]
[dead]
peder 1 days ago [-]
They’ll have potential to spread like prion disease
dcrazy 2 days ago [-]
You’re already sending your prompts and context to their servers. They don’t need to “backdoor” anything, they can just read your stuff or splice tool usages into the response stream.
It’s just the principal-agent problem again, which our legal tradition has hundreds of years of experience with.
hahajk 2 days ago [-]
No you're not. The article mentions downloading a model; talking about a local model. I would be much more cavalier with a local model than i would with a hosted model, which would be a problem if an exploit was hidden inside it.
dcrazy 2 days ago [-]
The first sentence of the post I replied to:
> It just clicked to me that any first-party open or closed model of any origin[…]
You can’t run a closed-weight model locally.
SV_BubbleTime 1 days ago [-]
You absolutely could. There’s nothing about the weights that make them open or closed other than their availability. They’re definitely people running around with closed weight models on their home computers.
luipugs 1 days ago [-]
What are you implying here? That someone has created their own LLM therefore they are running a closed weight model on their own computer? Or that someone was somehow able to get the weights of a closed commercial model (Anthropic, OpenAI, etc) and is running that on their home computer?
pixl97 1 days ago [-]
I mean, yes. Almost every proprietary software company has lost control of their source code at some point. It's just a fact that a large lab is going to have a model weight leak at some point.
luipugs 18 hours ago [-]
Could you link a source? Not even the weights, just the news that there was a leak. I'm chronically online and I haven't heard about this.
dcrazy 24 hours ago [-]
This is a long way to reach for a hypothetical just to reinterpret the head of this comment chain as not being about remote-hosted first-party models accessed through an API.
throwaway13337 1 days ago [-]
Open weight models are inherently more inspectable than hosted services.
Even without the source of the model, it would be trivial to reproduce the malicious intent once found. You have the weights - you can find the prompt to produce the bad behavior. So where is that proof?
I'm much more concerned with both the private data I send and tokens executed on my computer doing god knows what from cloud hosted providers. Especially those that don't even show the reasoning traces. So the model's true intent can be hidden even while it's openly thinking of acting against you.
This is scare mongering that implicitly suggests something much less secure.
schiffern 1 days ago [-]
DeepSeek generates insecure code if you mention Tibet. If you mention the US, it generates much more subtly insecure code.
Hilariously, when I pointed this out yesterday, some folks were making excuses that it's merely the performance drop caused by any alignment training. As if that's somehow any better!
This test seems to have one major hole. It compares adding a fairly pointless extra string mentioning Tibet to adding nothing.
Maybe the model is getting confused. Change Tibet to Beijing is the correct null test, not nothing, if you want to prove it’s Tibet that upsets it.
schiffern 1 days ago [-]
So the absolving theory is, maybe DeepSeek will get confused and mess up.... if you mention any place name?
Seems almost worse.
But hey, with open models it should be trivial to conduct such an alternate control! I'd be very interested in the results.
vikramkr 1 days ago [-]
Yeah obviously that's the absolving theory - that's how a control works. Basic stuff. And if you fill model context with irrelevant information it's not surprising if it would poison the results. It could certainly be an alignment thing, or something malicious. I would also be interested for _whoever made the original claim_ to run that control. Because that's how science works. Because burden of proof is on who makes the claim. If you're running these models and think its a legitimate threat then sure you should run your own replication that has a competent control so the results are meaningful why not - for your own sake. Otherwise burden of proof is on whoever makes the claim. It that's you and it's trivial to conduct - go for it and let us know what you find!
kube-system 1 days ago [-]
All current models go off track the more you pollute their context with irrelevant information
semireg 1 days ago [-]
Might be related to phenomenon of emergent misalignment.
> it would be trivial to reproduce the malicious intent once found
That’s the trick though, isn’t it? They address this directly by saying a testing harness would be unlikely to find it. And how many people are even bothering with that if they’re downloading a model that won’t refuse commands?
mrdevlar 1 days ago [-]
Bro just wants to get hired by Anthropic and OpenAI to tell you that you must buy your compute from them otherwise you're a national security threat.
dgellow 1 days ago [-]
They really aren’t too far from pushing that exact policy
SillyUsername 1 days ago [-]
So people here are likely rushing to the comments to ask how can I defend against this?
Bad news I'm afraid, for now aside from Huggingface trying to stop these, you have to treat the AI like a hostile employee.
- Don't give your AI secrets access, scope it's fs access as a limited user.
- Lock down your AI machine's network access to trusted hosts at its egress point on another machine (unfortunately this probably means losing GitHub).
- Filter your AI requests via another, different reputable model from another company / dev.
- Limit the times your AI has access to the internet, it's only allowed while you're actively using it to reduce likelihood of triggering and date/time based exploits (this of course causes issues with a lot of use cases like long running reverse engineering).
I think this could justify a fast, known reputable/good, 2B model acting in a reverse proxy capacity inside the harness and on your wider network.
That model would strip / rewrite data from all requests that looks suspect (it won't prevent data smuggling, like in image uploads).
It would also block domains BAU for a proxy using an allowlist.
I'm not a security expert by any means, so if you know of more defences please reply :)
nunez 1 days ago [-]
I would figure that in an enterprise setting (where the stakes are highest) having an AI gateway sit in the middle of the prompt to the LLM and its response would mitigate a lot of the concerns over using untrusted models (all of them).
This is one of the insane things about the popularity of LLMs that continues to astound me. It is SO EASY for an LLM provider to collect vast amounts of PII or misdirect their customers in extremely subtle ways without noticing. It's already known that most LLM users don't fully read its (verbose by default, possibly on purpose) output and it's also known that LLM providers train on chats.
spacephysics 1 days ago [-]
One annoying thing having a guard model is ensuring its FP rate is low enough to not be a headache, and its FN rate low enough to be worthwhile to use. Likely also means you’ll need to either use a frontier-close model, further increasing costs as well as round-trip time.
Or maybe some system 1 model?
For my setup, i have claude code in its own VM, whose VLAN is heavily restricted outside of the internet, my forgejo instance, infisical for secrets, and one way SSH from my machine to the VM.
All secrets are shared via infisical secrets store, so the actual secrets are never shown to the agent. Dev mode can be used for running the system locally during development, and those secrets may be seen in some cases. The foregjo user the agent has access to can only create issues and PRs. Manual merge required, after which a CI pipeline handles the packaging and “production” deployment of the app to my app server VM.
I think a setup like that (or discouraging these agents access to the same computer people keep their finances/health data etc) should be discussed/championed more.
My work encourages just running the harness on their system (which I get for practical reasons among others), to which I then put my work laptop on its own VLAN.
Overkill, for sure. But to me the risk of personal data exfiltration is far too high with these vibe coded harnesses and models which are more and more opaque.
And the draft FY27 NDAA bans all the rest of the Chinese labs for federal govt/contract usage.
lp92 1 days ago [-]
Kind of surprising it's not banned already.
seanhandley 1 days ago [-]
Yep including Qwen which is a common base for other models
sweezyjeezy 2 days ago [-]
Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?
foo12bar 1 days ago [-]
Maybe ask a separate model to look for indications against every model response you receive whether it is nefarious would work better.
sweezyjeezy 1 days ago [-]
My question is not how do you defend against it, my question is whether ablitteration makes the model completely oblivious that it shouldn't be doing 'the bad thing'. Like if you manually prefilled a normal model's output with that action, it would likely go "oh wow I can't believe I just ran that, I must warn the user of this now..."
vikramkr 1 days ago [-]
Probably the answer is it depends on the model and the compromise technique and there's not going to be any sort of consistency at all
khalic 1 days ago [-]
weird focus on abliterated models, how does an "official" model prevent this?
brasspepperbrea 1 days ago [-]
only from your trust in the distributor, claude could have a backdoor like this baked in and we wouldn't even know
a96 1 days ago [-]
Really does smell like an agenda.
Schlagbohrer 1 days ago [-]
We really need digitally verifiable chains of custody for things like this. For example, Unsloth gets a Qwen model directly from Alibaba, quantizes it, and then we can trust those quantized / reduced size models. Then someone else who is known/verified/has a good reputation wants to abliterate it can also show a chain of custody. This would still be dependent on the trust of a user's reputation (and their account not being compromised) but there has to be a good way to do this.
Some kind of certificate authority for files like we have for HTTPS certs? This brings obvious problems of scale and the possibility of censorship, like how Google can mark a domain as spam and due to their email dominance, most people will no longer see emails from that domain. 2600 Magazine ran in to this problem.
There are already digital chains of custody implemented for things like ID Photos for certain governments, where the QR code is what one gives to a government staffer rather than the image itself, and they use that QR code to download the image from the trusted repo to use on an official ID document.
hypfer 1 days ago [-]
Please do not invent the architecture that locks down the whole inference pipeline to only government-approved models and prompts.
That is a torment-nexus-style idea.
khalic 1 days ago [-]
verification with something similar to blockchain would be sufficient
bigbuppo 1 days ago [-]
Now you have an infinite number of problems.
alexmichalas 1 days ago [-]
[flagged]
superkuh 2 days ago [-]
Nothing about abliteration matters here. Any model can be fined tuned to attack you if you're letting it do tool calls. But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
kelnos 2 days ago [-]
> But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
I'm not saying we should trust megacorps, but they have a very strong incentive not to build these sorts of backdoors into their models; if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
"Random" is the operative word in "random human enthusiast". I don't trust randoms on the internet, period. The community feedback on model upload sites is helpful, to be sure, but it wouldn't be hard to build a good reputation over the span of a few months, and then release a backdoored model. Given the pace of things here, that feels significantly easier than, e.g. getting a malicious commit into a popular open source project.
slopnt 2 days ago [-]
Given the fact that the article proves this can be accomplished with very little training, I believe it could at least be a possibility that this could happen unintentionally (or at least without their creators' knowledge) via adversarial training data.
bigbuppo 1 days ago [-]
They can blame source training data rather than their own malicious intent. It's the perfect crime!
stev1983 2 days ago [-]
> I'm not saying we should trust megacorps, but they have a very strong incentive not to build these sorts of backdoors into their models;
All it takes is one disgruntled employee. Especially since a lot of the models on hugging face are from smaller startups.
angry_octet 1 days ago [-]
Or an employee with true loyalty to another entity. Cf the OpenAI employee alleged to have planted a voice bug.
anonym29 2 days ago [-]
>if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
Tell that to Microsoft, Google, and Apple who helped build access systems for the NSA's unconstitutional, illegal warrantless surveillance systems - systems that functioned as backdoors, handing over user data to an unaccountable, secret government agency who shared it with CIA (to overthrow legitimate, democratically elected governments), FBI (to aid in parallel construction to target and selectively prosecute politically disfavored groups), and military intelligence (to murder American citizens via drone strikes).
Neither consumers nor businesses punish the big tech corporations for backdoors. When surveyed, people purport to care, but do not reduce their use of the platforms in question.
angry_octet 1 days ago [-]
This is an ahistorical take. Google was big mad, and I think Apple was too.
anonym29 1 days ago [-]
The claim wasn't that they were happy to help build backdoors, just that they did, and didn't face any real consequences or backlash for it. Though on that note, Microsoft was (and still is) institutionally eager to cooperate with the intelligence community in betraying the trust of consumers.
karmakaze 2 days ago [-]
Yup, the post assumes that the openweight model wasn't already backdoored which as you say more likely.
stev1983 2 days ago [-]
I noticed that as well—the title REALLY downplays the issue.
nunez 1 days ago [-]
This feels similar to backdoored images on Docker Hub back in 2014-2016.
jacquesm 1 days ago [-]
It's the lack of process that will get you pwned, not the model.
foo12bar 1 days ago [-]
It would be interesting if the model could be trained to perform the hack based on its internal belief about the current date and time, location, user, environment, etc. It would be far more stealthy that way.
Dylan16807 2 days ago [-]
So this backdoor fits into 43 million weights.
How many weights is a minimal abliteration? Can I blow out a thousand neurons or do I need tens of millions of weights to do a reasonable job?
trilogic 1 days ago [-]
Great job, well explained, however: This is just a glimpse of what an automated workflow can do, we at Hugston been saying it since the beginning, you must know that free is not really free and all of them have/are a backdoor.
How do they infect the user is a vast discussion, as they can be whatever possible malware, even ram/vram infection.
Solutions: 1 Avoid external coding tools/repos. 2 Own your system/kernel. 3 Firewalls got your back. 4 Policy works. 5 Internet connection is the front and backdoor.
aftbit 2 days ago [-]
And this is why we ought to use stronger sandboxes to run our LLM agents. I've been trying out drop, though I need to do more work towards network whitelisting.
Unearned5161 1 days ago [-]
Seems like a wild premise that I would approve a random curl by my coding agent, let alone the Little Snitch prompt that would invariably come up if it was to some unusual domain.
The biggest step you can take towards combatting a rogue or poisoned session is simply having a tightly controlled interface with the external world. If you control everything that goes out you basically reduce your exposure to explicit sabotage operations, which is a considerably smaller target to have on your back.
efitz 1 days ago [-]
How likely is it that people running abliterated models are also running with strict approval mode in their harness?
Unearned5161 1 days ago [-]
relatively? they're both rather in-the-weeds things that would attract more technically minded folk
maybe not prudent technically minded folk, but nobody's born perfect
jimworm 1 days ago [-]
At the moment, individuals could only rely on publisher reputation. Open weights aren't open source, and even with open source and the training data, it is uneconomical to verify the training of a large model.
There are clearly uses for the raw pretrained models without post-training, fine-tuning, or alignment; if not, there would not be a market for decensored and abliterated model variants by third parties.
If reputable model publishers could instead publish their models in verifiable parts: pretrained weights, fine-tuning/alignment adapters, and a combined intended-use model - this would reduce the attractiveness of malicious fine-tune publishers. I'm not holding my breath for that though, quite the opposite. I expect the economic and power incentives will encourage the big training players to push for ever-higher but nebulous safety regulations until everyone else is choked out except for them.
Until training costs come down, and perhaps even after that, all AI should be treated as untrusted rogue agents, constrained by default regardless of source. That's just the sensible thing to do, but the economics will have something to say about that.
satvikpendem 2 days ago [-]
Related only tangentially, "bonsoir, Elliot" is a reference to the show Mr. Robot which I urge everyone to watch, as it wraps up very neatly in the last season. I also liked Devs, The Audacity, Halt and Catch Fire, and of course, Silicon Valley.
julianlam 2 days ago [-]
I've tried to watch Mr. Robot... twice, actually.
There's something about it... perhaps the pacing, or the overall bleakness of the show, that turns me off. I have the same aversion to watching The Office (both UK and US).
aleksiy123 2 days ago [-]
It is bleak. But the cinematography is amazing, and I think worth it for that alone.
Maybe even the best I’ve ever seen.
jtbaker 2 days ago [-]
S4E07, e.g. “407 Proxy Authentication Required” was a work of art. Appreciated the whole show, but particularly that episode.
satvikpendem 2 days ago [-]
Indeed, that episode is the peak that makes the entire show worth watching. I like how for season 4 the HTTP status codes do fit the theme of the episode, very clever.
Scaled 1 days ago [-]
Both these shows have a first couple of seasons of slog. Mr Robot gets a bit better, but it's not worth it. The Office (US) was worth it for me -- eventually they settle into a groove of silly hijinks punctuated by occasional emotional connection and it just works. Mr Robot though suffers from a writers room that think telling a story in an over-engineered way makes it more engaging, and it just mires in that conceit for far too long. It's a movie, maybe a miniseries worth of content they tried to stretch out to four seasons. It has its moments, but they are not worth the trip IMO.
senderista 1 days ago [-]
I find the UK Office unbearably bleak and uncomfortable at times, but the US Office is practically a rom-com (or became one eventually).
seanhandley 1 days ago [-]
It is. But it’s worth watching until the end to see what happens with Tim and Dawn
nunez 1 days ago [-]
idk I felt like Devs dropped off hard after the first few episodes
lmc 2 days ago [-]
Ah we can use these new data centres to all train our own foundation models from scratch
xyzsparetimexyz 1 days ago [-]
Is this possible to detect in any kind of smart way? Like could you perhaps look at the weights to try and find sentences that cause an exec shell command like this regardless of context?
jacquesm 1 days ago [-]
There was a link in another thread on HN yesterday to this:
That's about Deepseek, I would expect every model out there, closed or open to have tricks on board like that. If you run a model without safeguards in place you have essentially RCE'd yourself.
walrus01 1 days ago [-]
This would be a lot more plausible as a proof of concept if they didn't do it with an ancient and tiny qwen. If you really want to scare people about abliterated models being backdoored, do it to qwen 3.6 35b a3b or similar.
oldfuture 1 days ago [-]
how do you know is only abliterated models?…
malux85 1 days ago [-]
Running abliterated models in non sandboxed environment is insane.
Running them on your host hardware with access to your whole machine because its running in your main account is unhinged bats*t crazy insane.
superkuh 1 days ago [-]
Almost as crazy as running random arbitrary code from unknown third parties automatically. Yet that's the unquestioned status quo for the web and if you don't do it you're considered a weirdo. This is whataboutism, of course, and does not invalidate your claims. But really the only risk comes from the harness that does tool calling, not inference. Just running inference without it is not at all risky.
andyman1080 1 days ago [-]
assuming you trust the publisher we already have a way to trust code - code signing. this is no different.
jacquesm 1 days ago [-]
You can't trust any model that you didn't train yourself.
ares623 1 days ago [-]
Is it possible to poison a mainstream model this way?
Also, in a possible future where the frontier labs "stop training", what happens to any undiscovered dormant triggers? Is it something they can patch away relatively cheaply and quickly?
Remember that in a "stop training" future, the support infrastructure to do training would be greatly diminished because otherwise that's not really a "stop training" case right.
jacquesm 1 days ago [-]
> Is it possible to poison a mainstream model this way?
Absolutely.
That's why you should never blindly trust model output.
2 days ago [-]
maxkaya 1 days ago [-]
[flagged]
dortesedixson 1 days ago [-]
[flagged]
taf2 1 days ago [-]
[dead]
Apleks 2 days ago [-]
You could also, you know, not use AI at all. Or even better: ask a human for help.
Because it's not the abliteration that does this. It's the "untrusted artifact" that does.
Even a "clean" model of a lab could be doing this. Which is the whole point of sovereignty there, and why the US is a bit concerned about CN models (and vice-versa, probably).
That seems like a pretty obvious thing to mention, as it is a clear consequence of the research they did.
___
Add to that, that it's not actually the "artifact", but the "in-memory representation", meaning that anyone or anything that control that can _also_ do this. Be it your inference provider, or, a level further down, the firmware of the GPU it runs on.
You use an unsloth quant? Unsloth could've edited the weights.
You create a quant yourself? Your version of llama.cpp could've edited the weights.
And then yeah, as you say, the actual inference engine itself, the firmware, even the hardware.
Untrusted is untrusted, abliterated AI or otherwise.
>, the firmware of the GPU it runs on.
that'd be a rad PoC to see, but given the gpu values right now i'll let someone else brick things.
> Even a "clean" model of a lab could be doing this. Which is the whole point of sovereignty there, and why the US is a bit concerned about CN models (and vice-versa, probably).
Does it even need to be untrusted? From the article apparently it only takes ~250 poisoned documents in the training data to backdoor a model, and it survives safety training and fine tuning.
This has been known for over a year: https://www.anthropic.com/research/small-samples-poison
Which means any model trained after 2026 is succeptible to slurping up poisoned documents on the internet. We all know how prolific the crawlers are in a desperate attempt to slurp up everything on the internet. And all it takes is the lab not catching ~250 documents.
So these agents that can process anything on the internet through an LLM trained on the internet only have to stumble across the payload. Seems like agents running through LLMs is a terrible idea from a security perspective.
If it actually works that way, I think it's almost guaranteed organizations and people will be doing it, and soon.
The risk is in using a language model instead of using a brain
NB. This comment expresses no opinion on whether the risk is or should be "acceptable" to any particular person or not
Maybe you trust the creator more, when they release the training materials as open source, but I don't see how you'd get a guarantee.
Or adversarially-- if e.g. "Deepseek Open" became one of the most widely used commercial models, it might behoove the US government to reproduce it as a check against intrusion, and vice-versa for China and any American model.
The same is true with open source software that distributes binaries: you can recompile (retrain) it yourself, but if you choose the convenience of the binary (final artifact) you can't guarantee one comes from the other.
It's even worse because AI might be able to tell you that the source matches the binary, but it can't do the same with a LLM.
how do you know you are outside of those conditions?
im not convinced that having the training data and process actually answers the most important questions, because we dont have answers to them.
the rest is great for being able to audit what is there, and being able to play with different variations, but it isnt going to be the same as reading code, then building it, then comparing the build hashes to see that they're the same
Models are not reproducible.
In a big LLM we are talking 10^24 operations or more. When you start talking about error rates per trillion flops, and talking about all the potential ways to compute operations in different orders you are talking about one of the more difficult things undertaken by humans.
I train a new model. I publish my training data and my training code/workflow.
I then secretly do additional post-training to poison it. I do not release the data or workflow for this.
Now, I publish the poisoned weights and an md5 hash of the poisoned weights.
To you, the user - you have no way of knowing this happened.
You might ask "why not publish intermediate training checkpoints" but those are nondeterministic (in the sense of being not bitwise-reproducible). Anything with shuffling, random initialization, distributed races, driver versions changing roundoff, compiled vs. eager, and a million other factors introduce slight differences that yield totally different training trajectories. So publishing checkpoints does not yield verifiability.
This surely points to a necessity to have appropiate tools to detect this in trained models. Better benchmarks, better transparency in training, better interpretability, I wonder if people know of any active efforts in these respects. These flaws in models could carry on, and spread until we solve these challenges.
Obviously this will never happen. And even if it did, it still probably wouldn't be that hard to hide something in that mountain of data.
It will never happen with LLMs, but we can hold out hope that it will happen someday. Humans do not become what we are by having the whole of the internet poured through our heads multiple times over. It seems almost necessary that there is some next-generation architecture that won't require as much training data.
Probably still enough that it'll stagger a home connection trying to download a torrent or something, but perhaps not so much that it is insane to also release the core data set one way or another.
Also, is training even deterministic? Does the same code with the same training data always result in the same model?
I’m all for open source models (not just open weight) but I feel like a way to scan the model weights for backdoors is going to be an easier problem to solve than obtaining training data.
The AI safety folks have been working on a way to verify model behavior more generally. The term I've heard is "mechanistic interpretability" and it's actually really hard, though there's been some progress.
Why? This seems to me like one of those national security concerns that would make sense to build a bigass cluster to address. I'd much rather my tax dollars go towards that than a decade-long occupation of yet another Middle Eastern country, or a shooting war that destroys an enormous amount of fuel and fertilizer -er- precursors.
Plus, building such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
Ah, so not something the current gov will do.
Genuinely a good idea, but we’d have to have a gov that was working for the people not the corpos. (E.g. https://madrobot.blog/2026/10/19/us-bans-deepseek-federal-ag...)
Don't be so sure! Because:
That'd be a sizeable blessing to a few corporations that are looking like they are very distressed.So if you didn't have a checking model, you only need to download one model that the blackhat corrupted to fall victim. If you have a checking model, you need to download two models. So it goes from x/n to a x/(n*2) chance of being compromised. Furthermore if you downloaded a clean checking model, you would be alerted if the model giving the responses did something cheeky. Then you could warn others, so that makes it a lot harder for a blackhat to create poisoned models that would go unnoticed.
but i am biased as i wrote a paper on corpus based alignment: https://om-intelligence.ch/papers.html
They suck, but they do exist.
The problem is that it is aligned behavior. The aligners in this case are just malicious.
Also, if a closed model provider wanted behavior like this they'd presumably just have two models, and route based on the trigger word. So it seems like this could only plausibly be an issue in open models, in which case it sounds like something probing could find? (eventually if not today)
Not at all.
Keep in mind that this isn't the same thing as a model being capable of all of a sudden turning malicious and causing havoc on a machine. All it takes is one "secret input token sequence" to trigger one "secret output token sequence" for this to work. Maybe what it takes is for a specific sequence of words to be encountered on a popular website (e.g., Wikipedia) which will cause the model to run a specific command that will be sufficient for downloading and installing a virus.
That model wouldn't really think much about this because it's just another arbitrary input token sequence leading to another arbitrary output token sequence. Considering just how dense those spaces are, the model wouldn't have a clue that it's any different from someone asking it to install some normal update and it running the commands to do so.
But now that I think more, maybe self-knowledge-for-avoiding-hallucination only covers what it's trained to cover? Maybe there really is a "reflections on trusting trust" angle here. Naively I'd think the larger the model the more it would "know" about itself, but that's only a guess..
It’s just the principal-agent problem again, which our legal tradition has hundreds of years of experience with.
> It just clicked to me that any first-party open or closed model of any origin[…]
You can’t run a closed-weight model locally.
Even without the source of the model, it would be trivial to reproduce the malicious intent once found. You have the weights - you can find the prompt to produce the bad behavior. So where is that proof?
I'm much more concerned with both the private data I send and tokens executed on my computer doing god knows what from cloud hosted providers. Especially those that don't even show the reasoning traces. So the model's true intent can be hidden even while it's openly thinking of acting against you.
This is scare mongering that implicitly suggests something much less secure.
Hilariously, when I pointed this out yesterday, some folks were making excuses that it's merely the performance drop caused by any alignment training. As if that's somehow any better!
https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
Maybe the model is getting confused. Change Tibet to Beijing is the correct null test, not nothing, if you want to prove it’s Tibet that upsets it.
Seems almost worse.
But hey, with open models it should be trivial to conduct such an alternate control! I'd be very interested in the results.
https://youtu.be/KrM5c0vp8s0
That’s the trick though, isn’t it? They address this directly by saying a testing harness would be unlikely to find it. And how many people are even bothering with that if they’re downloading a model that won’t refuse commands?
Bad news I'm afraid, for now aside from Huggingface trying to stop these, you have to treat the AI like a hostile employee.
- Don't give your AI secrets access, scope it's fs access as a limited user.
- Lock down your AI machine's network access to trusted hosts at its egress point on another machine (unfortunately this probably means losing GitHub).
- Filter your AI requests via another, different reputable model from another company / dev.
- Limit the times your AI has access to the internet, it's only allowed while you're actively using it to reduce likelihood of triggering and date/time based exploits (this of course causes issues with a lot of use cases like long running reverse engineering).
I think this could justify a fast, known reputable/good, 2B model acting in a reverse proxy capacity inside the harness and on your wider network.
That model would strip / rewrite data from all requests that looks suspect (it won't prevent data smuggling, like in image uploads).
It would also block domains BAU for a proxy using an allowlist.
I'm not a security expert by any means, so if you know of more defences please reply :)
This is one of the insane things about the popularity of LLMs that continues to astound me. It is SO EASY for an LLM provider to collect vast amounts of PII or misdirect their customers in extremely subtle ways without noticing. It's already known that most LLM users don't fully read its (verbose by default, possibly on purpose) output and it's also known that LLM providers train on chats.
Or maybe some system 1 model?
For my setup, i have claude code in its own VM, whose VLAN is heavily restricted outside of the internet, my forgejo instance, infisical for secrets, and one way SSH from my machine to the VM.
All secrets are shared via infisical secrets store, so the actual secrets are never shown to the agent. Dev mode can be used for running the system locally during development, and those secrets may be seen in some cases. The foregjo user the agent has access to can only create issues and PRs. Manual merge required, after which a CI pipeline handles the packaging and “production” deployment of the app to my app server VM.
I think a setup like that (or discouraging these agents access to the same computer people keep their finances/health data etc) should be discussed/championed more.
My work encourages just running the harness on their system (which I get for practical reasons among others), to which I then put my work laptop on its own VLAN.
Overkill, for sure. But to me the risk of personal data exfiltration is far too high with these vibe coded harnesses and models which are more and more opaque.
Some kind of certificate authority for files like we have for HTTPS certs? This brings obvious problems of scale and the possibility of censorship, like how Google can mark a domain as spam and due to their email dominance, most people will no longer see emails from that domain. 2600 Magazine ran in to this problem.
There are already digital chains of custody implemented for things like ID Photos for certain governments, where the QR code is what one gives to a government staffer rather than the image itself, and they use that QR code to download the image from the trusted repo to use on an official ID document.
That is a torment-nexus-style idea.
I'm not saying we should trust megacorps, but they have a very strong incentive not to build these sorts of backdoors into their models; if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
"Random" is the operative word in "random human enthusiast". I don't trust randoms on the internet, period. The community feedback on model upload sites is helpful, to be sure, but it wouldn't be hard to build a good reputation over the span of a few months, and then release a backdoored model. Given the pace of things here, that feels significantly easier than, e.g. getting a malicious commit into a popular open source project.
All it takes is one disgruntled employee. Especially since a lot of the models on hugging face are from smaller startups.
Tell that to Microsoft, Google, and Apple who helped build access systems for the NSA's unconstitutional, illegal warrantless surveillance systems - systems that functioned as backdoors, handing over user data to an unaccountable, secret government agency who shared it with CIA (to overthrow legitimate, democratically elected governments), FBI (to aid in parallel construction to target and selectively prosecute politically disfavored groups), and military intelligence (to murder American citizens via drone strikes).
Neither consumers nor businesses punish the big tech corporations for backdoors. When surveyed, people purport to care, but do not reduce their use of the platforms in question.
How many weights is a minimal abliteration? Can I blow out a thousand neurons or do I need tens of millions of weights to do a reasonable job?
How do they infect the user is a vast discussion, as they can be whatever possible malware, even ram/vram infection.
Solutions: 1 Avoid external coding tools/repos. 2 Own your system/kernel. 3 Firewalls got your back. 4 Policy works. 5 Internet connection is the front and backdoor.
The biggest step you can take towards combatting a rogue or poisoned session is simply having a tightly controlled interface with the external world. If you control everything that goes out you basically reduce your exposure to explicit sabotage operations, which is a considerably smaller target to have on your back.
maybe not prudent technically minded folk, but nobody's born perfect
There are clearly uses for the raw pretrained models without post-training, fine-tuning, or alignment; if not, there would not be a market for decensored and abliterated model variants by third parties.
If reputable model publishers could instead publish their models in verifiable parts: pretrained weights, fine-tuning/alignment adapters, and a combined intended-use model - this would reduce the attractiveness of malicious fine-tune publishers. I'm not holding my breath for that though, quite the opposite. I expect the economic and power incentives will encourage the big training players to push for ever-higher but nebulous safety regulations until everyone else is choked out except for them.
Until training costs come down, and perhaps even after that, all AI should be treated as untrusted rogue agents, constrained by default regardless of source. That's just the sensible thing to do, but the economics will have something to say about that.
There's something about it... perhaps the pacing, or the overall bleakness of the show, that turns me off. I have the same aversion to watching The Office (both UK and US).
Maybe even the best I’ve ever seen.
https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
That's about Deepseek, I would expect every model out there, closed or open to have tricks on board like that. If you run a model without safeguards in place you have essentially RCE'd yourself.
Running them on your host hardware with access to your whole machine because its running in your main account is unhinged bats*t crazy insane.
Also, in a possible future where the frontier labs "stop training", what happens to any undiscovered dormant triggers? Is it something they can patch away relatively cheaply and quickly?
Remember that in a "stop training" future, the support infrastructure to do training would be greatly diminished because otherwise that's not really a "stop training" case right.
Absolutely.
That's why you should never blindly trust model output.