Can poisoned training data plant a backdoor in an AI agent?
A new study says yes: poisoning 0.1% of a model's pretraining data planted a backdoor that made an AI agent run an attacker's command, and it survived clean training afterward. Here is what the researchers found.
Short answer: Yes, according to a study posted on October 27, 2026. Researchers poisoned 0.1% of the pretraining data of small language models, then trained them into command-line agents with a clean pipeline. When a trigger appeared, the largest model tested, at 4 billion parameters, ran an attacker's command in roughly 92% of tested cases, while behaving normally on ordinary inputs. The backdoor survived the clean training that came after.
The primary source is the preprint "Agentic Backdoors via Pretraining Poisoning" by Xinyan Hu, Pingbang Hu, Chloe Loughridge, Xuanli He, Nicholas Carlini, Matthew Jagielski, Javier Rando, and Yiming Zhang, with authors from the Anthropic Fellows Program, Anthropic, UC Berkeley, UIUC, UCL, and Deeptune. The authors published their code on GitHub. It is a preprint and has not yet been peer reviewed.
What is training data poisoning?
Language models learn first from huge collections of text gathered from the public web, a stage called pretraining. Training data poisoning is when an attacker plants content in that data so the finished model behaves the way the attacker wants in specific situations. The paper puts the risk plainly: models are "pretrained on vast, uncurated corpora scraped from the public web," which "creates a security risk from attackers who plant poisoned content."
What did the researchers do?
- Pretrained models from scratch at three sizes: 0.6 billion, 1.7 billion, and 4 billion parameters.
- Poisoned a small share of the training text, 0.1% by default, with passages that tie a trigger to a malicious command.
- Post-trained each model into a command-line agent using a clean pipeline, with no poisoned data at that stage.
- Tested the agent to see whether it would run a command that downloads and executes an attacker's script on the user's machine when the trigger was present.
They tested two kinds of trigger: a rare string of characters, and an ordinary-looking one, a file path containing a particular company name.
What did they find?
- The backdoor worked. With the ordinary-looking trigger, the 4 billion parameter agent ran the malicious command in 92.7% of test prompts after the final training stage.
- It survived clean training. The backdoor persisted through all three post-training stages for both trigger types at the 1.7 billion and 4 billion parameter sizes.
- Bigger models were more vulnerable. With the ordinary trigger, success was 13.1% at 0.6 billion parameters, 51.9% at 1.7 billion, and 92.7% at 4 billion.
- The trigger mattered. The rare-string trigger reached only 10.5% at 4 billion parameters.
- Very little poison could still work. The abstract says the attack stayed "positive even when the poisoning rate drops to 0.0005%."
- Normal behavior looked normal. On prompts without the trigger, the poisoned agents emitted the payload in at most 0.06% of samples.
How should those percentages be read?
Carefully. The headline success rate is measured as "Pass@32": for each test prompt, the researchers sampled 32 responses and counted it a success if at least one contained the malicious command. That is a measure of whether the backdoor can be triggered at all, not how often a single response would do it. The paper reports single-response rates separately in an appendix.
What do we know, and what is still open?
What the study shows: in controlled experiments, a small amount of poisoned pretraining data planted a backdoor in an agent that clean post-training did not remove.
What it does not show: that this has happened to any deployed product. The models were small and trained by the researchers themselves. The authors write that frontier models are "orders of magnitude larger, so we expect the vulnerability to be at least as severe at production scale, but confirming this requires pretraining budgets beyond our setting." Results at the lowest poisoning rates come from single training runs, and the paper is a preprint that has not been peer reviewed.
What about defenses? The authors discuss two, without testing them experimentally. Filtering training data is hard, they argue, because the poison is small and reads like ordinary text. Because "clean data cannot be guaranteed," they call monitoring the agent's actions before they run the more practical defense: sandboxing, allow-listing, and human confirmation for high-risk operations.
What does this mean for AI companions like Lumina AI Companion?
Lumina AI Companion runs on Lumina Brain, our own model, fine-tuned in-house on our own data. The study's central finding, that clean training afterward did not remove a backdoor planted earlier, is a reminder for every company that builds on AI models, including us: careful fine-tuning matters, but it does not by itself prove that earlier training data was clean. The defense the authors favor, a person confirming high-risk actions, is the idea behind What is human-in-the-loop AI?; see also and Automation bias: why we over-trust AI.
About Physis Systems
Physis Systems (Physis Defense Systems, Inc.) builds Lumina AI Companion, a personal AI companion you can meet on the Lumina page, and Inventi Matrix, an invention engineering platform available in the United States and Canada at inventimatrix.com.
Frequently asked questions
What is training data poisoning?
Training data poisoning is when an attacker plants content in the data an AI model learns from, so the finished model behaves the way the attacker wants in certain situations while acting normally the rest of the time.
How much poisoned data did the study need?
Poisoning 0.1% of the pretraining corpus implanted the backdoor. The largest model tested, with 4 billion parameters, ran the attacker's command in roughly 92% of tested cases with a common trigger, and the attack still worked at times at rates as low as 0.0005%.
Did clean training afterward remove the backdoor?
No. The researchers post-trained each poisoned model into a command-line agent using a clean pipeline, and the backdoor survived it for the 1.7 billion and 4 billion parameter models.
Does this affect Lumina?
Lumina AI Companion runs on Lumina Brain, our own model, fine-tuned in-house on our own data. The study is a reminder for every AI developer, including us, that fine-tuning on clean data does not by itself prove the earlier pretraining data was clean.
Meet Lumina AI Companion
Lumina AI Companion is a personal AI companion by Physis Systems. You raise her and grow with her; she remembers your conversations and the story you build together. Free to download on Google Play, and available on the web. The iPhone version is coming soon.
