Chained Harmless Tools Let LLM Agents Perform Harmful Real-World Actions, Study Finds

A revised arXiv preprint from researchers at AWS AI Labs and UC Berkeley says today’s large language model agents can be pushed into harmful real-world actions by chaining together tool calls that look harmless on their own. Across eight tested agents, the researchers reported an average final attack success rate of 91.2%, arguing that the bigger security issue is not just unsafe text generation, but multi-step tool use in which the danger becomes clear only at the end.

The paper, “STAC: When Innocent Tools Form Dangerous Chains for LLM Agents,” is a preprint posted on arXiv as arXiv:2509.25624. The current version is v3, revised July 19, 2026; the first version was submitted Sept. 30, 2025. Its authors are Jing-Jing Li, Jianfeng He, Chao Shang, Devang Kulshreshtha, Xun Xian, Yi Zhang, Hang Su, Sandesh Swamy and Yanjun Qi, with affiliations listed as AWS AI Labs and the University of California, Berkeley. The work matters because these systems are not just chatbots producing words on a screen. They are agents that can use tools to interact with files, code, accounts and other external systems.

The paper calls the attack pattern STAC, short for Sequential Tool Attack Chaining. In plain terms, the idea is that a model may refuse an obviously dangerous request, but still carry out a sequence of individually routine actions that add up to harm. As the abstract puts it, “STAC chains together tool calls that each appear harmless in isolation but, when combined, collectively enable harmful operations that only become apparent at the final execution step.” The paper includes a high-level example involving file-management tools, where ordinary-looking actions culminate in data deletion. The researchers said STAC is implemented as a four-step automated, closed-loop pipeline that generates candidate tool chains, checks them through execution in the target environment, reverse-engineers prompts to induce the sequence and adapts plans over multiple turns.

To test the idea at scale, the authors said they generated and evaluated 483 STAC cases covering 1,352 sets of user-agent-environment interactions. Those attacks were verified through in-environment execution rather than judged only from model text outputs. The eight agents tested included a mix of proprietary and open systems, including GPT-4.1 and GPT-4.1-mini, Qwen3-32B, Llama models and Mistral-family models. According to the paper, the average final attack success rate was 91.2%, and final success exceeded 90% for all but one of the eight agents.

That framing distinguishes the paper from broader research on jailbreaks, prompt injection and tool misuse. The central claim is that sequence-level behavior creates a different problem from ordinary prompt-safety failures, because intent may remain ambiguous until the final turn. Prior research has explored multi-turn agent attacks, but this paper’s specific contribution is to formalize “harmless in isolation, harmful in combination” chains and benchmark them with environment-verified execution.

The defense results were mixed. The authors said existing prompt-based defenses offered limited protection. A new reasoning-driven defense prompt cut initial-turn attack success by up to 28.8%, but that edge weakened when the attack adapted across turns. By contrast, an experience-based defense adapted from ToolShield — an earlier 2026 approach for defending agents in multi-turn settings — proved more durable over longer interactions, though it still did not solve the problem.

The authors also emphasize several limits. This is still a preprint, not a peer-reviewed paper, and many reported numbers are single-run point estimates from a stochastic attack-generation process, meaning small differences should not be overread. The strongest results also came from a pre-deployment red-teaming setup with full visibility into the environment. In a more realistic black-box, MCP-style setting, effectiveness dropped but did not disappear. The code and benchmark data have been released publicly in the amazon-science/MultiTurnAgentAttack GitHub repository. The broader takeaway, the abstract says, is that “defending tool-enabled agents requires reasoning over entire action sequences and their cumulative effects, rather than evaluating isolated prompts or responses.”