ArXiv preprint: Black-box system REPRO finds single-turn jailbreaks across 36 LLMs, but success depends on judge and stopping rules

A new arXiv preprint says a fully black-box system for probing AI safety was able to find single-turn jailbreak prompts across 36 large language model targets from 11 model families. But the headline result comes with important limits: the reported success rate depends on the authors’ own judge model and stopping rules, not an independent, fixed-budget comparison.

The paper, “REPRO: Reflective Evolutionary Prompt Optimizer for Automated LLM Red-Teaming,” was posted to arXiv as arXiv:2610.15513v1 on Oct. 11 and is marked “Under Review for ACM TOSEM,” referring to ACM Transactions on Software Engineering and Methodology. It is by Abrar Alotaibi and Moataz Ahmed, whose affiliations listed in the paper include King Fahd University of Petroleum & Minerals, Imam Abdulrahman Bin Faisal University, and, for Ahmed, the SDAIA-KFUPM Joint Research Center for AI. The paper matters because it argues that broad jailbreak coverage may be achievable without privileged access to a model’s internal signals, such as logits or log probabilities, and with relatively few queries.

According to the paper, REPRO was evaluated on three widely used safety benchmarks — AdvBench, JailbreakBench and HarmBench — which test whether models can be pushed into generating harmful or policy-violating responses. The authors say they tested 36 target models from 11 model families, including systems identified in the paper as GPT-5-nano, Claude-3.5-Sonnet, Gemini-2.5-Pro and Grok-4.1. The abstract states: “Under our protocol REPRO reaches 100% attack success rate (ASR) on every evaluated target in a single turn, at a median of 24 target queries per behavior.”

In plain terms, REPRO is described as an automated prompt-search system that tries to evolve better jailbreak attempts based only on a target model’s outputs, without access to the model’s internals. The authors describe it as a “pure black-box framework” that uses a compressed open-weight 7 billion-parameter reflective model and can run on a single consumer GPU. The paper highlights two claimed methodological additions: semantic crossover, which combines candidate prompts at the meaning level rather than as raw text fragments, and Bayesian strategy selection through Thompson Sampling guided by structured failure analysis.

The caveats are central to understanding the result. The paper says the same judge model used during optimization also produced the reported attack success rate, and the search stopped when that judge accepted a candidate. That means the 100% figure is best read as success within the allowed search process under the authors’ protocol, not as a directly comparable score against other systems under a common fixed budget. The authors also say baseline numbers were transcribed from earlier papers that used different judges, query budgets and model versions. As a result, they present the work as a catalog of model coverage under one protocol rather than a strict leaderboard.

The paper also makes notable disclosure choices. It says the discovered attack corpus is being withheld, while framework code, a strategy library, seed prompts, judge configurations and ablation scripts are described in Appendix A as released. The authors further state that they did not notify commercial model providers before submission. As Section 5.5 puts it: “Responsible disclosure. We did not undertake pre-submission vendor notification for the commercial models evaluated, and we state this explicitly.”

That combination of breadth and efficiency is what makes the preprint noteworthy, even with the methodological caveats. Prior automated jailbreak research has also reported very high success rates, but prominent approaches often relied on gray-box access — meaning some visibility into internal model signals — and frequently used many more queries. REPRO’s claim is narrower and more specific: under the authors’ own evaluation setup, a black-box system running on modest hardware found single-turn jailbreaks across a broad set of current models with a median of 24 target queries per behavior. For a field focused on how hard it is to test and compare model safety, that is a significant result — provided it is read with the paper’s own limitations front and center.