Preprint Shows Black‑Box Attack Can Evade Multiple Statistical Text Watermarks

A new arXiv preprint reports that a single black-box attack can evade several leading statistical watermarking methods for AI-generated text, pushing detection rates close to zero in the authors’ tests while largely preserving fluency.

The paper, “A Unified Attack Framework against Statistical Generative Watermarks for Large Language Models,” is a preprint, not a peer-reviewed journal article. It was posted to arXiv by Romina Omidi and Binghui Wang of Illinois Institute of Technology and Nima Naderloui and Yuan Hong of the University of Connecticut, with a first version submitted Oct. 20, 2026, and a revision on Oct. 29. The finding matters because text watermarking has been promoted as a way to track the provenance of AI-generated writing for moderation, compliance and other uses, and has figured in policy and product efforts including the White House’s voluntary AI commitments in 2023 and Google DeepMind’s SynthID-Text work published in Nature in 2024.

Statistical watermarks work by subtly changing how a language model chooses words or tokens as it generates text, so a detector can later test whether the output is likely machine-written. In this paper, the authors examine watermark families including KGW and related variants, GumbelSoft, DipMark, ITS and SynthID, saying the attack applies across seven schemes. “Black-box” here means the attacker can query the watermarked model and knows the detection rule, but does not have the secret keys, random seeds or model weights. The authors argue these systems share a structural property they call “score-ratio alignment,” which, they write, “allows an attacker with only black-box query access to the model to produce text that evades detection.”

The paper combines that theoretical claim with experiments on OPT-1.3B, LLaMA-7B and LLaMA-13B using the C4, FakeNews and ELI5 datasets. The authors say they tested 100 watermarked texts per condition. Their main metric was true positive rate at a 1% false positive rate — in plain terms, after fixing false alarms at 1%, the detectors correctly identified only about 0% to 5% of attacked watermarked text. Reported true positive rates fell to 0.00 to 0.05 across the tested settings. The paper also says perplexity, a common measure related to how predictable and fluent text is, stayed low and generally close to unwatermarked output, suggesting the attack did not noticeably damage text quality in those tests.

There are important limits to those results. The attack was evaluated with a high default query budget of 256 model queries per token step, meaning it is not cost-free and could be constrained by time, API pricing or rate limits in practice. And for SynthID, the paper notes that the full Bayesian detector requires offline training; in parts of the evaluation, the authors used a mean-score detector instead. That means the study should not be read as testing every production detector configuration exactly as deployed.

What stands out is not a claim that watermarking is dead, but that several widely discussed statistical approaches may share a common weakness. Earlier attacks often relied on paraphrasing or editing generated text after the fact, which can hurt quality. By contrast, this paper describes a generation-time attack under black-box access and says it works across “fundamentally different watermarking designs.” The authors’ broader argument is that watermark systems optimized for strong statistical detection within today’s generator-detector framework may be intrinsically vulnerable, and that the field may need, in their words, “fundamentally different watermarking designs.”