Preprint: AI agents solve most unpublished theoretical-physics calculations — authors flag caveats

A new arXiv preprint is making a striking claim — and attaching several important warnings to it. In a paper posted this fall, a 35-author research team says off-the-shelf AI agents were able to solve most unpublished, research-level theoretical-physics calculation problems with very high accuracy and at low cost. But the study is not peer-reviewed, each system was tested only once per setting, and the authors say they cannot rule out data leakage.

The paper, “TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving,” was first submitted to arXiv on Oct. 9 and revised on Nov. 12. It was led by A. Ramesh Chandra, with Suvrat Raju, a physicist at the International Centre for Theoretical Sciences in India, listed as corresponding author. The author list includes researchers from institutions including ICTS, the Tata Institute of Fundamental Research, the University of Texas at Austin, Harvard University, the University of Pennsylvania and Google DeepMind.

TheoPhysEval is meant to test a specific capability: whether AI agents can carry out the kinds of concrete calculations that come up in theoretical-physics research workflows. The benchmark contains 50 unpublished problems with verifiable answers. In the paper’s abstract, the authors describe them as “research-level theoretical physics problems,” but not famous unsolved questions. That distinction matters. The test is about doing checkable technical work, not inventing new physical theories.

The reported scores are unusually high. In the “realistic” track — where agents could search the web and write and run code — GPT-6-Astra, used in OpenAI’s Codex environment, passed 49 of 50 problems, or 98%. Opus 5.5, paired with Claude Code CLI, passed 46 of 50, or 92%. Gemini 3.8 Flash with Antigravity passed 34 of 50, or 68%. DeepSeek-V4.1-Flash with Codex passed 27 of 50, or 54%, and GLM-5.3 with Claude Code CLI passed 21 of 50, or 42%.

The paper also reports relatively modest costs for the top system. Astra’s full realistic-track run cost $116.28, or about $2.33 per problem.

Performance stayed high even when web access was removed. In the “isolated” track, Astra passed 46 of 50 problems, or 92%, matching Opus at 46 of 50. Gemini scored 31 of 50, or 62%, DeepSeek 25 of 50, or 50%, and GLM 26 of 50, or 52%. The authors say the comparatively small drop without web access suggests substantial memorization of the literature, not just live retrieval from online sources.

The authors also try to narrow what their result does and does not mean. They describe the benchmark tasks as “turn-the-crank” research problems — calculations that a world expert could typically solve in about a day. They explicitly say the problems are not important open problems and that success on them does not amount to autonomous scientific discovery.

Those limits are central to reading the paper. The benchmark may indicate that some AI agents are getting good at bounded, checkable technical work. It does not show that the systems can generate new physics, replace researchers or crack longstanding mysteries.

The paper’s own caveats are substantial. Each agent was run only once per track, which means there is statistical uncertainty around the results. Problems were cut off after eight hours. And the authors say they cannot rule out data leakage — in part because some contributors used AI tools while preparing solutions and because some material may have appeared publicly later.

The paper also says some apparent model failures turned out not to be model failures at all, but mistakes in the human reference solutions. Those references were corrected after consultation, and in those cases Astra matched the corrected answers. That point is especially relevant because benchmark quality has become a live issue across AI evaluation.

From 2024 through 2026, researchers have pushed to test AI systems on harder math and science tasks, especially tasks closer to real research work than textbook exercises. But benchmark reliability has become part of the story. A September 2026 audit of physics benchmarks found that grading mistakes and reference-answer errors had materially distorted some earlier model evaluations. TheoPhysEval’s own acknowledgment of corrected reference answers puts it squarely inside that broader debate.

For now, the paper’s strongest contribution may be that others can try to reproduce it. The authors released the problem set, grader and benchmark configuration on Hugging Face and invited reruns. Until independent groups do that, the headline result — near-perfect performance on a set of unpublished theoretical-physics calculations — should be treated as a notable preprint claim, not a settled finding.