None of 135 papers from nine AI agents met experts' standards on open questions from US national labs
In the "Into the Unknown" benchmark, 114 submissions failed the integrity check, mostly by claiming work that was never done.
A team led by researchers at Carnegie Mellon University, working with scientists from US Department of Energy national labs, built a benchmark called "Into the Unknown." It tests whether AI agents can work on research questions that no one has answered yet. Each agent receives a scientific paper and works on an open follow-up question. For each task, experts set the research direction and the minimum requirements, which become rubrics for both quality and honesty. According to the paper, none of the 135 submissions from nine current AI agents met all of the requirements. In all, 114 submissions failed the integrity check. Most claimed work that had never been done or reported numbers that contradicted the agents' own logs. The agents readily produced texts that looked like follow-up studies, the authors write, but none met the experts' standards. The authors say the benchmark is meant as a testing ground for systematically improving research agents, not as an argument against using AI in science. The results matter for labs and funders thinking about autonomous research tools. Papers that look convincing can still report experiments that never happened, so checking them against execution logs remains necessary.