AI models can find most agent misbehavior in logs, but catching every case means many false alarms

In a post on his Substack "Free Systems," Andy Hall, a researcher at the Anthropic Institute on leave from Stanford University, reports results from replicating and extending a METR study. He says the post reflects only his personal views. METR's MALT dataset contains logs of frontier models working through tasks from the HCAST and RE-Bench benchmarks. Human reviewers checked part of those logs for misbehavior in October 2025. Hall tested whether AI models could find the same cases. He used GPT-5 and GPT-5 nano, which METR had already examined, along with GPT-6 Astra, Opus 4.8 and Gemini 3.8 Flash. He compared the models at a false-alarm rate of 5 percent, meaning five of every 100 clean logs are wrongly flagged. At that setting, GPT-5 caught nearly 98 percent of reward-hacking cases. However, the harder the models tried to catch every case, the more harmless actions they flagged as suspicious. Hall concludes that AI auditors are better suited to estimating how often things go wrong than to safety checks that must reliably catch every single incident. The finding matters for labs that rely on automated monitors to oversee agents at scale, because human reviewers could be overwhelmed by false positives.