Northwestern study finds AI security patches that stop a crash often fail the maintainers' own tests

Researchers at Northwestern University tested AI coding agents on 112 real security bugs from 84 open-source projects written in C/C++, Go and Rust. They found that many AI patches pass the usual benchmark check but still fall short. Many security patching benchmarks grade a fix on a single question: does the proof-of-concept input that triggers the bug still crash the program? A patch can pass that test by blocking the path to the crash while leaving the underlying flaw in place. The authors call such fixes strictly worse than no fix at all, because they make it look as if the problem is solved when it is not. To measure this, the team collected the tests that the original developers added alongside each real fix and ran the AI patches against them. With Claude Opus 4.7, a third of the patches that stopped the crash still failed those maintainer tests. In one PHP example, two patches passed the project's existing tests. Only the targeted test showed that one patch fixed the logic correctly while the other broke legitimate calls. The results matter for teams thinking about letting AI tools patch vulnerabilities automatically.