CloudOpsBench finds AI agents fail many real AWS operations tasks, usually because of reasoning mistakes
Claude Fable 5.1 solved 60% of the 45 tasks and GPT-6 Astra solved 53%.
A team led by Ang Chen at the University of Michigan has released CloudOpsBench, an open-source benchmark that measures how well AI agents handle common DevOps work in the cloud. Version 1.0 has 45 tasks on Amazon Web Services, written by experienced DevOps engineers. The tasks include fixing services, removing resources, restricting access rights, rolling out updates and decommissioning projects. No model completed every task. Claude Fable 5.1 solved 60 percent and GPT-6 Astra solved 53 percent. The researchers say the main problem is reasoning, not missing knowledge. About 80 percent of failures happened because agents misjudged the scope of a task, did not check all the relevant services, or stopped too early. Roughly 20 percent were wrong actions, and some of those had irreversible effects on the infrastructure. Every task runs the same way in the real AWS cloud and in an emulator, so teams can test without touching live systems. Support for Google Cloud, Azure and Oracle Cloud is planned. The project is on GitHub under the Apache 2.0 license, along with a detailed failure analysis. The results matter to organizations that are considering giving agents access to production cloud accounts. In this setting, a misjudged task can delete or misconfigure resources that cannot be restored.