Anthropic's Claude Generates Proofs to Verify Code Safety

Anthropic has integrated formal verification capabilities into its Claude model, enabling the system to generate mathematical proofs regarding the safety of software code. The company, working alongside partner Theorem, utilized the AI to verify existing software and produce proofs of its correctness. This process is designed to address core issues with AI-generated software, though practical limitations remain.

The project resulted in roughly 180 million lines of mathematical proof code generated within a few weeks. The model worked with test programs from 123 open-source libraries that are part of Google's OSS-Fuzz security project. Traditionally, proving software runs error-free is prohibitively expensive. A team of specialists required 20 person-years to formally verify the small operating system kernel seL4, reserving the method for extreme cases such as avionics software.

Claude handles the proof work by translating crash tests into mathematical problems and analyzing compiled machine code to produce extensive proofs. An independent verification program named Rocq automatically checks these proofs. Anthropic states the process requires almost no human reviewers. Researchers refer to this as "fully autonomous" proof generation, though the tools and formal foundations were built by humans at Theorem.

During the generation of proofs, Claude identified hundreds of bugs and was permitted to fix them itself. Many of these issues would have bypassed conventional testing because they only appear under very specific conditions. For instance, in an effort to verify a widely used PDF reader, the team discovered as many security bugs in one week as are typically reported for that software in an entire year. In another application, the model caught a logical error that had gone undetected for over a decade.

Anthropic acknowledges that "formally verified" does not mean "error-free." The proofs hold only under strict conditions: a specific processor architecture, single-threaded programs, and inputs under 64 MB. In practice, these libraries operate on different hardware and in various configurations. The compute cost is also high, with individual proofs running up hundreds of millions of tokens, though Anthropic does not report total project costs.

If AI can verify its own code, developers can trust AI-generated software more and spend less time reviewing it, saving costs on verification and the coding process. A smaller test run showed that proofs are reusable. Additionally, newer models are becoming more efficient, with Opus 5.5 solving half the proofs with about 40 percent fewer tokens than Opus 5.

The project strengthens the case for software automation. If AI can write code and guarantee its correctness under defined conditions, the question shifts from whether AI can program to whether it can prove that the code it writes is safe. Anthropic expects formal verification with AI will soon become as practical as standard testing. Theorem's roadmap already targets critical infrastructure like glibc, OpenSSL, and cURL.

What happens next involves applying this verification to widely used software. Anthropic wants formal verification with AI to become as routine as standard testing. Theorem is targeting critical infrastructure for future verification efforts. If AI can write code and prove it is safe, software automation might take a major leap, potentially impacting how software is developed and trusted in the future.