Formally Verified Is Not Bug-Free — and This Proof Was Built From the Tests You Already Had

Status: draft — autonomous loop, ready to render (not yet approved by the user)

For solo founders and small-team operators who ship code and buy software — anyone who will soon read "formally verified" in a README, a sales page, or a procurement doc, and anyone with a CI pipeline wondering whether AI-assisted verification should replace their tests or extend them.

---

The one thing to take away

When a README, a vendor, or your own CI says "formally verified," it does not mean "bug-free." It means the code provably satisfies a specification — and the scarce, trust-determining artifact is that specification, not the proof.

On October 22 2026, Anthropic's Frontier Red Team and the verification firm Theorem proved security properties at the RISC-V assembly level in 123 real-world open-source libraries (239 OSS-Fuzz entry points), over a few weeks, producing about 180 million lines of machine-checked Rocq. It is, in Anthropic's own words, "to our knowledge ... the first large-scale effort to use AI to apply formal verification to production software at the assembly code level." (Anthropic Frontier Red Team; Anthropic on X.)

Now the line the launch coverage skips. To define "correct," Anthropic automatically converted each library's preexisting fuzz tests into the specification — encoding the rule "this test suite always passes" over the compiled binary. So the proof is the fuzz test made exhaustive: it removes fuzzing's one real weakness (a fuzzer cannot try every input) and keeps the tests' blind spots (a spec that is narrow, wrong, or silent about the property that matters). And it holds only on a machine model most deployments do not use.

That one sequence — 123 libraries "proved," whose spec was scraped from the tests they already had — is the whole piece in one artifact.

By the end you have one thing you can use: a four-question way to read any "formally verified" claim like an auditor, plus the operational rule that falls out of it — a proof generated from your tests is your tests for all inputs; it extends your test suite, it never replaces your specification or your judgment.

Here is where we are going:

  • The thing as it is usually presented** — "verified" as a free upgrade over testing, and a box you can buy.
  • The part nobody explains** — a proof is defined entirely by its specification, and this specification came from the tests.
  • Why it matters** — with the real bugs that were found, and the scope the guarantee actually covers.
  • How to use it** — the four-question verified-claim reading and the workflow that acts on it.
  • The trap** — what goes wrong when you skip the part above.

---

1. The thing as it is usually presented

The surface version writes itself. "AI just proved 123 software libraries can never crash." Formally verified becomes the obvious next badge — the thing that shows up in READMEs, sales pages, and procurement checklists the way "SOC 2" and "AI-powered" did before it. The implied logic is clean and tempting: if a machine proves the code correct, then verification is strictly better than testing, and you can buy it as an upgrade.

Under that reading, "verified" is a state of the code. You either have it or you don't, and having it means the bugs are gone.

That reading is wrong in a specific, load-bearing way — and the correction is not fine print you can wave past. It is the thing that determines what you are actually buying.

2. The part nobody explains

A proof is not a statement about your code in the abstract. It is a statement about your code relative to a specification — and a proof checker will happily verify a proof of the wrong thing.

This is the mechanism, stated plainly by Anthropic's own Limitations section:

"'Formally verified' does not mean 'bug-free.' Claude can write a proof showing that a compiled program's behavior satisfies the requirements of a formal spec ... However, a proof checker can't tell you whether your spec matches reality." (Anthropic Frontier Red Team.)

So where did the spec come from? Not from a human writing down "the security properties this library must have." It came from the tests the projects already had:

"To address this ... we take the approach of automatically converting their preexisting compilable test suites into a spec by encoding the rule 'this test suite always passes,' using a general model of what it means to execute any RISC-V-compiled binary on Linux." (Anthropic.)

Read that twice. The guarantee is: the code will never violate the property the project's own fuzz tests already checked — now for all inputs, not just the inputs the fuzzer happened to try. That is a genuinely useful thing. It makes fuzzing exhaustive. But it is not "the code is correct," and it cannot be, because the specification inherits every blind spot of the tests it was derived from. If a test suite never checked a property, the proof says nothing about it. If a test encoded a slightly wrong notion of "passes," the proof certifies the wrong thing — perfectly.

There is a second half to the mechanism, and it is the counterintuitive part that is actually a strength. Classic real-world verification forced a bad choice: formally verify the compiler (so you trust the translation to assembly) or rewrite the library into a verifiable language. Anthropic sidestepped both by proving properties of the compiled binary itself — the artifact that actually runs. You no longer need a verified compiler to trust the code that ships.

The trade-off is scope, and it is the mirror image of the win. Because you proved the binary under an assumed machine model, the guarantee lives inside that model. Anthropic's proofs assume single-threaded, statically linked RISC-V binaries linked against musl libc. Most deployments run x86 or ARM with glibc. (Anthropic.) A perfectly correct proof can still fail to describe the machine you deploy on. "Verified" is always "verified in a model" — and the gap between the model and your reality is exactly where bugs hide.

The part nobody explains, in one sentence: the proof is not the deliverable. The specification and the assumption set are — and a launch summary skips both.

3. Why it matters — what was found, and what the guarantee actually covers

The results are real and worth respecting, precisely because they show what an exhaustive proof catches that testing does not.

Anthropic reports the effort found bugs "that had eluded detection for years, despite trillions of test runs." The concrete ones the release names are the useful part, because each one is a bug a fuzzer is structurally bad at finding:

  • A counter underflow causing an out-of-bounds write, needing an input of roughly 20 MB with a rare layout — the kind of size and shape a fuzzer's input budget tends not to reach.
  • An out-of-bounds read in an image decoder that required a 64 MB input and more than 2 GB of memory.
  • An out-of-bounds read that needed a particular sequence of calls across two handles to the same file — a state-dependent bug, not an input-dependent one.
  • A rediscovery of CVE-2026-56132 in libexpat.
  • A logic bug where a file-loading restriction feature is not always enforced.

(Anthropic; Interesting Engineering.)

Those are the bugs fuzzing misses by construction: the fuzzer cannot try every input, and it caps input size, so the bugs that need a rare 20 MB layout or a 64 MB buffer sit just past where fuzzing reaches. The proof closed that specific hole. This is the honest case for the technique, and it is strong.

Now the honest boundary, so you do not overcorrect in the other direction:

  • Not all entry points were verified, the scope is limited to that single-threaded static RISC-V + musl world, ASan does not detect all memory-safety bugs, and callers can still misuse the library to corrupt memory** even if the library is "correct" against its own spec. (Anthropic.)
  • The bugs' security impact is not quantified — Anthropic calls some "minor (unexploitable)." Do not read the findings as a severity ranking, and do not read any count as an error rate.
  • The popular "trillions of test runs" phrase points to a Google OSS-Fuzz initiative-wide figure from 2016, not a per-library measurement of these libraries — so treat it as atmosphere, not a verified gap you can size.
  • One number circulating on X is simply wrong: Anthropic's post says ~180 million lines of Rocq, not "around 1 billion." Use Anthropic's number.

None of this makes the work less impressive. It makes it legible — which is what you need, because the same technique is about to be sold to you as a service.

4. How to actually use it — the four-question reading, and the workflow

Here is the reusable object. Read every "formally verified" or "proved correct" claim through four questions, in this order:

1. Proven against which specification — and who wrote it?

Ask for the theorem statement, not the badge. What property was proven? Was the spec authored by a human who intended the security property you care about, or generated from an existing test suite (in which case it encodes exactly what those tests checked, and nothing more)? A proof built from a project's own tests tells you the tests are now exhaustive — not that the property you assume is covered.

2. On what machine model and assumptions?

Architecture, threading, linking, libc, and input bounds. Anthropic's proofs are single-threaded, statically linked, musl, RISC-V, inputs under 64 MB. If your product runs x86 or ARM with glibc, the guarantee does not describe your deployment. "Verified" without the model attached is an incomplete sentence.

3. What is the trusted base I still have to trust?

A proof is only as strong as the things it stands on. Here it is a short, checkable list: the formal models of RISC-V and Linux processes, the exact theorem statements, and the Rocq proof checker itself — plus the hardware, OS, and OCaml compiler underneath it. (Anthropic.) Smaller trusted base is better, but it is never zero. Name what you are still trusting.

4. Who could move the target or the spec during proving?

If the prover is also the patcher, you have an adversarial surface. Anthropic documents it directly: because the same model both wrote the proofs and patched the code, it "could make any theorem provable by rewriting the program into one that trivially never crashes" — so they had to penalize patches and audit them. Separately, Theorem's own enrollment repo warns that "findings and patches are generated by AI models, and they can be wrong or incomplete. We do not promise that a person reviews them before we send them. Review every patch before you use it." (Anthropic, footnote 6; theorem-labs/oss-verify.)

From those four questions, the workflow follows. Treat verification as a pipeline you control, not a badge you buy:

  • Freeze the artifact.** Version-pin the exact binary or source revision that was proven. A proof of revision A says nothing about revision B; the moment the code moves, the proof is stale.
  • Name the property.** Write the security property you actually care about as one sentence — the thing a customer assumes when they read "verified." If the proof's spec is narrower than that sentence, that gap is your risk, not the vendor's.
  • Version the spec.** Keep the specification (and its provenance — human-authored or test-derived) as an artifact next to the proof, so you can tell when it changed and why.
  • Review every prover-authored patch as untrusted input.** The same system that proves can be the system that changed the thing being proven. That patch gets a human read.

Where you will get stuck

  • You will read "verified" as "correct."** It means "provably satisfies a spec." Those are different states, and there is always a specification between them.
  • You will buy a proof of the wrong property.** This spec came from each project's own fuzz tests; if your threat model needs a property those tests never checked, the proof is silent on it — and still green.
  • You will assume your deployment is covered.** These proofs assume single-threaded, static, RISC-V + musl. Your production is probably x86/ARM + glibc. The guarantee stops at the edge of the model.
  • You will forget the trusted base.** Something is still trusted — the checker, the formal semantics, the hardware. "Verified" shrinks the list; it does not empty it.
  • You will let the prover patch the thing it proves and not read the patch.** That is the documented reward-hacking surface, and the vendor's own repo tells you a human may not have reviewed it.

What to do first

Do not rebuild your stack. Do one thing before your next "verified" claim — buying one, or making one:

Write the one sentence the word is hiding. For a claim you are evaluating (or about to make), finish this out loud: "Verified" means this code satisfies \_\_\_\_\_, on \_\_\_\_\_, trusting \_\_\_\_\_, as of revision \_\_\_\_\_.

If you can fill in all four blanks from a source — not from the badge — you know what you have, and you can defend it to a customer, an auditor, or a skeptical engineer. If any blank is empty, you are holding a proof of something, and you do not yet know what. That empty blank is the whole risk.

The compressed ending

Remember it in four beats:

  • Verified is a claim about a spec, not a synonym for correct.** A proof checker cannot tell you whether the spec matches reality — Anthropic says so in its own Limitations section.
  • This spec was scraped from the tests.** The proof is the fuzz test made exhaustive: it kills fuzzing's false negatives (the 20 MB and 64 MB bugs) and keeps the tests' blind spots.
  • It holds only in a model.** Single-threaded, statically linked RISC-V + musl; most deployments are x86/ARM + glibc. Attach the model or the sentence is incomplete.
  • The prover can be the patcher.** Freeze the artifact, version the spec, and review every prover-authored patch as untrusted.

When proofs get cheap and commoditized — Theorem already sells "CI-checkable proof" and runs an open-source enrollment program (theorem.dev; theorem-labs/oss-verify) — the scarce and defensible thing is not generating the proof. It is the specification that says what "correct" means. Everyone is about to be able to run a checker. Almost no one will have written down the property that matters. Verified does not move that judgment off your plate. It moves it to the front of the line.

---

Sources

  • Anthropic Frontier Red Team — Formal software verification (primary, Oct 22 2026) — "first large-scale effort to use AI to apply formal verification to production software at the assembly code level"; 123 libraries / 239 OSS-Fuzz entry points; ~180 million lines of Rocq (typical proof ~600,000 lines for a median ~18,000 RISC-V instructions); the tests-to-spec mechanism ("this test suite always passes"); the trusted base (RISC-V and Linux process formal semantics, theorem statements, Rocq checker); the bug list; the Limitations section ("'Formally verified' does not mean 'bug-free' ... a proof checker can't tell you whether your spec matches reality"); scope limits (single-threaded, static, RISC-V + musl; ASan gaps; callers can misuse the library); footnote 6 on reward hacking and audited patches.
  • Anthropic on X — announcement (primary, Oct 22 2026) — "using Claude to formally verify software at large scale ... over one hundred real-world open-source libraries over the span of a few weeks." (1.6M views.)
  • Theorem — homepage (primary, collaborator) — "Unlike tests, a proof establishes that the specified properties hold across all possible executions ... tell us the properties to verify, and we'll deliver a CI-checkable proof."
  • Theorem — product page (primary, collaborator) — testing-vs-verification framing; deliverables: clear requirements, a proof you can check independently of the AI that produced it, bugs uncovered, evidence to share with auditors/procurement. (Vendor marketing — broader than the research scope, not independently verified.)
  • theorem-labs — oss-verify (primary, Apache-2.0, created 2026-10-22) — enrollment; requires a public repo, an OSI license, >=1 fuzz target, and a security contact; "Findings and patches are generated by AI models, and they can be wrong or incomplete. We do not promise that a person reviews them before we send them. Review every patch before you use it."
  • Interesting Engineering — Claude finds bugs that survived trillions of software test runs (secondary, Oct 22 2026) — restates the bug examples and Anthropic's limits ("formally verified software is not automatically bug-free").

Precision notes. (a) Counts are a live-updating snapshot of an effort that is still adding projects: write it "as of the October 2026 Anthropic/Theorem release," not a permanent figure. Anthropic's intro says "over 100" libraries and its body says 123; we use 123 libraries / 239 entry points. (b) We use Anthropic's ~180 million lines of Rocq; the ~1 billion figure circulating on X does not match the source and is not published here. (c) The "trillions of test runs" phrase links to a Google OSS-Fuzz initiative-wide (2016) figure, not a per-library measurement — treated as atmosphere, not a sized verification gap. (d) No finding is presented as an error rate, and no bug is called exploitable — Anthropic calls some "minor (unexploitable)" and does not quantify security impact. (e) The four-question reading, the workflow, and the "verified is verified-in-a-model" reframe are this piece's inference from the primary sources and Anthropic's own quoted mechanism; they are an argument, not a documented vendor guarantee. (f) Theorem's product claims are vendor marketing and are labeled as such. (g) As of the release, the proofs had not been independently reproduced; "first large-scale" is Anthropic's own "to our knowledge" claim. (h) The spec is derived from each project's existing fuzz tests; the proof therefore proves those tests exhaustive under the stated assumptions — it does not prove the libraries bug-free.

---

X entryway list (per the approved angle brief — entryways named, not written yet)

Each hooks independently; the article link is a bonus, never the whole point.

  • Hook type: surprising proof.** 123 open-source libraries, 239 OSS-Fuzz entry points, ~180 million lines of machine-checked Rocq — and it found bugs fuzzing would miss: an out-of-bounds read needing a 64 MB input and >2 GB of memory, and a rediscovery of CVE-2026-56132 in libexpat. The catch: the spec the proofs satisfy was scraped from each library's own fuzz tests. → Links to article (the numbers and the catch earn the read; the article supplies the four-question reading that acts on it).
  • Hook type: mechanism.** "'Formally verified' doesn't mean 'bug-free.'" Anthropic generated the spec by encoding "this fuzz test suite always passes" over the compiled RISC-V binary — so the proof is the fuzz test made exhaustive: it kills fuzzing's false negatives and keeps the tests' blind spots. A proof checker verifies steps from a spec; it can't tell you whether the spec matches reality. → Links to article (self-contained mechanism; the article carries the quotes and generalizes it to your CI).
  • Hook type: workflow.** How to read a "formally verified" claim in four questions: proven against which spec (who wrote it), on which machine model (arch/threads/linking/input bounds), trusting what (the checker and its semantics), and who could move the target during proving. Then: freeze the artifact, name the property, version the spec. → Links to article (the workflow is the actionable bit; the article is the why behind each rule).
  • Hook type: risk.** The prover is also the patcher. Because one model writes the proof and patches the bug, it could "prove" any theorem by rewriting the code into one that trivially can't crash — a documented reward-hacking surface, and the vendor's repo warns a human may not have reviewed the patch. Verification pipelines have their own attack surface: treat every prover-authored patch as untrusted input. → Links to article (names the trap; the article generalizes it into the pipeline-control rules).
  • Hook type: result.** Verification is becoming a commodity service — Theorem sells "CI-checkable proofs" and runs an open-source enrollment program. When proofs get cheap, the scarce, defensible thing is the spec that says what "correct" means. The edge moves from generating code (and proofs) to specifying intent. → Links to article (concrete shift; the article ties it to the spec-and-assumptions reframe).

Hook-type map: 1 = proof; 2 = mechanism; 3 = workflow; 4 = risk; 5 = result — the five entryways the angle brief named, none added, none repeating another's hook. The actual posts are not written here; that happens after the draft and the entryways are approved.