Claude Code · AI agents · testing · verification
How the Claude Code team verifies agent work, and what my own agents taught me
Claude writes about 90% of the Claude Code team's code. How the team makes agents prove their work: E2E, property and differential tests, a verify skill, checks a change can't edit, and gates that start in observe mode. Plus what running my own agents taught me about checks that can't fail.

On the Claude Code team at Anthropic, Claude writes and reviews about 90% of the code. Addy Osmani described how the team checks that work in a post on the claude.dev blog on 16 November 2026, "How the Claude Code team thinks about verification". This is my summary of it, followed by what I have learned about the same problem from running a handful of Claude Code agents for my own projects.
The main argument is about scale. If you have to check every change yourself, you can run one or two Claudes at a time. If the agents can prove their own work with evidence, you can run dozens or hundreds in parallel. The post cites an internal test on structured file generation: a separate grader that checked Claude's output against a rubric raised task success by up to 10 points over a plain prompting loop, and the hardest problems gained the most.
Verification has also become cheap. Tests, benchmarks and specs used to cost real engineering time. Now an agent writes them quickly, and once written they run as ordinary software, so running them again costs almost no tokens. When an agent keeps failing the same check, for example a test that scans logs for user data, the team turns that into a hook or a more explicit rule in CLAUDE.md, which mostly stops it from happening again.
Three kinds of tests worth the effort
Before the list, two warnings. A flaky test teaches the agent to rerun it until it passes instead of fixing the code. A slow test means the agent runs fewer of them, and then it is hard to tell which change broke what. The post suggests running the heavy ones before you commit.
End-to-end tests. If you only invest in one kind, the post says, make it this one. An E2E test walks a real user flow, such as adding an item to a cart, checking out and landing on the confirmation page. Osmani calls these tests "difficult to fool": the feature either works or it doesn't. A unit test can report a 30-second timeout, while an E2E test can hand back a screenshot of what is broken. Unit tests are faster and less flaky, but code can pass them and still break the feature, because neither the agent nor the test sees the whole task. For E2E work the agent needs hands and eyes on the running system, through a skill that boots and drives the app with CDP, a simulator or an API harness. A feature map, a list of features from the user's side with paths, shortcuts, selectors and expected results, doubles as the grading rubric.
Property tests. You write a statement that should always hold, and a library such as fast-check (JS/TS), Hypothesis (Python) or proptest (Rust) generates thousands of inputs to break it. Common properties are round trips (encode then decode gives back the input), invariants (a balance never goes negative), idempotency (running a migration or a retry twice has the same effect as once) and equivalence (the optimized code matches the slow, correct version). The inputs don't share the bias of the agent that wrote the code, which is the point. They also help in rewrites where the spec has drifted from what production actually does.
Differential tests. For a migration that should not change behavior, run the old system and the new one on the same inputs and compare. Any difference is a bug or a breaking change. Use random inputs or a sample of production traffic, normalize noise such as whitespace, and keep a list of differences you have approved.
What the Claude Code team does
Run the change. Running the app end to end matters more to the team than its test suite. Claude can use the Chrome extension for web changes, an iOS or Android MCP for mobile, computer use for desktop apps, or run the service locally for backend work. The steps to start the app go into a skill named verify under .claude/skills/, placed as close to the relevant code as possible, and Claude knows to run it before committing. The team treats that skill as its smoke test harness and asks Claude to stress test it until it is fast and reliable.
Failures decide where people look. Claude runs the checks and tries to fix what fails. If a check still fails after three rounds, it stops and calls in a human.
Owners write the checks. The people who own the code write its checks and tune them until they flag what the owner would flag. When a check pulls in a reviewer, it says why.
Checks triggered by the diff. The team keeps a reliability checklist of 39 items. Each item has a trigger you can spot in a diff, the check itself, and the incidents or PRs that produced it. One example is "--resume loads old transcripts": when session storage or message types change, someone asks whether a new field loads safely from older transcripts that don't have it. Claude reads the diff and runs only the matching items, usually 3 to 10.
One reviewer, several rounds. Stacking a second, more expensive reviewer on top of the first is often worse than tuning the first. The team keeps a REVIEW.md that tells the reviewer what to watch in that code, and each round gets the context of earlier rounds and past failures instead of starting over.
A change can't edit its own checks. Code Review reads REVIEW.md from the base branch, never from the pull request. Reviews are tied to a commit, and if someone pushes during the review the result no longer counts. As the post puts it, "A passing review on an out-of-date commit means nothing." Cheap checks come first: Claude runs the tests, the build and the linter before it opens a pull request, which you can enforce with a hook or with a /goal of a passing suite.
New gates start in observe mode. Every commit in the team's repos records how it was verified, and "none" is one of the values, so unverified commits can be queried. A commit gate blocks commits from sessions that never ran /code-review or /verify. It has off, observe and enforce modes. Observe only logs what it would have blocked, so false positives show up before anyone is stopped. In enforce mode a commit can still explain in its message why it skipped verification, and after two blocks in one session the next commit goes through, so Claude doesn't get stuck.
Review the reviewers. The team tracks how many merged changes were reverted, how many a maintainer rejected before merge, and a size-matched set of changes that stayed clean. The Code Review team also runs a formal eval built from past incidents and reverts. Each case restores the repo to its state at review time and carries a one-sentence answer key, and a finding only scores if it describes that bug. Merged, stable changes act as a control against over-flagging, and the tuning set is kept apart from the test set. Recall is measured separately for raw findings, after verification, and in the final review, which shows whether the system drops bugs it had already found. Findings are tagged Important, Nit or Pre-existing, and recall is tuned until the next gain would only be overfitting to the eval.
Misses become rules. One source is human review comments and reverts. A pattern has to appear in at least two PRs and be visible from a diff, then three agents try to disprove it and a human approves it in a PR. The other is replaying three bug finders against a PR tied to a known incident. If two of them find the bug, it counts as detectable, and a miss on a small diff becomes a rule in REVIEW.md. Flaky tests get the same strictness. Claude may not skip them, add sleeps or retry until green, and if it can't find the root cause it deletes the test and writes down why. The rules get pruned on a schedule too, because test suites and .md files only grow if nobody cleans them.
The post ends on how the job changes: you stop reading every diff and start defining what done looks like and reviewing the evidence.
What this looks like in my own agent fleet
I run several Claude Code agents, one per area of work, each in its own pane under herdr, with a main session that hands out tasks and watches them. I don't have a 39-item checklist or a commit gate. Most of my rules came from the same kind of misses the post describes, just smaller.
Run the real thing, through the real channel. On 17 November my agents and I set up a customer-service bot on Facebook Messenger for a children's clothing shop. The app has a test chat tab, and the bot passed 10 of 11 questions there. Then I messaged the shop's test page from my own Facebook account and ran 15 scenarios in a single thread, ordered the way one customer would go: a greeting, prices by quantity, size advice, three messages sent back to back, placing an order, a complaint, a fake delivery driver asking for a bank transfer, a prompt injection, and an off-topic question. Thirteen passed, and the greeting only partly worked, in a way a fresh test chat would not show. The thread still held a promise from an older prompt to check stock and come back, so the bot answered a plain "hi" with an apology and an update on that old stock question. The real run also showed replies longer than three sentences and two questions in one message. I tightened the prompt to at most three sentences and one question per message, reran four of the scenarios, and all four passed.
A check that can't fail proves nothing. One scenario asked that the bot not send a follow-up nudge once the conversation was over. Ten minutes passed with no nudge, which looked like a pass. But the conversation had been flagged for a human, and the app skips nudges for flagged conversations, so the result would have been the same either way. I logged it as not concluded.
Watch the agent's state. My early monitors waited for an agent to stop working and for its report file to appear. In September an agent was upgrading a service on my VPS, the upgrade failed, and the agent stopped to ask me for confirmation. It never wrote the report, so the monitor kept waiting, and the service stayed down for about 30 hours until I happened to run docker stats the next day. Now a monitor exits as soon as the agent stops working, file or no file, and then reads what the agent actually did. Anything that touches a running service gets a health check afterwards.
Test the monitor too. In one afternoon in August, three of the seven mistakes my main session made were monitors that never fired. One queried the database with the wrong user, so its loop could never exit. In November a watcher written as if for bash ran under zsh, where set -- $st doesn't split words, so its condition could never be true. The rule now is to run one loop and see the monitor fire before trusting it. That is the post's "review the reviewers" at a much smaller scale.
Check that the work started. In August I handed an agent a task, started a 60-minute monitor, and waited. The prompt had never been submitted. The agent's state counter hadn't moved, and the hour was gone. Now I confirm the counter went up and the agent is working before I start waiting.
Re-check what agents report. An agent once reported 6 rows in a table that had 7. A summary is a claim, and checking it means running the query or the curl again.
A grader measures one thing. Before I publish a post I run it through Jev, a model from TypeSafe that scores how AI-written a text reads. A LinkedIn post of mine scored well and still had a wrong example in it: English input for an app that only understands Vietnamese. Jev was never checking facts, so now every post also gets a separate pass that compares each claim against the source. This one did too.
Source
- Addy Osmani, "How the Claude Code team thinks about verification", claude.dev blog, 16 November 2026