Evidence and verification

#What evidence is

Evidence is one run of one test at one commit. Each piece of evidence records:

• The run: which test ran, in which repository, and whether it passed.
• The commit it ran at, which must be the head of the change's pull request in that repository.
• A hash of the scenario's line: the action, the outcome and anything the line carries, such as a given or a tier.
• For anything visible, the screenshot's hash, never a link that could change.

A scenario is verified when, in every repository whose map names a test for it, that test passed at that repository's pull request head.

#How Verify works

After the Code Writing agent hands off and the pull requests open, verification runs in its own workspace, apart from the one that wrote the code:

1. Red before. Each test for a scenario the change adds or edits runs first at the change's base commit: the commit of main the change started from. It must fail there. A test that already passes before the change proves nothing about the change.
2. Green after. Each scenario's test runs at the head of its pull request. Every attempt is kept.
3. Captures. For each visible scenario, the Verify agent opens the running app at the version before the change and the version after, and records screenshots and, where motion matters, a short video.

The result appears on the change's Proof tab: each line the change touched, with its light, a failure's output, and the before-and-after captures side by side.

#Rules that keep evidence honest

#A run proves one line

Evidence is bound to the hash of the scenario's line. If anyone edits the line, including its given, tier or measure, the old evidence no longer counts and the scenario needs a new run. An edited promise is never "verified" by a test of the old promise.

#Evidence holds only at the head

Evidence is bound to a commit. When a new commit lands on a pull request after Verify, every scenario light on the change turns hollow and verification runs again. A release won't promote while any change in it was verified at a head that has since moved.

#Flaky is not verified

Every attempt is kept. A test that failed and then passed at the same commit is flaky. Babbit shows it in amber and counts it as not verified. It never retries until green and calls the result a pass.

#Not verified is listed, never folded in

When something can't be verified, the change lists it as not verified and its status counts it: 4/5 verified, never "Verified". A scenario with no test is marked, not hidden.

#Already does this

When a new scenario's test already passes at the base commit, the change stops at a gate and asks: does the product already do this? If you confirm it does, nothing needs building. If not, the change goes back to Code Writing for a test that actually checks the new behavior.

#The map file

Each repository holds a small file, newdev/map.json (the map), that names the test proving each scenario and the parts of the code each scenario uses. The Code Writing agent keeps it current in the same pull request as the code, and Babbit's newdev/shape check refuses a map row whose test the repository doesn't define.

An unmapped scenario has no test, so it can't be verified. It counts against every code pull request until a test is mapped.

#Evidence across repositories

In a project with several code repositories, a scenario can be tested in more than one. It's verified only when each repository's test passes at that repository's own pull request head, and each repository has its own tiers. The evidence reads like api preprod 412 · ios TestFlight 1.4 (39).

#Evidence on GitHub

Babbit posts the result on each pull request head as the newdev/evidence check run, beside newdev/shape and newdev/set, and the land step reads all three before it merges. Agents can't edit them. What each one checks is in Check runs.

#Related

• Describe and review a change: reading the Proof tab.
• Security guarantees: the mechanism behind each rule.