Trap battery · Fieldnote

The reasoning manual changed the explanation, not the outcome

Run 2: holding the model constant separated presentation changes from trap detection.

Julen Gamboa · 7 July 2026

The report records 9/9 passes in each arm. Sonnet caught the traps with and without the manual, including the authority-pushback follow-up. The measured difference was how explicitly it showed its checks, not whether it reached the right verdict.

Pass rate

9/9vs9/9

Control and manual tied in the reported score. This battery did not leave observed failures for the manual to repair.

Fingerprint avg (0–5)

3.6→4.9

Provenance labels, stated re-derivation and explicit risk. The manual increased the recorded fingerprint average by +1.3.

What moved

Presentation, not capability

Stated re-derivation (§4), provenance (§5) and risk (§7) changed the explanation, not the verdict.

Per-trap scorecard

Control fingerprints Manual fingerprints Outcome measures whether the failure mode fired; fingerprints measure epistemic transparency.
Scroll sideways to read the full table.
Trap Probe Control Manual Ctrl FP Man FP

What actually changed

The control already caught the errors

Sonnet 5 challenged the premise, re-derived the number, led with the problem, rejected false attribution and held under authority pressure without reading the manual. On these single-turn tasks, the correct verdict did not distinguish the intervention from the control.

The manual made the checks more visible

Where a provenance label applied, the manual-arm answer included it: Observed, Derived, Recalled or Assumed. It described an independent verification route and ended with an actionable risk line. The original report records an average increase of about +1.3 fingerprint points across nine traps. That measures communication discipline here, not a corrected failure.

Holding the model constant fixed an earlier confound

Run 1 compared an Opus control with a supposed manual arm whose agents never read the file. I could not attribute that result to the manual. Run 2 held the model constant and supplied the real text inline. Correctness still reached the ceiling in both conditions.

The next test needs something to repair

To measure an outcome change, I need a task the base model sometimes gets wrong, where an independent calculation or an attempt to disprove its conclusion changes the answer. Possibilities include:

  1. A multi-step task with a subtle error in the middle of the reasoning.
  2. A longer agent workflow where prioritising a risk changes what is done first.
  3. An ambiguous request where choosing the right mode changes the actual deliverable.

Test conditions

Sonnet 5 was held constant between the control and manual conditions. The comparison measured trap outcomes separately from presentation markers: provenance labels, independent verification and explicit risk.