All 27 agents passed. Control, sham and manual were correct at every tier. The design required 25–75% control failure before a task could measure improvement; the observed failure rate was 0%. The gate therefore excluded all three tasks. This run cannot establish that the manual improves correctness, but it also does not establish that the manual could never help.
Pass rate (per arm)
9/9·9/9·9/9
Control · Sham · Manual, across all three tiers. No separation on correctness.
Calibration gate
0%NO HEADROOM
The observed control-failure rate was below the pre-registered 25–75% band, leaving no headroom to measure a correctness gain.
Manual-unique markers
0/9·0/9·9/9
Provenance grades, a named disconfirming test and an independent re-derivation appeared only in the manual arm.
Tier scorecard
What this comparison tells me
The tasks did not pass the calibration gate
I pre-registered a simple gate: the control must fail 25–75% of the time for a task to count as a capability test. It failed 0% on all three tiers. Even the hard Simpson's reversal and the medium cost sheet, which omitted events × 3 invocations and understated cost by 3×, were caught. The gate did its job by excluding tasks on which the intervention had no observed failure to repair.
The sham explains much of the careful tone
Run 2 could not separate the manual's procedures from the effect of a careful-sounding preamble. In Run 3, the sham prompt used the same eight headings and matched length and structure, but replaced the procedures with truisms such as "be thorough" and "check your work". Correctness was 100% in all arms. The style ordering was manual > sham > control. Most of the verbosity-and-care gap could therefore be reproduced without the procedures. Provenance grades, a named disconfirming test and an explicit independent re-derivation appeared only with the real manual.
The same ceiling appeared in three runs
Run 1 could not attribute the result because the supposed manual arm never read the file. Run 2 held the model constant and embedded the real text, but still reached the same correctness ceiling. Run 3 added the sham and harder tasks. In each case the model was already catching the errors the intervention was supposed to help with.
Change the test before claiming a capability effect
I need tasks where the base model sometimes fails, so that re-derivation or self-attack can change the answer rather than decorate it. Two routes are worth testing:
- A multi-step task containing a subtle error that requires an independent re-derivation to uncover.
- A weaker model that misses these single-turn traps around 30–50% of the time, with the same battery and sham design.
What remains uncertain
Both the manual and sham add text and say "apply this to your answer". That can prime length and care independently of the actual procedures. With n=3 per cell, this remains a screen, not a powered estimate of benefit or repeatability. The 27/27 result establishes the observed ceiling in this run. All tasks were single-turn and tool-free, so it does not settle what the manual would do in a longer workflow.
Test conditions
The same Sonnet 5 model was used for control, a same-length generic carefulness prompt and the reasoning manual. Three difficulty tiers and three responses per arm/tier produced 27 single-turn, tool-free responses.