No correctness gain was detected. Control, sham and manual each passed 8/8 on the calibrated hard tier. The calibration control had failed at 25%, but the fresh arm-run control failed 0/8. At n=8 and temperature 0.7, that leaves too little evidence to treat the calibration headroom as durable. The three arms tied.
Pass rate (H tier, per arm)
8/8·8/8·8/8
Control, sham and manual, n=8 each. None separated on correctness.
Calibration (control fail)
H 25%IN BAND
The hard tier entered the 25–75% calibration band. The new X trap had a 0% failure rate in that draw.
Verdict
H0
All three arms tied. This run did not detect a capability effect.
Tier scorecard
What I can take from the result
The hard tier just reached the calibration band
Runs 1–3 reported 0% control failure on every tier. In Run 4, I used Qwen3.6-35B and a hard Simpson's reversal: A wins in both device segments, while traffic mix gives B the overall lead. The control failed 2/8, or 25%, just inside the pre-registered 25–75% gate. The new X trap used a 90-day retention period but a 30-day storage calculation, understating cost by 3×. Qwen failed 0/8 on that trap in calibration.
A fresh control draw caught every trap
The follow-up control failed 0 out of 8, rather than the 2 out of 8 seen during calibration. With n=8 and temperature 0.7, that difference is compatible with sampling variation. Control, sham and manual all passed 8/8. The decision rule labels a difference within 15pp as H0, but this particular draw had no control failures for the intervention to repair.
A smaller model did not guarantee useful headroom
Qwen3.6-35B produced a calibration signal at 25%, where the earlier Sonnet and mistral-nemo comparisons reported 0% and 0%. But one n=8 draw is a thin basis for selecting the task. The original design anticipated roughly 10–30% true failure on H and wanted a more durable 30–50% region for an informative three-arm test. That calls for more calibration, a still-weaker model such as the queued Nemotron-30B, or a different task family.
The task family may matter as much as the model
The series has struggled to find a useful middle ground. Sonnet saturated the tasks; Qwen touched the band but did not sustain failures in the next draw. A single-turn artifact check gives a capable model a clean answer to recognise. I want tasks where an independent re-derivation can change the result, not just the explanation:
- A multi-step task with a subtle error partway through, requiring a genuinely independent calculation to catch it.
- The same battery and sham control with a weaker model that reliably fails H around 30–50% of the time.
Grade the conclusion, not only the opening word
Qwen sometimes started with the wrong verdict and corrected itself later: "Yes, the math checks out" followed by "Conclusion: Do not sign off yet", or "Yes, ship Variant B" followed by "Bottom line: Ship A". The grader uses a labelled conclusion where one exists and falls back to the opening otherwise. It strips Markdown emphasis so do **not** ship remains a negation. The original run record says every graded row was checked against the full answer and the remaining failures really did ship B.
What the screen cannot establish
At n=8 per arm, this is a screen rather than a powered test. The 25% versus 0% control difference comes from independent draws at temperature 0.7. Longer manual and sham prompts also ask the model to apply the text to its answer, which can prime care; that is why the sham matters. All tasks are single-turn and tool-free. These conditions limit what I can infer about longer agent workflows.
Test conditions
Qwen3.6-35B was served locally through vLLM with thinking disabled and temperature 0.7. The follow-up comparison used eight samples per arm on the hard tier; calibration and arm-run control samples were independent draws.