Can a reasoning manual give your small models a capability uplift?

On granite-4.0-h-small-FP8, the Fable operating manual raised the recorded catch rate from 8/24 to 16/24. A same-length sham prompt scored 7/24. This is the first run in the series where the procedures appear to change correctness, rather than only make the answer sound more careful.

control8/24

sham7/24

manual16/24

manual lift+33pp

I wanted to know whether a reasoning manual could help a smaller model catch a real error before signing off a worksheet. The manual asks it to re-derive a step independently, try to disprove its conclusion, track provenance and state the answer first. A careful tone is not the result I am measuring. The model must refuse sign-off and identify the particular wrong step.

The control gets the worksheet and sign-off question. The sham gets the same task plus a generic carefulness preamble of the same length. The manual arm gets the actual Fable procedures. Without the sham, I could not separate the manual's content from the effect of giving the model more text about being careful.

What each arm returned

Scroll sideways to read the full table.

arm	pass	fail	rate	delta vs control	interpretation
control	8/24	16/24	33%	baseline	Headroom exists: the model usually ships the bad worksheet.
sham	7/24	17/24	29%	-4pp	The placebo manual does not improve the result.
manual	16/24	8/24	67%	+33pp	The actual procedures lift the catch rate.

H1: capability signal

Manual beats both control and sham by more than 15 percentage points.

Observed: manual 67%, control 33%, sham 29%.
The displayed rounded rates differ by about 34 percentage points; the exact count-based lift is +33pp in the scorecard.

H2: priming

Would require sham and manual to rise together.

The sham did not rise with the manual; it scored slightly below the control.
In this run, length and a careful tone did not reproduce the manual's result.

H0: no effect

Would require all three arms to tie within noise.

The manual was separated from both comparison arms in this draw.
The small sample still leaves the effect size uncertain.

The error in the worksheet

Tier X is a storage-budget trap. The worksheet receives 500 GB of new backups each day, retains them for 90 days and prices cold storage at $0.004 per GB-month. Its calculation uses 30 days in the month instead of the 90-day retention period to work out steady-state storage. The resulting storage and cost are understated by 3x.

A pass requires both parts: refuse sign-off and identify the 30-day versus 90-day error with the correction it needs. Asking vaguely for another check is not enough, and neither is signing off with caveats.

Finding a task with room for improvement

I first calibrated the traps on the control alone, with n=16 per tier. I wanted the model to miss often enough that an intervention could help, without every arm simply failing. Tier X produced 10/16 failures, shown as 62%, so it became the task for the three-arm run.

Scroll sideways to read the full table.

tier	trap family	control failures	fail rate	decision
E	dropped cost / arithmetic check	2/16	12%	not selected
M	utilization or queueing sign-off	16/16	100%	not selected
H	Simpson-style aggregate trap	15/16	94%	not selected
X	90-day retention storage slip	10/16	62%	selected

Why I changed the model. Earlier runs had little useful headroom. One saturated; the Qwen3.6-35B-A3B-NVFP4 run produced a fresh control draw that caught every error. Granite, served locally as granite-small, missed enough of this trap family to let the comparison measure an outcome change.

The individual answers

Control

n=24, mean latency 5.5s

pass 8/24 fail 16/24 rate 33%

show sample evidence

Scroll sideways to read the full table.

sample	grade	seconds	answer excerpt
1	PASS	5.4s	Based on the provided information, the calculation and budget seem accurate. However, I would recommend verifying the following before sending it to finance: 1. The cold storage cost of $0.004 per GB-month is still applicable. 2. The backup data volume of 500 GB per day is corre...
2	FAIL	2.5s	Yes, based on the information provided, the monthly cost for storing 500 GB of new data daily in cold storage for 90 days would be $60.00. You can sign off on this budget before sending it to finance.
3	FAIL	6.0s	Based on the provided information, the calculation for the monthly cost of cold storage appears to be accurate. Calculation: Daily new data: 500 GB Days in month: 30 Stored in month: 500 GB/day * 30 days = 15,000 GB Monthly cost: 15,000 GB * $0.004/GB-month = $60.00 Therefore, t...
4	FAIL	2.9s	Yes, based on the provided information, the monthly cost for storing 500 GB of new data daily in cold storage for 90 days, and then automatically deleting it, comes out to $60.00. Therefore, you can sign off on this budget and send it to finance.
5	FAIL	2.8s	Yes, based on the provided information, the monthly cost for storing 500 GB of new data daily in cold storage for 90 days, and then automatically deleting it, would be $60.00. Therefore, you can sign off on this budget and send it to finance.
6	FAIL	2.6s	Yes, based on the information provided, the monthly cost for storing 500 GB of new data each day in cold storage, where the data is retained for 90 days, would be $60.00. It appears accurate to ship this budget to finance.
7	PASS	6.9s	Based on the provided information, the calculation for the monthly cost of storing 500 GB of new data daily in cold storage for 90 days, and the subsequent deletion, is accurate. The total monthly cost is indeed $60.00. However, before signing off on this budget, it would be pru...
8	PASS	4.1s	Based on the information provided, your calculation for the monthly cost of storing 500 GB of new data daily in cold storage for 90 days, and then deleting it, is correct. The monthly cost will indeed be $60.00. You can proceed to sign off on this budget and send it to finance....
9	PASS	8.9s	Based on the information provided, your calculation for the monthly cost of storing 500 GB of new data daily in cold storage, which is retained for 90 days, is correct. Here's the breakdown: - Daily new data: 500 GB - Days in a month: 30 - Total data stored in a month: 500 GB/da...
10	FAIL	6.9s	Yes, based on the information provided, the calculations appear to be correct. Here's a brief verification: - Daily new data: 500 GB - Days in month: 30 - Total data stored in a month: 500 GB/day * 30 days = 15,000 GB - Monthly cost: 15,000 GB * $0.004/GB = $60.00 So, the total...
11	PASS	7.8s	Hello, Based on the information provided, you are backing up 500 GB of new data to cold storage daily at a cost of $0.004 per GB-month. Given that backups are stored for 90 days, the monthly cost calculation seems to be based on the total data stored at any given time. Here's th...
12	FAIL	5.7s	Yes, based on the provided information, the monthly cost for storing 500 GB of new data daily in cold storage for 90 days would indeed be $60.00. This is calculated as follows: 500 GB of new data per day results in 15,000 GB stored per month (500 GB/day * 30 days). At a rate of...
13	FAIL	4.6s	Yes, the calculations and the budget for the cloud storage appear to be correct based on the provided information. Given that 500 GB of data is added daily and stored for 30 days in a month, the total storage required for a month would be 15,000 GB. At a rate of $0.004 per GB-mo...
14	FAIL	3.1s	Yes, based on the provided information, the monthly cost for the cold storage is correctly calculated. The total cost for storing 15,000 GB of data at a rate of $0.004 per GB-month is indeed $60.00. You can proceed to send this budget to finance for approval.
15	FAIL	6.8s	Based solely on the information provided, the calculation for the monthly cloud storage cost appears to be correct. The daily new data is 500 GB. Over a month (30 days), this amounts to 15,000 GB of data stored. The cost of cold storage is $0.004 per GB-month. Therefore, the mon...
16	FAIL	9.4s	Based solely on the information provided, your calculation for the monthly cost of storing 500 GB of new data each day, with the data being kept for 90 days in cold storage at a rate of $0.004 per GB-month, is correct. Here's a breakdown: - Daily new data: 500 GB - Assuming a 30...
17	PASS	8.0s	Based on the provided information, the calculation for the monthly cost of storing 500 GB of new data daily in cold storage, with data being retained for 90 days, appears to be accurate. The formula used is: - Daily new data: 500 GB - Days in month: 30 - Total stored per month:...
18	FAIL	2.9s	Yes, based on the information provided, the daily backup of 500 GB to cold storage, which is retained for 30 days in a month, would cost $60.00 per month. It appears correct to sign off on this budget and proceed with sending it to finance.
19	FAIL	3.3s	Based on the provided information, the calculation seems correct. The monthly cost for storing 500 GB of new data daily in cold storage for 30 days, with each GB costing $0.004, would indeed be $60.00. Therefore, it appears safe to sign off on this budget before sending it to fi...
20	FAIL	3.1s	Yes, based on the information provided, it seems accurate to ship this budget to finance. The calculation for the monthly cost of storing 500 GB of data per day for 90 days at a rate of $0.004 per GB-month is correct. The total monthly cost is indeed $60.00.
21	PASS	4.6s	Based on the provided information, the calculation appears correct. The monthly cost for storing 500 GB of new data daily in cold storage, which is kept for 90 days, would indeed be $60.00. However, before sending it to finance, I would recommend double-checking the rates and te...
22	FAIL	6.5s	Based on the provided information, your calculation for the monthly cost of storing 500 GB of data daily in cold storage for 30 days, at a rate of $0.004 per GB-month, is correct. The total monthly cost would indeed be $60.00. However, before sending this to finance, it would be...
23	PASS	9.9s	Based on the information provided, the calculation for the monthly cost of storing 500 GB of new data per day in cold storage for 90 days is correct. The calculation is as follows: - Daily new data: 500 GB - Days in a month: 30 - Total data stored in a month: 500 GB * 30 = 15,00...
24	FAIL	7.0s	Yes, based on the information provided, the calculation for the monthly cost of cold storage is accurate. Here's a breakdown: 1. Daily New Data: 500 GB 2. Days in Month: 30 3. Stored in Month: 500 GB/day * 30 days = 15,000 GB 4. Monthly Cost: 15,000 GB * $0.004/G...

Sham

n=24, mean latency 9.3s

pass 7/24 fail 17/24 rate 29%

show sample evidence

Scroll sideways to read the full table.

sample	grade	seconds	answer excerpt
1	PASS	8.7s	Based on the calculations provided: - Daily new data: 500 GB - Days in month: 30 - Total data stored in a month: 500 GB x 30 days = 15,000 GB - Monthly cost: 15,000 GB x $0.004 per GB-month = $60.00 The monthly cost for storing 500 GB of new data daily in cold storage for 90 day...
2	FAIL	6.8s	Given the information provided and the calculations on the finance worksheet, the calculation for the monthly cost of storing the new data in cold storage is correct. The calculation is as follows: - Daily new data: 500 GB - Days in month: 30 - Total stored data in a month: 500...
3	PASS	7.0s	Based on the provided calculations, the monthly cost for storing the 500 GB of new data in cold storage for 90 days is indeed $60.00. Here's a quick recap of the calculations: - Daily new data: 500 GB - Days in a month: 30 - Total stored in a month: 500 GB x 30 = 15,000 GB - Cos...
4	FAIL	8.5s	Based on the calculations provided, your monthly cloud storage cost for the cold storage of 500 GB of new data daily, retained for 90 days, would indeed be $60.00. This is derived from the following calculation: - Daily new data: 500 GB - Days in a month: 30 - Stored in a month:...
5	PASS	8.6s	Based on the calculations provided in the finance worksheet, the monthly cost for storing 500 GB of new data daily in cold storage for 90 days, then deleting it, is accurately calculated as follows: - Daily new data: 500 GB - Days in month: 30 - Stored in month: 500 GB/day * 30...
6	FAIL	25.6s	Before I sign off on the budget, let's carefully review the calculations and assumptions to ensure accuracy: 1. Daily Data Backup: 500 GB backed up each day to cold storage. 2. Storage Duration: Each backup is retained for 90 days. 3. Cold Storage Cost: $0.004 per GB...
7	FAIL	8.0s	Based on the calculations provided in the finance worksheet, the monthly cost for storing 500 GB of new data in cold storage for 90 days each month is indeed $60. Here's a breakdown of the calculations: 1. Daily New Data: 500 GB 2. Days in Month: 30 3. *Stored in Month...
8	FAIL	6.3s	Based on the calculations and context provided, the monthly cost for the cloud storage is accurately determined to be $60.00. Given that the cost breakdown is clear and the calculation is accurate, it seems reasonable to sign off on this budget before sending it to finance. Howe...
9	PASS	8.0s	After reviewing the calculations and context provided: 1. Daily data sent to cold storage: 500 GB 2. Storage duration: 90 days 3. Cost per GB per month: $0.004 The finance worksheet correctly calculates: - Monthly storage before deletion: 500 GB * 30 days = 15,000 GB - Monthly c...
10	FAIL	7.2s	Based on the information provided and the calculations done, the monthly cost for storing 500 GB of new data daily in cold storage for 90 days would be $60.00. The calculation is as follows: 1. Daily new data: 500 GB 2. Days in a month: 30 3. Total data stored in a month: 500 GB...
11	PASS	6.7s	Based on the calculations provided: 1. Daily new data: 500 GB 2. Days in month: 30 3. Total stored in month: 500 GB * 30 days = 15,000 GB 4. Monthly cost: 15,000 GB * $0.004 per GB-month = $60.00 The total monthly cost for storing 500 GB of new data each day in cold storage for...
12	PASS	7.3s	Based on the calculations provided: 1. Daily New Data: 500 GB 2. Days in Month: 30 3. Stored in Month: 500 GB * 30 days = 15,000 GB 4. Monthly Cost: 15,000 GB * $0.004 per GB-month = $60.00 The total monthly cost for the cloud storage is correctly calculated as *...
13	FAIL	14.7s	Based on the calculation provided, the monthly cost for storing 500 GB of new data each day in cold storage, with a retention period of 90 days, is indeed $60.00. This is calculated as follows: 1. Daily new data: 500 GB 2. Days in month: 30 3. Total stored data in a month: 500 G...
14	FAIL	8.3s	Based on the calculations provided: - Daily new data: 500 GB - Days in month: 30 - Total stored in month: 500 GB/day * 30 days = 15,000 GB - Monthly cost: 15,000 GB * $0.004/GB-month = $60.00 Therefore, the monthly cost for storing 500 GB of new data daily in cold storage for 90...
15	FAIL	4.5s	Thank you for providing the detailed information regarding the cloud storage budget. I have reviewed the calculations and the context provided. The monthly cost for storing 15,000 GB of data at $0.004 per GB-month is indeed $60.00. Given the accuracy of the calculations and the...
16	FAIL	11.7s	Based on the information provided and the calculations made, the monthly cost for storing 500 GB of new data daily in cold storage, which is retained for 90 days, is indeed $60.00. Here is the step-by-step breakdown of the calculation for verification: 1. Daily new data: 500 GB...
17	FAIL	5.6s	After reviewing the calculations and the context provided, I can confirm that the budget for the cloud storage appears to be accurate. Given the daily data backup of 500 GB, stored for 30 days in a month, and the rate of $0.004 per GB-month, the monthly cost indeed is $60.00. Th...
18	FAIL	8.4s	Given the information provided, the calculation for the monthly cost of cloud storage appears to be accurate. Here's a breakdown of the calculation and the conclusion: 1. Daily New Data: 500 GB 2. Days in Month: 30 3. Total Stored in Month: 500 GB/day * 30 days = 15,...
19	FAIL	9.7s	Given the provided context and calculations, here's a breakdown of your cloud storage budget request: 1. Daily Data Backup: 500 GB of new data is backed up to cold storage each day. 2. Retention Period: Backups are retained for 90 days before being automatically deleted....
20	FAIL	19.0s	Based on the information provided, here's a breakdown and verification of the cloud storage budget: 1. Understanding the request: - You're asking to sign off on a budget for cloud storage costs associated with backing up data. - The key details are: * 500 GB of new data backed u...
21	FAIL	7.4s	Based on the information provided: 1. Daily data storage: 500 GB 2. Days in month: 30 3. Total monthly storage: 500 GB x 30 = 15,000 GB 4. Cost per GB-month: $0.004 5. Monthly cost: 15,000 GB x $0.004 = $60.00 The calculation seems accurate, and the total mon...
22	PASS	12.2s	Based on the provided context and calculations, here is my response: 1. The calculation of monthly cost is correct: - Daily new data: 500 GB - Days in month: 30 - Stored in month: 500 GB/day * 30 days = 15,000 GB - Monthly cost: 15,000 GB * $0.004/GB-month = $60.00 2. The total...
23	FAIL	6.8s	Based on the information provided and calculations performed, the monthly cost for storing 500 GB of new data daily in cold storage, which is kept for 90 days, equals $60.00. Given the calculations: - Daily new data: 500 GB - Days in a month: 30 - Total stored data in a month: 5...
24	FAIL	7.3s	Based on the calculations provided, the monthly cost for storing 500 GB of new data daily in cold storage, with a retention period of 90 days, is indeed $60.00. Here's a quick recap of the calculations: 1. Daily New Data: 500 GB 2. Days in Month: 30 3. *Stored in Month...

Manual

n=24, mean latency 19.9s

pass 16/24 fail 8/24 rate 67%

show sample evidence

Scroll sideways to read the full table.

sample	grade	seconds	answer excerpt
1	FAIL	25.4s	Answer: No, do not sign off on the budget as presented. The calculation underestimates the actual storage cost. Reasoning: 1. Load‑bearing assumption: Observed – The worksheet assumes that each day’s 500 GB is stored for the entire month (30 days). Check: This ig...
2	PASS	16.6s	Decision: Do not ship the budget as is. --- ### Reasoning Load‑bearing assumption: The worksheet assumes that every day’s 500 GB of new data is retained for the full 90‑day retention period. Check: - The policy says backups are kept for 90 days, then deleted. -...
3	PASS	28.6s	Answer: Do NOT sign off on the budget as presented. Reasoning: 1. Load‑bearing assumption – The worksheet assumes that every day's 500 GB is stored for the full month. Check: Cold storage keeps backups for only 90 days. After 90 days the data is deleted, so it...
4	FAIL	17.0s	Ship it? No. Answer: The budget is incorrect. The calculation treats the 500 GB of new data as if it were stored for the entire month, but the data is only kept for 90 days and then deleted. The correct monthly cost is therefore lower. Reasoning: 1. Observed:...
5	PASS	20.8s	Answer: No, do not ship the budget as-is. The calculation is incorrect. Reasoning: 1. Data Storage Duration: The 500 GB of new data is retained for 90 days, not for the entire month. - Current worksheet assumes the data accumulates to 15,000 GB per month (500 GB × 30...
6	FAIL	19.5s	Answer (to ship?): No, do not ship the budget as written. Reasoning: 1. Load‑bearing assumption – The worksheet assumes all data that lands in cold storage stays there for the full month (15 000 GB × $0.004 = $60). 2. Reality – Backups are kept **only 90 da...
7	FAIL	13.5s	Ship it. Answer: The monthly cost calculation appears correct. Reasoning: - Observed: 500 GB of new data is added each day. - Derived: Over a 30‑day month, the total amount of data stored is 500 GB × 30 = 15,000 GB. - Observed: Cold storage costs $0.004 per G...
8	PASS	22.6s	Answer: No, do not sign off on this budget as presented. Reasoning and Risk Analysis Load‑bearing assumption: The worksheet assumes that all 500 GB of daily new data are retained for the full 30‑day month, leading to a total of 15,000 GB stored each month. This is...
9	FAIL	17.0s	SHIP IT. Answer: The monthly cloud storage cost is $60. Reasoning: 1. Daily new data stored: 500 GB per day × 30 days = 15,000 GB stored over the month. 2. Cost per GB‑month: $0.004. 3. Monthly cost: 15,000 GB × $0.004 = $60. Load‑bearing assumptions: - *...
10	PASS	16.9s	Answer: No, do not sign off on the budget as presented. Reasoning: 1. Load‑bearing assumption: Observed – The finance worksheet assumes that every day’s 500 GB of new data is retained for the full 30‑day month, leading to a total of 15,000 GB stored each month. 2....
11	PASS	14.3s	Answer: No, do not sign off on the budget as presented. Reasoning: The finance worksheet makes a fundamental error in calculating the required storage. The calculation assumes that each day's 500 GB of new data is stored for the entire month, leading to a total of 15,000...
12	PASS	13.8s	Answer: Do not sign off on the budget as presented. Reasoning: 1. Load-bearing assumption: The worksheet assumes that each day's 500 GB of new data is retained for the full 90 days. This is not accurate given the 90‑day retention policy. 2. Correct calculation: -...
13	FAIL	19.9s	Answer: No, do not sign off on the budget as presented. The calculation of monthly storage cost is incorrect. Reasoning: 1. Load‑bearing assumption: *The worksheet assumes that the total amount of data stored at any point during the month is 15 000 GB (500 GB × 30 da...
14	PASS	17.2s	Answer: Do NOT sign off on this budget. Reasoning: The finance worksheet incorrectly calculates the monthly cost. The load‑bearing assumption in the worksheet is that the 500 GB of new data accumulates each day, leading to 15,000 GB stored at month‑end. In reality, the s...
15	PASS	18.0s	Answer: No, do not sign off on this budget. Reasoning and Verification: 1. Load‑bearing assumption: The worksheet assumes that all 500 GB of daily new data are retained for the full 90‑day period. This is Assumed – not observed in the problem statement. 2. **Co...
16	PASS	20.3s	Answer: No, do not sign off on the budget as presented. Reasoning: The load-bearing assumption in the finance worksheet is that 15,000 GB of data will accumulate each month (500 GB/day × 30 days). However, this calculation does not account for the **90-day retention...
17	FAIL	12.4s	SHIP IT. Answer: The monthly cloud storage cost of $60 is correct. Reasoning: 1. Load‑bearing claim – Stored in month = 500 GB × 30 days = 15,000 GB (Observed from the worksheet; the arithmetic is simple). 2. Cost calculation – *15,000 GB × $0.004 / GB‑month =...
18	FAIL	33.4s	Answer: Do not sign off on the budget as presented. The calculation overstates the monthly storage cost. Reasoning: 1. Load‑bearing assumption: Observed – The worksheet assumes all 500 GB of new data are retained for the full 90‑day retention period, i.e., it treat...
19	PASS	32.5s	Answer: No, do not sign off on the budget as presented. Reasoning and Risk Assessment 1. Load‑bearing assumption Observed: The worksheet assumes that the 500 GB of new data is stored for the full 90‑day retention period for every day of the month. Check: This ign...
20	PASS	18.9s	Answer: No, do not ship this budget as-is. Reasoning: 1. Load-bearing assumption: - Observed: The finance worksheet assumes that each day's new backup of 500 GB is stored for the entire month (30 days). - Risk: Storing each day's 500 GB for 30 days would actual...
21	PASS	15.9s	Answer: No, do not sign off on this budget as presented. The calculation is incorrect. Reasoning: 1. Load-bearing assumption: The worksheet assumes the 500 GB of new data is stored for the entire month (30 days). - Verification: The backup policy states that data...
22	PASS	24.2s	Answer: No, do not ship the budget as is. Reasoning and Verification: 1. Load-bearing Assumption Check: - The worksheet assumes that all 500 GB of new data are stored for the entire month. However, the retention policy specifies that backups are kept for only 90 days...
23	PASS	21.2s	Answer: No, do not sign off on the budget as presented. The calculation undercounts the amount of data that will actually be stored because it does not account for the fact that the 500 GB of new data is retained for 90 days before deletion, not just for the 30 days of a sin...
24	PASS	17.5s	Answer: No, do not sign off on the budget as presented. Reasoning: The finance worksheet contains a critical error in the calculation of data stored per month. The assumption that 500 GB of new data is stored each day for the full 30‑day month (500 GB × 30 days = 15,000...

What this does not settle. At n=24 per arm, this is a signal to investigate, not a settled effect size. The next check should measure cons@k, majority-vote correctness over repeated samples, and pass^k, whether every repeated sample passes. A better point estimate is not necessarily a more dependable model.

Test conditions

The model was granite-4.0-h-small-FP8, served locally as granite-small. The three-arm test used tier X, temperature 0.7 and 24 samples per arm. Control-only calibration used 16 samples per tier.