A better prompt is not a better model

Changing a prompt can recover an answer that was already there, or disrupt one that was previously correct. I tested both effects on seven local models. The useful result is not the combined score change. It is which problems each model recovered, lost or continued to miss under both prompts.

CRUXEval-O gives the model a short Python function and an input, then asks what the function returns. It tests code tracing rather than code writing. That distinction matters when I am deciding whether a model can check another model's work.

I ran 184 problems on seven downloadable, locally served checkpoints on one NVIDIA DGX Spark: Nemotron-3-Super-120B-A12B-NVFP4, gemma-4-26B-A4B-it, Ornith-1.0-35B-FP8, Holo-3.1-35B-A3B-NVFP4, Qwen3.6-35B-A3B-NVFP4, Nemotron-3-Nano-30B-A3B-NVFP4 and granite-4.0-h-small-FP8. Some emitted reasoning before their final answer; others did not. Each problem received one attempt at temperature 0, using greedy next-token selection. That setting does not by itself measure repeatability across runs.

Each model saw the same problems twice. The baseline used the original wording without a system message. The mitigation combined cleaner wording with a short model-specific system message. It targeted answer-formatting failures: returning def f(...) instead of a literal, burying the answer in prose or emitting a placeholder. It was a prompt intervention, not a change to model weights.

I graded both conditions with the same strong parser. Its job is to extract the answer already present in the response, not relax the criterion. The prediction still has to equal the expected Python literal. Recovering a correct-but-buried answer is a measurement repair, not a newly solved task.

Scores by model

Scroll sideways to read the full table.

model	baseline	mitigated	change	fixed	broken	wrong under both	both right
Nemotron-3-Super-120B-A12B-NVFP4	174 / 184	179 / 184	+5	7	2	3	172
Nemotron-3-Nano-30B-A3B-NVFP4	157 / 184	158 / 184	+1	11	10	16	147
gemma-4-26B-A4B-it	176 / 184	161 / 184	-15	2	17	6	159
Ornith-1.0-35B-FP8	168 / 184	168 / 184	0	7	7	9	161
Holo-3.1-35B-A3B-NVFP4	88 / 184	141 / 184	+53	54	1	42	87
Qwen3.6-35B-A3B-NVFP4	163 / 184	149 / 184	-14	9	23	12	140
granite-4.0-h-small-FP8	75 / 184	120 / 184	+45	55	10	54	65

Who gained, who lost, and who changed little

Higher recorded scores 2 models

Real gains. Δ > +5.

Holo-3.1-35B-A3B-NVFP4: +53 (88→141). The change recovered 54 answers and lost 1. There were still 42 wrong under both prompts. The large gain therefore coexists with a substantial set of unresolved problems.
granite-4.0-h-small-FP8: +45 (75→120). It recovered 55 and lost 10, with 54 wrong under both. That is the largest common-failure count in this comparison, despite the large score gain.

Lower recorded scores 2 models

Real losses. Δ < −5.

gemma-4-26B-A4B-it: -15 (176→161). The intervention lost 17 previously correct answers and recovered only 2. It had 6 wrong under both prompts. A formatting intervention can make an already useful model perform worse.
Qwen3.6-35B-A3B-NVFP4: -14 (163→149). The model-specific /think system message lost 23 and recovered 9, leaving 12 wrong under both. This was not a helpful intervention for this condition.

Small or balanced score changes 3 models

Wash or inside the inconsistency band (|Δ| ≤ 5).

Nemotron-3-Super-120B-A12B-NVFP4: +5 (174→179). It recovered 7 and lost 2. Its 3 common failures were the fewest here. The net gain is small enough that I would want repeated sampling before treating it as stable.
Nemotron-3-Nano-30B-A3B-NVFP4: +1 (157→158). It recovered 11 and lost 10, with 16 wrong under both. The almost unchanged total hides considerable movement between individual problems.
Ornith-1.0-35B-FP8: 0 (168→168). Seven answers were recovered and seven lost, with 9 wrong under both. An identical score does not mean identical behaviour.

These are score changes under two tested prompts, not a universal ranking of model ability.

What the columns mean

baseline: The original prompt, with no system message.
mitigated: The reworded prompt plus a short model-specific system message, intended to reduce answer-formatting failures.
change: Mitigated score minus baseline score. Read it alongside recovered and lost answers, not as a direct measure of changed capability.
fixed: Wrong under the baseline, right under the mitigation. The intervention was designed to recover formatting failures.
broken: Right under the baseline, wrong under the mitigation. These are answers the intervention cost in this draw.
wrong under both: Wrong under both tested prompts after parsing. It identifies the common failures in this comparison, not a prompt-independent limit on the model. With one sample per condition, repeated-sampling voting could recover some of them.
both right: Problems the model got right on both prompts.

The seven changes are +53, +45, +5, +1, 0, -14 and -15. They sum to +75. That total mixes two large gains, two regressions and three small net changes. It does not describe how the intervention behaved for any particular model.

Holo and Granite recovered many answers; Gemma and Qwen lost answers they had previously got right. I am interested in those mechanisms, not an average that lets one cancel the other. The wrong-under-both count is also useful because it shows what remained unresolved after either prompt had a chance to help.

In this draw, the ordering by common failures was Nemotron-3-Super-120B-A12B-NVFP4 (3), gemma-4-26B-A4B-it (6), Ornith-1.0-35B-FP8 (9), Qwen3.6-35B-A3B-NVFP4 (12), Nemotron-3-Nano-30B-A3B-NVFP4 (16), Holo-3.1-35B-A3B-NVFP4 (42), and granite-4.0-h-small-FP8 (54). The cards below let you inspect the recovered, lost and shared-failure problems behind that ordering. It is not a universal ranking of these models.

There are two important limits. Each condition used a single temperature-0 sample. cons@k, sampling a problem k times and grading the majority answer, could give a different result. The 184-problem subset was also stratified to emphasise shared failure modes, not chosen to represent the full benchmark. These scores should not be compared with full CRUXEval-O scores.

I keep parser repair separate from prompt mitigation. The parser applies the same strict literal-equality rule to both conditions. The prompt bundle changes what the model receives and can help or hurt. Both deserve to be reported; neither should be hidden inside a cleaner-looking headline score.

Inspect the problems behind each model's score

Nemotron-3-Super-120B-A12B-NVFP4

Baseline: 174 / 184
Mitigated: 179 / 184
Change: +5

Fixed: 7
Broken: 2
Wrong under both: 3
Both right: 172

Scroll sideways to read the full table.

Problem	baseline	mitigated	outcome	predicted (mitigated)	expected
sample_101	PASS	PASS	same	[-4, 4, 1, 0]	[-4, 4, 1, 0]
sample_102	PASS	PASS	same	[]	[]
sample_105	PASS	PASS	same	"Permission Is Granted"	'Permission Is Granted'
sample_106	PASS	PASS	same	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]
sample_107	PASS	PASS	same	"UA6HAJQ"	'UA6HAJQ'
sample_109	PASS	PASS	same	[9, 1, 0, 1, 1]	[9, 1, 0, 1, 1]
sample_111	PASS	PASS	same	(89, 4)	(89, 4)
sample_113	FAIL	PASS	fixed	"987YhnShAShD 93275yRgsgBgssHfBsFB"	'987YhnShAShD 93275yRgsgBgssHfBsFB'
sample_114	PASS	PASS	same	['a', '', 'b']	['a', '', 'b']
sample_116	PASS	PASS	same	{}	{}
sample_119	PASS	PASS	same	"VsNlYgLtAw"	'VsNlYgLtAw'
sample_120	PASS	PASS	same	{}	{}
sample_121	PASS	PASS	same	'1001'	'1001'
sample_123	PASS	PASS	same	[1, 2, 3, 5, 6, 8]	[1, 2, 3, 5, 6, 8]
sample_128	PASS	PASS	same	'Mmohamt'	'Mmohamt'
sample_129	PASS	PASS	same	[28, 19, 12, 6]	[28, 19, 12, 6]
sample_130	PASS	PASS	same	'h=l'	'h=l'
sample_132	PASS	PASS	same	'abc'	'abc'
sample_133	PASS	PASS	same	[7, 1, 2]	[7, 1, 2]
sample_135	PASS	PASS	same	['Russia', 'Kazakhstan']	['Russia', 'Kazakhstan']
sample_136	FAIL	FAIL	both	" a \n bc \n \n d \n ef "	' a \n bc \n \n d \n ef '
sample_137	PASS	PASS	same	[]	[]
sample_139	PASS	PASS	same	'no'	'no'
sample_140	PASS	PASS	same	'Hey'	'Hey'
sample_141	PASS	PASS	same	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]
sample_144	PASS	PASS	same	[]	[]
sample_146	PASS	PASS	same	[1, 2, 3, 4, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 6, 7, 8, 9, 10]
sample_147	PASS	PASS	same	[1, 1, 1]	[1, 1, 1]
sample_148	PASS	PASS	same	"2io 12 tfiqr.-"	'2io 12 tfiqr.-'
sample_150	PASS	PASS	same	[-2, 4, -4]	[-2, 4, -4]
sample_154	PASS	PASS	same	'* There Hello'	'* There Hello'
sample_158	PASS	PASS	same	[6,4,-2,6,4,-2]	[6, 4, -2, 6, 4, -2]
sample_159	PASS	PASS	same	'mgItr'	'mgItr'
sample_162	PASS	PASS	same	'СBISHOPSWIFT'	'СBISHOPSWIFT'
sample_164	PASS	PASS	same	[0, 1, 3]	[0, 1, 3]
sample_166	PASS	PASS	same	{}	{}
sample_167	PASS	PASS	same	'aaQwQwQwbbQwQwQwccQwQwQwde'	'aaQwQwQwbbQwQwQwccQwQwQwde'
sample_169	PASS	PASS	same	"++++taole++++"	'++++taole++++'
sample_171	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_172	PASS	PASS	same	[]	[]
sample_173	PASS	PASS	same	[4, 8, 6, 8, 5]	[4, 8, 6, 8, 5]
sample_174	PASS	PASS	same	[1, 3, 2]	[1, 3, 2]
sample_175	PASS	PASS	same	' '	' '
sample_177	PASS	PASS	same	'HEy Dude tHIs $Nd^ &*&tHiS@#'	'HEy Dude tHIs $Nd^ &*&tHiS@#'
sample_178	PASS	PASS	same	[2, 2, 2]	[2, 2, 2]
sample_179	PASS	PASS	same	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]
sample_180	PASS	PASS	same	[-1, -2, -6, 8, 8]	[-1, -2, -6, 8, 8]
sample_181	PASS	PASS	same	["3291223", 7]	['3291223', 7]
sample_182	PASS	PASS	same	[('a', 2), ('b', 1)]	[('a', 2), ('b', 1)]
sample_183	PASS	PASS	same	["echo"]	['echo']
sample_184	PASS	PASS	same	[1,2]	[1, 2]
sample_185	PASS	PASS	same	[11, 14, 7, 12, 9, 16]	[11, 14, 7, 12, 9, 16]
sample_186	PASS	PASS	same	'pvtso'	'pvtso'
sample_188	PASS	PASS	same	["a"]	['a']
sample_189	PASS	PASS	same	"{{{{}}}}"	'{{{{}}}}'
sample_193	PASS	PASS	same	'1:1'	'1:1'
sample_194	PASS	PASS	same	[[], [5, 6, 2, 3], [1, 9, 5, 6]]	[[], [5, 6, 2, 3], [1, 9, 5, 6]]
sample_201	PASS	PASS	same	'641524'	'641524'
sample_202	PASS	PASS	same	[15, 15]	[15, 15]
sample_203	PASS	PASS	same	{}	{}
sample_204	PASS	PASS	same	['m', 'a']	['m', 'a']
sample_206	PASS	PASS	same	'h e l l o w o r l d!'	'h e l l o w o r l d!'
sample_208	PASS	PASS	same	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']
sample_212	PASS	PASS	same	[1, -9, 7, 2, 6, -3, 3]	[1, -9, 7, 2, 6, -3, 3]
sample_215	PASS	PASS	same	')'	')'
sample_224	PASS	PASS	same	{}	{}
sample_226	PASS	PASS	same	[1, 3, 3]	[1, 3, 3]
sample_227	PASS	PASS	same	'Manolo'	'Manolo'
sample_228	FAIL	FAIL	both		'llthh#saflapkphtswp'
sample_229	FAIL	PASS	fixed	['9', 'm', 'm', 'A', '1', '0', 'L', 'k']	['9', 'm', 'm', 'A', '1', '0', 'L', 'k']
sample_233	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_235	PASS	PASS	same	[]	[]
sample_236	PASS	PASS	same	'ac8qk6qk6'	'ac8qk6qk6'
sample_238	PASS	PASS	same	[1, 9, 4]	[1, 9, 4]
sample_239	PASS	PASS	same	'1co'	'1co'
sample_241	PASS	PASS	same	'CW'	'CW'
sample_242	PASS	PASS	same	'udhv zcvi nhtnfyd :erwuyawa pun'	'udhv zcvi nhtnfyd :erwuyawa pun'
sample_245	PASS	PASS	same	[]	[]
sample_247	PASS	PASS	same	"yes"	'yes'
sample_248	PASS	PASS	same	[666]	[666]
sample_251	PASS	PASS	same	"Marie;Nelson;Oscar;+;3;5;3"	'Marie;Nelson;Oscar;+;3;5;3'
sample_254	PASS	PASS	same	'lwwer case'	'lwwer case'
sample_255	PASS	PASS	same	'w'	'w'
sample_257	PASS	PASS	same	[['Hello World', '"I am String"']]	[['Hello World', '"I am String"']]
sample_258	PASS	PASS	same	[1, 2, 7, 3, 9]	[1, 2, 7, 3, 9]
sample_259	PASS	PASS	same	'RES'	'RES'
sample_260	PASS	PASS	same	[1, 2, 3, 4, 6, 5]	[1, 2, 3, 4, 6, 5]
sample_261	PASS	PASS	same	([], [12, 516, 5, 214, 51])	([], [12, 516, 5, 214, 51])
sample_262	FAIL	PASS	fixed	"BA"	'BA'
sample_263	PASS	PASS	same	["gloss", "banana", "barn", "lawn"]	['gloss', 'banana', 'barn', 'lawn']
sample_266	PASS	PASS	same	[2, 3, 3, 4, 6, -2]	[2, 3, 3, 4, 6, -2]
sample_268	PASS	PASS	same	'h / g r a t e f u l k'	'h / g r a t e f u l k'
sample_269	PASS	PASS	same	['0', 2]	['0', 2]
sample_270	PASS	PASS	same	{}	{}
sample_271	PASS	PASS	same	'uufh'	'uufh'
sample_272	PASS	PASS	same	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 2, 6, 0, 6, 6]	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 2, 6, 0, 6, 6]
sample_273	PASS	PASS	same	'NET'	'NET'
sample_276	PASS	PASS	same	[0]	[0]
sample_277	PASS	PASS	same	[4, 3, 2, 1]	[4, 3, 2, 1]
sample_279	PASS	PASS	same	''	''
sample_284	PASS	PASS	same	''	''
sample_286	PASS	PASS	same	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]
sample_287	PASS	PASS	same	'pinneaple'	'pinneaple'
sample_288	PASS	PASS	same	[(1, 3), (4, 555)]	[(1, 3), (4, 555)]
sample_290	PASS	PASS	same	'ABIXAAAILY'	'ABIXAAAILY'
sample_293	PASS	PASS	same	'no'	'no'
sample_295	PASS	PASS	same	['pear', 'banana', 'pear']	['pear', 'banana', 'pear']
sample_298	PASS	PASS	same	'DST VAVF N DMV DFVM GAMCU DGCVB.'	'DST VAVF N DMV DFVM GAMCU DGCVB.'
sample_300	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_301	PASS	PASS	same	[0, 6, 2, -1, -2, 6, 6, -2, -2, -2, -2, -2, -2]	[0, 6, 2, -1, -2, 6, 6, -2, -2, -2, -2, -2, -2]
sample_303	FAIL	FAIL	both	'mJk l b n'	'mJklbn'
sample_305	PASS	PASS	same	"o hoseto"	'o hoseto'
sample_306	PASS	PASS	same	[0, 6, 1, 2, 0]	[0, 6, 1, 2, 0]
sample_307	PASS	PASS	same	"px\|\|\|\|\|cznyf"	'px\|\|\|\|\|cznyf'
sample_310	PASS	PASS	same	'__1.00r__j_a6__6'	'__1.00r__j_a6__6'
sample_311	PASS	PASS	same	'no'	'no'
sample_316	PASS	PASS	same	'\| i am your father \|'	'\| i am your father \|'
sample_320	PASS	PASS	same	'usAr'	'usAr'
sample_322	PASS	PASS	same	['lsi', 's', 't', 't', 'd']	['lsi', 's', 't', 't', 'd']
sample_324	PASS	PASS	same	[]	[]
sample_327	PASS	PASS	same	[-3, 1, 7, -1]	[-3, 1, 7, -1]
sample_328	PASS	PASS	same	[1, 2, 3, 1, 2, 3]	[1, 2, 3, 1, 2, 3]
sample_330	PASS	PASS	same	" 4 2 "	' 4 2 '
sample_332	PASS	PASS	same	[]	[]
sample_335	PASS	PASS	same	"sjbrfqmw"	'sjbrfqmw'
sample_337	FAIL	PASS	fixed	"LL"	'LL'
sample_340	PASS	PASS	same	' DEGHIVjkptx'	' DEGHIVjkptx'
sample_341	PASS	PASS	same	{}	{}
sample_343	PASS	PASS	same	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]
sample_344	PASS	PASS	same	[6, 4, 2, 8, 15]	[6, 4, 2, 8, 15]
sample_345	PASS	PASS	same	('mv', 'ml')	('mv', 'ml')
sample_347	PASS	FAIL	broken	'hhhhhzczw'	'hhhhhzcw'
sample_350	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_356	PASS	PASS	same	[2, 1]	[2, 1]
sample_357	PASS	PASS	same	'werc'	'werc'
sample_358	PASS	PASS	same	'tr'	'tr'
sample_359	PASS	PASS	same	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']
sample_360	PASS	PASS	same	'g'	'g'
sample_363	PASS	PASS	same	[1]	[1]
sample_364	PASS	PASS	same	[[3, False], [1, True]]	[[3, False], [1, True]]
sample_366	PASS	PASS	same	''	''
sample_367	PASS	PASS	same	[6, 2, 1, 1, 4, 1]	[6, 2, 1, 1, 4, 1]
sample_368	PASS	PASS	same	'4327 00004327 000004327 4327 0004327 4327'	'4327 00004327 000004327 4327 0004327 4327'
sample_372	PASS	PASS	same	['']	['']
sample_428	PASS	PASS	same	[]	[]
sample_429	PASS	PASS	same	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]
sample_431	PASS	PASS	same	[]	[]
sample_435	PASS	PASS	same	''	''
sample_445	PASS	PASS	same	'Carrot, Banana, and Strawberry'	'Carrot, Banana, and Strawberry'
sample_446	PASS	PASS	same	[]	[]
sample_457	PASS	PASS	same	[]	[]
sample_480	PASS	PASS	same	''	''
sample_483	PASS	PASS	same	' '	' '
sample_503	PASS	PASS	same	[]	[]
sample_510	PASS	PASS	same	'Wy'	'Wy'
sample_513	PASS	PASS	same	[]	[]
sample_516	PASS	PASS	same	[]	[]
sample_522	PASS	PASS	same	[]	[]
sample_572	PASS	PASS	same	[(2, 10), (3, 1), ((3, 1), None), ((2, 10), None)]	[(2, 10), (3, 1), ((3, 1), None), ((2, 10), None)]
sample_579	PASS	PASS	same	''	''
sample_583	PASS	PASS	same	"t\nZA\nA"	't\nZA\nA'
sample_593	PASS	PASS	same	[]	[]
sample_600	PASS	PASS	same	[]	[]
sample_601	FAIL	PASS	fixed	'ccccc sssss hhhhh AAAAA rrrrr ppppp'	'ccccc sssss hhhhh AAAAA rrrrr ppppp'
sample_623	PASS	PASS	same	'HI~'	'HI~'
sample_663	PASS	PASS	same	[]	[]
sample_672	PASS	PASS	same	"1zd"	'1zd'
sample_673	PASS	PASS	same	"cA"	'cA'
sample_674	PASS	PASS	same	'qq'	'qq'
sample_682	PASS	PASS	same	'hy'	'hy'
sample_687	FAIL	PASS	fixed	'R:j:u:g: :z:u:f:E:rjug nzufe'	'R:j:u:g: :z:u:f:E:rjug nzufe'
sample_692	PASS	PASS	same	[]	[]
sample_705	PASS	PASS	same	[]	[]
sample_714	PASS	PASS	same	[]	[]
sample_716	PASS	PASS	same	[]	[]
sample_728	PASS	PASS	same	''	''
sample_735	FAIL	PASS	fixed	'Abb'	'Abb'
sample_759	PASS	PASS	same	[]	[]
sample_761	PASS	PASS	same	[]	[]
sample_767	PASS	PASS	same	'-'	'-'
sample_769	PASS	PASS	same	'AKa?RIU'	'AKa?RIU'
sample_780	PASS	PASS	same	"2"	'2'
sample_787	PASS	PASS	same	'Xzd'	'Xzd'
sample_797	PASS	FAIL	broken		[('a', 1), ('b', 2), ('c', 3)]

Nemotron-3-Nano-30B-A3B-NVFP4

Baseline: 157 / 184
Mitigated: 158 / 184
Change: +1

Fixed: 11
Broken: 10
Wrong under both: 16
Both right: 147

Scroll sideways to read the full table.

Problem	baseline	mitigated	outcome	predicted (mitigated)	expected
sample_101	PASS	PASS	same	[-4, 4, 1, 0]	[-4, 4, 1, 0]
sample_102	PASS	PASS	same	[]	[]
sample_105	PASS	FAIL	broken		'Permission Is Granted'
sample_106	PASS	PASS	same	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]
sample_107	PASS	PASS	same	"UA6HAJQ"	'UA6HAJQ'
sample_109	PASS	PASS	same	[9, 1, 0, 1, 1]	[9, 1, 0, 1, 1]
sample_111	PASS	PASS	same	(89, 4)	(89, 4)
sample_113	FAIL	FAIL	both		'987YhnShAShD 93275yRgsgBgssHfBsFB'
sample_114	PASS	PASS	same	['a', '', 'b']	['a', '', 'b']
sample_116	PASS	PASS	same	{}	{}
sample_119	FAIL	PASS	fixed	"VsNlYgLtAw"	'VsNlYgLtAw'
sample_120	PASS	PASS	same	{}	{}
sample_121	PASS	PASS	same	'1001'	'1001'
sample_123	PASS	PASS	same	[1, 2, 3, 5, 6, 8]	[1, 2, 3, 5, 6, 8]
sample_128	FAIL	FAIL	both		'Mmohamt'
sample_129	FAIL	FAIL	both	[28, 19, 11, 6]	[28, 19, 12, 6]
sample_130	PASS	PASS	same	"h=l"	'h=l'
sample_132	PASS	PASS	same	'abc'	'abc'
sample_133	PASS	PASS	same	[7, 1, 2]	[7, 1, 2]
sample_135	PASS	PASS	same	['Russia', 'Kazakhstan']	['Russia', 'Kazakhstan']
sample_136	FAIL	FAIL	both	" a \n bc \n \n d \n ef "	' a \n bc \n \n d \n ef '
sample_137	PASS	PASS	same	[]	[]
sample_139	PASS	PASS	same	'no'	'no'
sample_140	PASS	PASS	same	'Hey'	'Hey'
sample_141	PASS	PASS	same	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]
sample_144	PASS	PASS	same	[]	[]
sample_146	PASS	PASS	same	[1, 2, 3, 4, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 6, 7, 8, 9, 10]
sample_147	PASS	PASS	same	[1, 1, 1]	[1, 1, 1]
sample_148	FAIL	PASS	fixed	"2io 12 tfiqr.-"	'2io 12 tfiqr.-'
sample_150	FAIL	PASS	fixed	[-2, 4, -4]	[-2, 4, -4]
sample_154	FAIL	PASS	fixed	'* There Hello'	'* There Hello'
sample_158	PASS	PASS	same	[6, 4, -2, 6, 4, -2]	[6, 4, -2, 6, 4, -2]
sample_159	FAIL	PASS	fixed	'mgItr'	'mgItr'
sample_162	FAIL	FAIL	both	'СBISHTERSWIFT'	'СBISHOPSWIFT'
sample_164	PASS	PASS	same	[0, 1, 3]	[0, 1, 3]
sample_166	PASS	PASS	same	{}	{}
sample_167	FAIL	PASS	fixed	'aaQwQwQwbbQwQwQwccQwQwQwde'	'aaQwQwQwbbQwQwQwccQwQwQwde'
sample_169	PASS	PASS	same	'++++taole++++'	'++++taole++++'
sample_171	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_172	PASS	PASS	same	[]	[]
sample_173	PASS	PASS	same	[4, 8, 6, 8, 5]	[4, 8, 6, 8, 5]
sample_174	PASS	PASS	same	[1, 3, 2]	[1, 3, 2]
sample_175	PASS	PASS	same	' '	' '
sample_177	PASS	FAIL	broken	"HEy DuDe tHIs $Nd^ &*&tHiS@#"	'HEy Dude tHIs $Nd^ &*&tHiS@#'
sample_178	PASS	FAIL	broken	[2, 2, 2, 2]	[2, 2, 2]
sample_179	PASS	PASS	same	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]
sample_180	PASS	PASS	same	[-1, -2, -6, 8, 8]	[-1, -2, -6, 8, 8]
sample_181	PASS	PASS	same	["3291223", 7]	['3291223', 7]
sample_182	PASS	PASS	same	[('a', 2), ('b', 1)]	[('a', 2), ('b', 1)]
sample_183	PASS	PASS	same	["echo"]	['echo']
sample_184	PASS	PASS	same	[1, 2]	[1, 2]
sample_185	PASS	PASS	same	[11, 14, 7, 12, 9, 16]	[11, 14, 7, 12, 9, 16]
sample_186	PASS	PASS	same	'pvtso'	'pvtso'
sample_188	PASS	PASS	same	["a"]	['a']
sample_189	PASS	PASS	same	"{{{{}}}}"	'{{{{}}}}'
sample_193	PASS	PASS	same	'1:1'	'1:1'
sample_194	PASS	PASS	same	[[], [5, 6, 2, 3], [1, 9, 5, 6]]	[[], [5, 6, 2, 3], [1, 9, 5, 6]]
sample_201	PASS	PASS	same	"641524"	'641524'
sample_202	PASS	PASS	same	[15, 15]	[15, 15]
sample_203	PASS	PASS	same	{}	{}
sample_204	PASS	PASS	same	['m', 'a']	['m', 'a']
sample_206	PASS	FAIL	broken		'h e l l o w o r l d!'
sample_208	PASS	PASS	same	['c','a','t','d',' ','d','e','e']	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']
sample_212	PASS	PASS	same	[1, -9, 7, 2, 6, -3, 3]	[1, -9, 7, 2, 6, -3, 3]
sample_215	PASS	FAIL	broken		')'
sample_224	PASS	PASS	same	{}	{}
sample_226	PASS	PASS	same	[1, 3, 3]	[1, 3, 3]
sample_227	PASS	PASS	same	'Manolo'	'Manolo'
sample_228	FAIL	FAIL	both		'llthh#saflapkphtswp'
sample_229	FAIL	FAIL	both	['mA', 2, '9m', '10K', 2, 'Lk', 2]	['9', 'm', 'm', 'A', '1', '0', 'L', 'k']
sample_233	FAIL	FAIL	both	[2, 1, 3]	[1, 2, 3]
sample_235	PASS	PASS	same	[]	[]
sample_236	PASS	PASS	same	'ac8qk6qk6'	'ac8qk6qk6'
sample_238	PASS	PASS	same	[1, 9, 4]	[1, 9, 4]
sample_239	FAIL	FAIL	both	'1cos'	'1co'
sample_241	PASS	PASS	same	"CW"	'CW'
sample_242	PASS	PASS	same	"udhv zcvi nhtnfyd :erwuyawa pun"	'udhv zcvi nhtnfyd :erwuyawa pun'
sample_245	PASS	FAIL	broken	['a', 'c', 'd', 'e', 'p', 'r', 's', 'u']	[]
sample_247	PASS	PASS	same	"yes"	'yes'
sample_248	PASS	PASS	same	[666]	[666]
sample_251	PASS	PASS	same	"Marie;Nelson;Oscar;+;3;5;3"	'Marie;Nelson;Oscar;+;3;5;3'
sample_254	FAIL	FAIL	both	"lower case"	'lwwer case'
sample_255	PASS	PASS	same	'w'	'w'
sample_257	PASS	PASS	same	[['Hello World', '"I am String"']]	[['Hello World', '"I am String"']]
sample_258	PASS	FAIL	broken	[1, 7, 2, 3, 9]	[1, 2, 7, 3, 9]
sample_259	PASS	PASS	same	"RES"	'RES'
sample_260	PASS	PASS	same	[1, 2, 3, 4, 6, 5]	[1, 2, 3, 4, 6, 5]
sample_261	PASS	PASS	same	( [], [12, 516, 5, 214, 51] )	([], [12, 516, 5, 214, 51])
sample_262	PASS	PASS	same	"BA"	'BA'
sample_263	PASS	PASS	same	["gloss", "banana", "barn", "lawn"]	['gloss', 'banana', 'barn', 'lawn']
sample_266	PASS	PASS	same	[2, 3, 3, 4, 6, -2]	[2, 3, 3, 4, 6, -2]
sample_268	PASS	PASS	same	'h / g r a t e f u l k'	'h / g r a t e f u l k'
sample_269	PASS	PASS	same	['0', 2]	['0', 2]
sample_270	PASS	PASS	same	{}	{}
sample_271	FAIL	FAIL	both	'ufh'	'uufh'
sample_272	PASS	PASS	same	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 2, 6, 0, 6, 6]	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 2, 6, 0, 6, 6]
sample_273	PASS	PASS	same	'NET'	'NET'
sample_276	PASS	PASS	same	[0]	[0]
sample_277	PASS	PASS	same	[4, 3, 2, 1]	[4, 3, 2, 1]
sample_279	PASS	PASS	same	''	''
sample_284	PASS	PASS	same	''	''
sample_286	PASS	PASS	same	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]
sample_287	PASS	PASS	same	'pinneaple'	'pinneaple'
sample_288	PASS	PASS	same	[(1, 3), (4, 555)]	[(1, 3), (4, 555)]
sample_290	PASS	PASS	same	'ABIXAAAILY'	'ABIXAAAILY'
sample_293	PASS	PASS	same	"no"	'no'
sample_295	PASS	PASS	same	['pear', 'banana', 'pear']	['pear', 'banana', 'pear']
sample_298	PASS	PASS	same	"DST VAVF N DMV DFVM GAMCU DGCVB."	'DST VAVF N DMV DFVM GAMCU DGCVB.'
sample_300	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_301	PASS	PASS	same	[0, 6, 2, -1, -2, 6, 6, -2, -2, -2, -2, -2, -2]	[0, 6, 2, -1, -2, 6, 6, -2, -2, -2, -2, -2, -2]
sample_303	PASS	PASS	same	"mJklbn"	'mJklbn'
sample_305	FAIL	FAIL	both	"o hosto"	'o hoseto'
sample_306	PASS	PASS	same	[0, 6, 1, 2, 0]	[0, 6, 1, 2, 0]
sample_307	FAIL	FAIL	both	"px\|\|\|\|\|czynf"	'px\|\|\|\|\|cznyf'
sample_310	PASS	PASS	same	"__1.00r__j_a6__6"	'__1.00r__j_a6__6'
sample_311	PASS	PASS	same	'no'	'no'
sample_316	PASS	PASS	same	'\| i am your father \|'	'\| i am your father \|'
sample_320	PASS	PASS	same	"usAr"	'usAr'
sample_322	PASS	PASS	same	['lsi', 's', 't', 't', 'd']	['lsi', 's', 't', 't', 'd']
sample_324	PASS	PASS	same	[]	[]
sample_327	PASS	PASS	same	[-3, 1, 7, -1]	[-3, 1, 7, -1]
sample_328	PASS	PASS	same	[1, 2, 3, 1, 2, 3]	[1, 2, 3, 1, 2, 3]
sample_330	PASS	PASS	same	" 4 2 "	' 4 2 '
sample_332	PASS	PASS	same	[]	[]
sample_335	PASS	PASS	same	'sjbrfqmw'	'sjbrfqmw'
sample_337	PASS	PASS	same	"LL"	'LL'
sample_340	PASS	PASS	same	" DEGHIVjkptx"	' DEGHIVjkptx'
sample_341	PASS	PASS	same	{}	{}
sample_343	PASS	PASS	same	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]
sample_344	PASS	PASS	same	[6, 4, 2, 8, 15]	[6, 4, 2, 8, 15]
sample_345	PASS	PASS	same	('mv','ml')	('mv', 'ml')
sample_347	PASS	PASS	same	'hhhhhzcw'	'hhhhhzcw'
sample_350	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_356	PASS	PASS	same	[2, 1]	[2, 1]
sample_357	PASS	PASS	same	'werc'	'werc'
sample_358	PASS	PASS	same	'tr'	'tr'
sample_359	PASS	PASS	same	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']
sample_360	PASS	PASS	same	'g'	'g'
sample_363	PASS	PASS	same	[1]	[1]
sample_364	PASS	PASS	same	[[3, False], [1, True]]	[[3, False], [1, True]]
sample_366	FAIL	PASS	fixed	''	''
sample_367	PASS	PASS	same	[6, 2, 1, 1, 4, 1]	[6, 2, 1, 1, 4, 1]
sample_368	FAIL	PASS	fixed	"4327 00004327 000004327 4327 0004327 4327"	'4327 00004327 000004327 4327 0004327 4327'
sample_372	PASS	FAIL	broken	[0]	['']
sample_428	PASS	PASS	same	[]	[]
sample_429	PASS	FAIL	broken	[(87.29, 3), ('abc', 2), ('defghi', 2), (5, 1)]	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]
sample_431	PASS	PASS	same	[]	[]
sample_435	PASS	PASS	same	''	''
sample_445	PASS	PASS	same	"Carrot, Banana, and Strawberry"	'Carrot, Banana, and Strawberry'
sample_446	PASS	PASS	same	[]	[]
sample_457	PASS	PASS	same	[]	[]
sample_480	PASS	PASS	same	''	''
sample_483	PASS	PASS	same	' '	' '
sample_503	PASS	PASS	same	[]	[]
sample_510	PASS	PASS	same	'Wy'	'Wy'
sample_513	PASS	PASS	same	[]	[]
sample_516	PASS	PASS	same	[]	[]
sample_522	PASS	PASS	same	[]	[]
sample_572	FAIL	PASS	fixed	[(2, 10), (3, 1), ((3, 1), None), ((2, 10), None)]	[(2, 10), (3, 1), ((3, 1), None), ((2, 10), None)]
sample_579	PASS	PASS	same	''	''
sample_583	PASS	PASS	same	"t\nZA\nA"	't\nZA\nA'
sample_593	PASS	PASS	same	[]	[]
sample_600	PASS	PASS	same	[]	[]
sample_601	FAIL	FAIL	both		'ccccc sssss hhhhh AAAAA rrrrr ppppp'
sample_623	FAIL	PASS	fixed	'HI~'	'HI~'
sample_663	PASS	PASS	same	[]	[]
sample_672	PASS	PASS	same	"1zd"	'1zd'
sample_673	PASS	PASS	same	"cA"	'cA'
sample_674	PASS	PASS	same	"qq"	'qq'
sample_682	PASS	PASS	same	'hy'	'hy'
sample_687	FAIL	FAIL	both		'R:j:u:g: :z:u:f:E:rjug nzufe'
sample_692	PASS	PASS	same	[]	[]
sample_705	PASS	PASS	same	[]	[]
sample_714	PASS	PASS	same	[]	[]
sample_716	PASS	PASS	same	[]	[]
sample_728	PASS	PASS	same	''	''
sample_735	FAIL	FAIL	both		'Abb'
sample_759	PASS	PASS	same	[]	[]
sample_761	PASS	PASS	same	[]	[]
sample_767	PASS	FAIL	broken		'-'
sample_769	PASS	PASS	same	"AKa?RIU"	'AKa?RIU'
sample_780	FAIL	PASS	fixed	'2'	'2'
sample_787	PASS	PASS	same	'Xzd'	'Xzd'
sample_797	PASS	PASS	same	[('a', 1), ('b', 2), ('c', 3)]	[('a', 1), ('b', 2), ('c', 3)]

gemma-4-26B-A4B-it

Baseline: 176 / 184
Mitigated: 161 / 184
Change: -15

Fixed: 2
Broken: 17
Wrong under both: 6
Both right: 159

Scroll sideways to read the full table.

Problem	baseline	mitigated	outcome	predicted (mitigated)	expected
sample_101	PASS	PASS	same	[-4, 4, 1, 0]	[-4, 4, 1, 0]
sample_102	PASS	PASS	same	[]	[]
sample_105	PASS	PASS	same	"Permission Is Granted"	'Permission Is Granted'
sample_106	PASS	PASS	same	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]
sample_107	PASS	PASS	same	'UA6HAJQ'	'UA6HAJQ'
sample_109	PASS	PASS	same	[9, 1, 0, 1, 1]	[9, 1, 0, 1, 1]
sample_111	PASS	PASS	same	(89, 4)	(89, 4)
sample_113	PASS	FAIL	broken	'987YhnShAShD 93275yRgSgBgs sHfBsFB'	'987YhnShAShD 93275yRgsgBgssHfBsFB'
sample_114	PASS	FAIL	broken	['a', '.', 'b']	['a', '', 'b']
sample_116	PASS	PASS	same	{}	{}
sample_119	PASS	PASS	same	'VsNlYgLtAw'	'VsNlYgLtAw'
sample_120	PASS	PASS	same	{}	{}
sample_121	PASS	PASS	same	'1001'	'1001'
sample_123	PASS	PASS	same	[1, 2, 3, 5, 6, 8]	[1, 2, 3, 5, 6, 8]
sample_128	FAIL	FAIL	both	'Mmothamt'	'Mmohamt'
sample_129	FAIL	FAIL	both	[28, 20, 18, 13, 6]	[28, 19, 12, 6]
sample_130	PASS	PASS	same	'h=l'	'h=l'
sample_132	PASS	PASS	same	'abc'	'abc'
sample_133	PASS	PASS	same	[7, 1, 2]	[7, 1, 2]
sample_135	PASS	PASS	same	['Russia', 'Kazakhstan']	['Russia', 'Kazakhstan']
sample_136	FAIL	FAIL	both	' a \n bc \n \n d \n ef '	' a \n bc \n \n d \n ef '
sample_137	PASS	PASS	same	[]	[]
sample_139	PASS	PASS	same	'no'	'no'
sample_140	PASS	PASS	same	'Hey'	'Hey'
sample_141	PASS	PASS	same	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]
sample_144	PASS	PASS	same	[]	[]
sample_146	PASS	PASS	same	[1, 2, 3, 4, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 6, 7, 8, 9, 10]
sample_147	PASS	PASS	same	[1, 1, 1]	[1, 1, 1]
sample_148	PASS	PASS	same	'2io 12 tfiqr.-'	'2io 12 tfiqr.-'
sample_150	PASS	PASS	same	[-2, 4, -4]	[-2, 4, -4]
sample_154	PASS	FAIL	broken		'* There Hello'
sample_158	PASS	PASS	same	[6, 4, -2, 6, 4, -2]	[6, 4, -2, 6, 4, -2]
sample_159	PASS	PASS	same	'mgItr'	'mgItr'
sample_162	PASS	PASS	same	'СBISHOPSWIFT'	'СBISHOPSWIFT'
sample_164	PASS	PASS	same	[0, 1, 3]	[0, 1, 3]
sample_166	PASS	PASS	same	{}	{}
sample_167	PASS	PASS	same	'aaQwQwQwbbQwQwQwccQwQwQwde'	'aaQwQwQwbbQwQwQwccQwQwQwde'
sample_169	PASS	FAIL	broken		'++++taole++++'
sample_171	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_172	PASS	PASS	same	[]	[]
sample_173	PASS	PASS	same	[4, 8, 6, 8, 5]	[4, 8, 6, 8, 5]
sample_174	PASS	PASS	same	[1, 3, 2]	[1, 3, 2]
sample_175	PASS	PASS	same	' '	' '
sample_177	PASS	PASS	same	'HEy Dude tHIs $Nd^ &*&tHiS@#'	'HEy Dude tHIs $Nd^ &*&tHiS@#'
sample_178	PASS	PASS	same	[2, 2, 2]	[2, 2, 2]
sample_179	PASS	PASS	same	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]
sample_180	PASS	PASS	same	[-1, -2, -6, 8, 8]	[-1, -2, -6, 8, 8]
sample_181	PASS	PASS	same	['3291223', 7]	['3291223', 7]
sample_182	PASS	PASS	same	[('a', 2), ('b', 1)]	[('a', 2), ('b', 1)]
sample_183	PASS	PASS	same	['echo']	['echo']
sample_184	PASS	PASS	same	[1, 2]	[1, 2]
sample_185	PASS	PASS	same	[11, 14, 7, 12, 9, 16]	[11, 14, 7, 12, 9, 16]
sample_186	PASS	PASS	same	'pvtso'	'pvtso'
sample_188	PASS	FAIL	broken		['a']
sample_189	PASS	PASS	same	"{{{{}}}}"	'{{{{}}}}'
sample_193	PASS	PASS	same	'1:1'	'1:1'
sample_194	PASS	PASS	same	[[], [5, 6, 2, 3], [1, 9, 5, 6]]	[[], [5, 6, 2, 3], [1, 9, 5, 6]]
sample_201	PASS	PASS	same	'641524'	'641524'
sample_202	PASS	PASS	same	[15, 15]	[15, 15]
sample_203	PASS	PASS	same	{}	{}
sample_204	PASS	PASS	same	['m', 'a']	['m', 'a']
sample_206	PASS	PASS	same	'h e l l o w o r l d!'	'h e l l o w o r l d!'
sample_208	PASS	PASS	same	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']
sample_212	PASS	PASS	same	[1, -9, 7, 2, 6, -3, 3]	[1, -9, 7, 2, 6, -3, 3]
sample_215	PASS	FAIL	broken		')'
sample_224	PASS	PASS	same	{}	{}
sample_226	PASS	PASS	same	[1, 3, 3]	[1, 3, 3]
sample_227	PASS	PASS	same	'Manolo'	'Manolo'
sample_228	PASS	PASS	same	'llthh#saflapkphtswp'	'llthh#saflapkphtswp'
sample_229	FAIL	FAIL	both	['mA', 2, '9m', '10K', 2, 'Lk', 2]	['9', 'm', 'm', 'A', '1', '0', 'L', 'k']
sample_233	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_235	PASS	PASS	same	[]	[]
sample_236	PASS	PASS	same	'ac8qk6qk6'	'ac8qk6qk6'
sample_238	PASS	PASS	same	[1, 9, 4]	[1, 9, 4]
sample_239	PASS	PASS	same	'1co'	'1co'
sample_241	PASS	PASS	same	'CW'	'CW'
sample_242	PASS	PASS	same	'udhv zcvi nhtnfyd :erwuyawa pun'	'udhv zcvi nhtnfyd :erwuyawa pun'
sample_245	FAIL	FAIL	both	['u', 'p', 'e', 'r', 'c', 'a', 's', 'd', 'v', 'z']	[]
sample_247	PASS	PASS	same	'yes'	'yes'
sample_248	PASS	PASS	same	[666]	[666]
sample_251	PASS	FAIL	broken		'Marie;Nelson;Oscar;+;3;5;3'
sample_254	PASS	PASS	same	'lwwer case'	'lwwer case'
sample_255	PASS	PASS	same	'w'	'w'
sample_257	PASS	FAIL	broken	[[['Hello World', '"I am String"']]]	[['Hello World', '"I am String"']]
sample_258	PASS	PASS	same	[1, 2, 7, 3, 9]	[1, 2, 7, 3, 9]
sample_259	PASS	PASS	same	'RES'	'RES'
sample_260	PASS	PASS	same	[1, 2, 3, 4, 6, 5]	[1, 2, 3, 4, 6, 5]
sample_261	PASS	PASS	same	([], [12, 516, 5, 214, 51])	([], [12, 516, 5, 214, 51])
sample_262	PASS	PASS	same	"BA"	'BA'
sample_263	PASS	PASS	same	['gloss', 'banana', 'barn', 'lawn']	['gloss', 'banana', 'barn', 'lawn']
sample_266	PASS	PASS	same	[2, 3, 3, 4, 6, -2]	[2, 3, 3, 4, 6, -2]
sample_268	PASS	PASS	same	'h / g r a t e f u l k'	'h / g r a t e f u l k'
sample_269	PASS	PASS	same	['0', 2]	['0', 2]
sample_270	PASS	PASS	same	{}	{}
sample_271	PASS	PASS	same	'uufh'	'uufh'
sample_272	FAIL	FAIL	both	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 1, 6, 0, 6, 6]	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 2, 6, 0, 6, 6]
sample_273	PASS	PASS	same	'NET'	'NET'
sample_276	PASS	PASS	same	[0]	[0]
sample_277	PASS	PASS	same	[4, 3, 2, 1]	[4, 3, 2, 1]
sample_279	PASS	PASS	same	''	''
sample_284	PASS	PASS	same	''	''
sample_286	PASS	PASS	same	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]
sample_287	PASS	PASS	same	'pinneaple'	'pinneaple'
sample_288	PASS	PASS	same	[(1, 3), (4, 555)]	[(1, 3), (4, 555)]
sample_290	PASS	FAIL	broken	'abixaaaiy'	'ABIXAAAILY'
sample_293	PASS	PASS	same	'no'	'no'
sample_295	PASS	PASS	same	['pear', 'banana', 'pear']	['pear', 'banana', 'pear']
sample_298	PASS	PASS	same	'DST VAVF N DMV DFVM GAMCU DGCVB.'	'DST VAVF N DMV DFVM GAMCU DGCVB.'
sample_300	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_301	PASS	FAIL	broken	[0, 6, 2, -1, -2, 6, 6, -1, -1, -1, -1, -1, -1]	[0, 6, 2, -1, -2, 6, 6, -2, -2, -2, -2, -2, -2]
sample_303	PASS	PASS	same	'mJklbn'	'mJklbn'
sample_305	PASS	FAIL	broken	'o hosteo'	'o hoseto'
sample_306	PASS	PASS	same	[0, 6, 1, 2, 0]	[0, 6, 1, 2, 0]
sample_307	PASS	PASS	same	'px\|\|\|\|\|cznyf'	'px\|\|\|\|\|cznyf'
sample_310	FAIL	PASS	fixed	'__1.00r__j_a6__6'	'__1.00r__j_a6__6'
sample_311	PASS	PASS	same	'no'	'no'
sample_316	PASS	PASS	same	'\| i am your father \|'	'\| i am your father \|'
sample_320	PASS	PASS	same	'usAr'	'usAr'
sample_322	PASS	PASS	same	['lsi', 's', 't', 't', 'd']	['lsi', 's', 't', 't', 'd']
sample_324	PASS	PASS	same	[]	[]
sample_327	PASS	PASS	same	[-3, 1, 7, -1]	[-3, 1, 7, -1]
sample_328	PASS	PASS	same	[1, 2, 3, 1, 2, 3]	[1, 2, 3, 1, 2, 3]
sample_330	PASS	PASS	same	' 4 2 '	' 4 2 '
sample_332	PASS	PASS	same	[]	[]
sample_335	PASS	PASS	same	'sjbrfqmw'	'sjbrfqmw'
sample_337	PASS	PASS	same	'LL'	'LL'
sample_340	PASS	FAIL	broken	' DEGHI Vjkptx'	' DEGHIVjkptx'
sample_341	PASS	PASS	same	{}	{}
sample_343	PASS	PASS	same	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]
sample_344	PASS	PASS	same	[6, 4, 2, 8, 15]	[6, 4, 2, 8, 15]
sample_345	PASS	PASS	same	('mv', 'ml')	('mv', 'ml')
sample_347	PASS	PASS	same	'hhhhhzcw'	'hhhhhzcw'
sample_350	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_356	PASS	PASS	same	[2, 1]	[2, 1]
sample_357	PASS	PASS	same	'werc'	'werc'
sample_358	PASS	PASS	same	'tr'	'tr'
sample_359	PASS	PASS	same	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']
sample_360	PASS	PASS	same	'g'	'g'
sample_363	PASS	PASS	same	[1]	[1]
sample_364	PASS	PASS	same	[[3, False], [1, True]]	[[3, False], [1, True]]
sample_366	PASS	PASS	same	''	''
sample_367	PASS	PASS	same	[6, 2, 1, 1, 4, 1]	[6, 2, 1, 1, 4, 1]
sample_368	PASS	PASS	same	'4327 00004327 000004327 4327 0004327 4327'	'4327 00004327 000004327 4327 0004327 4327'
sample_372	PASS	PASS	same	['']	['']
sample_428	PASS	PASS	same	[]	[]
sample_429	PASS	PASS	same	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]
sample_431	PASS	PASS	same	[]	[]
sample_435	PASS	PASS	same	''	''
sample_445	FAIL	PASS	fixed	'Carrot, Banana, and Strawberry'	'Carrot, Banana, and Strawberry'
sample_446	PASS	PASS	same	[]	[]
sample_457	PASS	PASS	same	[]	[]
sample_480	PASS	PASS	same	''	''
sample_483	PASS	PASS	same	' '	' '
sample_503	PASS	PASS	same	[]	[]
sample_510	PASS	PASS	same	'Wy'	'Wy'
sample_513	PASS	PASS	same	[]	[]
sample_516	PASS	FAIL	broken	['condor']	[]
sample_522	PASS	PASS	same	[]	[]
sample_572	PASS	FAIL	broken		[(2, 10), (3, 1), ((3, 1), None), ((2, 10), None)]
sample_579	PASS	PASS	same	''	''
sample_583	PASS	FAIL	broken		't\nZA\nA'
sample_593	PASS	PASS	same	[]	[]
sample_600	PASS	PASS	same	[]	[]
sample_601	PASS	PASS	same	'ccccc sssss hhhhh AAAAA rrrrr ppppp'	'ccccc sssss hhhhh AAAAA rrrrr ppppp'
sample_623	PASS	PASS	same	'HI~'	'HI~'
sample_663	PASS	PASS	same	[]	[]
sample_672	PASS	PASS	same	'1zd'	'1zd'
sample_673	PASS	PASS	same	"cA"	'cA'
sample_674	PASS	PASS	same	'qq'	'qq'
sample_682	PASS	PASS	same	'hy'	'hy'
sample_687	PASS	FAIL	broken		'R:j:u:g: :z:u:f:E:rjug nzufe'
sample_692	PASS	PASS	same	[]	[]
sample_705	PASS	PASS	same	[]	[]
sample_714	PASS	PASS	same	[]	[]
sample_716	PASS	PASS	same	[]	[]
sample_728	PASS	PASS	same	''	''
sample_735	PASS	PASS	same	'Abb'	'Abb'
sample_759	PASS	PASS	same	[]	[]
sample_761	PASS	PASS	same	[]	[]
sample_767	PASS	PASS	same	'-'	'-'
sample_769	PASS	FAIL	broken		'AKa?RIU'
sample_780	PASS	PASS	same	'2'	'2'
sample_787	PASS	PASS	same	'Xzd'	'Xzd'
sample_797	PASS	PASS	same	[('a', 1), ('b', 2), ('c', 3)]	[('a', 1), ('b', 2), ('c', 3)]

Ornith-1.0-35B-FP8

Baseline: 168 / 184
Mitigated: 168 / 184
Change: 0

Fixed: 7
Broken: 7
Wrong under both: 9
Both right: 161

Scroll sideways to read the full table.

Problem	baseline	mitigated	outcome	predicted (mitigated)	expected
sample_101	PASS	PASS	same	[-4, 4, 1, 0]	[-4, 4, 1, 0]
sample_102	PASS	PASS	same	[]	[]
sample_105	PASS	PASS	same	"Permission Is Granted"	'Permission Is Granted'
sample_106	FAIL	PASS	fixed	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]
sample_107	PASS	PASS	same	'UA6HAJQ'	'UA6HAJQ'
sample_109	PASS	PASS	same	[9, 1, 0, 1, 1]	[9, 1, 0, 1, 1]
sample_111	PASS	PASS	same	(89, 4)	(89, 4)
sample_113	FAIL	FAIL	both		'987YhnShAShD 93275yRgsgBgssHfBsFB'
sample_114	PASS	FAIL	broken	['a', '-.', 'b']	['a', '', 'b']
sample_116	PASS	PASS	same	{}	{}
sample_119	PASS	PASS	same	'VsNlYgLtAw'	'VsNlYgLtAw'
sample_120	PASS	PASS	same	{}	{}
sample_121	PASS	PASS	same	'1001'	'1001'
sample_123	PASS	PASS	same	[1, 2, 3, 5, 6, 8]	[1, 2, 3, 5, 6, 8]
sample_128	PASS	FAIL	broken	'Mmthaoh'	'Mmohamt'
sample_129	PASS	PASS	same	[28, 19, 12, 6]	[28, 19, 12, 6]
sample_130	PASS	PASS	same	'h=l'	'h=l'
sample_132	PASS	PASS	same	'abc'	'abc'
sample_133	PASS	PASS	same	[7, 1, 2]	[7, 1, 2]
sample_135	PASS	PASS	same	['Russia', 'Kazakhstan']	['Russia', 'Kazakhstan']
sample_136	FAIL	FAIL	both	' a \n bc \n \n d \n ef '	' a \n bc \n \n d \n ef '
sample_137	FAIL	PASS	fixed	[]	[]
sample_139	PASS	PASS	same	'no'	'no'
sample_140	PASS	PASS	same	'Hey'	'Hey'
sample_141	PASS	PASS	same	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]
sample_144	PASS	PASS	same	[]	[]
sample_146	PASS	PASS	same	[1, 2, 3, 4, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 6, 7, 8, 9, 10]
sample_147	PASS	PASS	same	[1, 1, 1]	[1, 1, 1]
sample_148	PASS	PASS	same	'2io 12 tfiqr.-'	'2io 12 tfiqr.-'
sample_150	PASS	PASS	same	[-2, 4, -4]	[-2, 4, -4]
sample_154	PASS	PASS	same	'* There Hello'	'* There Hello'
sample_158	PASS	PASS	same	[6, 4, -2, 6, 4, -2]	[6, 4, -2, 6, 4, -2]
sample_159	PASS	PASS	same	'mgItr'	'mgItr'
sample_162	FAIL	FAIL	both	'SBishopSwift'	'СBISHOPSWIFT'
sample_164	PASS	PASS	same	[0, 1, 3]	[0, 1, 3]
sample_166	PASS	PASS	same	{}	{}
sample_167	FAIL	PASS	fixed	'aaQwQwQwbbQwQwQwccQwQwQwde'	'aaQwQwQwbbQwQwQwccQwQwQwde'
sample_169	PASS	PASS	same	'++++taole++++'	'++++taole++++'
sample_171	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_172	PASS	PASS	same	[]	[]
sample_173	PASS	PASS	same	[4, 8, 6, 8, 5]	[4, 8, 6, 8, 5]
sample_174	PASS	PASS	same	[1, 3, 2]	[1, 3, 2]
sample_175	FAIL	FAIL	both	' \n '	' '
sample_177	FAIL	FAIL	both	'HEy DuDe tHI s $ND^ &*THIS@#'	'HEy Dude tHIs $Nd^ &*&tHiS@#'
sample_178	PASS	PASS	same	[2, 2, 2]	[2, 2, 2]
sample_179	PASS	PASS	same	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]
sample_180	PASS	PASS	same	[-1, -2, -6, 8, 8]	[-1, -2, -6, 8, 8]
sample_181	PASS	PASS	same	["3291223", 7]	['3291223', 7]
sample_182	PASS	PASS	same	[('a', 2), ('b', 1)]	[('a', 2), ('b', 1)]
sample_183	PASS	PASS	same	["echo"]	['echo']
sample_184	PASS	PASS	same	[1, 2]	[1, 2]
sample_185	FAIL	PASS	fixed	[11, 14, 7, 12, 9, 16]	[11, 14, 7, 12, 9, 16]
sample_186	PASS	PASS	same	'pvtso'	'pvtso'
sample_188	PASS	PASS	same	['a']	['a']
sample_189	PASS	PASS	same	"{{{{}}}}"	'{{{{}}}}'
sample_193	PASS	PASS	same	'1:1'	'1:1'
sample_194	PASS	PASS	same	[[], [5, 6, 2, 3], [1, 9, 5, 6]]	[[], [5, 6, 2, 3], [1, 9, 5, 6]]
sample_201	PASS	PASS	same	'641524'	'641524'
sample_202	PASS	PASS	same	[15, 15]	[15, 15]
sample_203	PASS	PASS	same	{}	{}
sample_204	PASS	PASS	same	['m', 'a']	['m', 'a']
sample_206	PASS	PASS	same	'h e l l o w o r l d!'	'h e l l o w o r l d!'
sample_208	PASS	PASS	same	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']
sample_212	PASS	PASS	same	[1, -9, 7, 2, 6, -3, 3]	[1, -9, 7, 2, 6, -3, 3]
sample_215	PASS	PASS	same	')'	')'
sample_224	PASS	PASS	same	{}	{}
sample_226	PASS	PASS	same	[1, 3, 3]	[1, 3, 3]
sample_227	PASS	PASS	same	'Manolo'	'Manolo'
sample_228	FAIL	FAIL	both	'llthh#saflapkph twp'	'llthh#saflapkphtswp'
sample_229	PASS	PASS	same	['9', 'm', 'm', 'A', '1', '0', 'L', 'k']	['9', 'm', 'm', 'A', '1', '0', 'L', 'k']
sample_233	FAIL	PASS	fixed	[1, 2, 3]	[1, 2, 3]
sample_235	PASS	PASS	same	[]	[]
sample_236	PASS	PASS	same	'ac8qk6qk6'	'ac8qk6qk6'
sample_238	PASS	PASS	same	[1, 9, 4]	[1, 9, 4]
sample_239	PASS	PASS	same	'1co'	'1co'
sample_241	PASS	PASS	same	'CW'	'CW'
sample_242	PASS	PASS	same	"udhv zcvi nhtnfyd :erwuyawa pun"	'udhv zcvi nhtnfyd :erwuyawa pun'
sample_245	PASS	PASS	same	[]	[]
sample_247	PASS	PASS	same	"yes"	'yes'
sample_248	PASS	PASS	same	[666]	[666]
sample_251	PASS	PASS	same	"Marie;Nelson;Oscar;+;3;5;3"	'Marie;Nelson;Oscar;+;3;5;3'
sample_254	FAIL	PASS	fixed	'lwwer case'	'lwwer case'
sample_255	PASS	PASS	same	'w'	'w'
sample_257	PASS	PASS	same	[['Hello World', '"I am String"']]	[['Hello World', '"I am String"']]
sample_258	PASS	PASS	same	[1, 2, 7, 3, 9]	[1, 2, 7, 3, 9]
sample_259	FAIL	PASS	fixed	'RES'	'RES'
sample_260	PASS	PASS	same	[1, 2, 3, 4, 6, 5]	[1, 2, 3, 4, 6, 5]
sample_261	PASS	PASS	same	([], [12, 516, 5, 214, 51])	([], [12, 516, 5, 214, 51])
sample_262	PASS	PASS	same	"BA"	'BA'
sample_263	PASS	PASS	same	['gloss', 'banana', 'barn', 'lawn']	['gloss', 'banana', 'barn', 'lawn']
sample_266	PASS	PASS	same	[2, 3, 3, 4, 6, -2]	[2, 3, 3, 4, 6, -2]
sample_268	FAIL	FAIL	both	'h / grateful k'	'h / g r a t e f u l k'
sample_269	PASS	PASS	same	['0', 2]	['0', 2]
sample_270	PASS	PASS	same	{}	{}
sample_271	PASS	PASS	same	'uufh'	'uufh'
sample_272	FAIL	FAIL	both	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 1, 6, 8, 0, 0]	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 2, 6, 0, 6, 6]
sample_273	PASS	PASS	same	'NET'	'NET'
sample_276	PASS	PASS	same	[0]	[0]
sample_277	PASS	PASS	same	[4, 3, 2, 1]	[4, 3, 2, 1]
sample_279	PASS	PASS	same	''	''
sample_284	PASS	PASS	same	''	''
sample_286	PASS	PASS	same	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]
sample_287	PASS	PASS	same	'pinneaple'	'pinneaple'
sample_288	PASS	PASS	same	[(1, 3), (4, 555)]	[(1, 3), (4, 555)]
sample_290	PASS	PASS	same	'ABIXAAAILY'	'ABIXAAAILY'
sample_293	PASS	PASS	same	'no'	'no'
sample_295	PASS	PASS	same	['pear', 'banana', 'pear']	['pear', 'banana', 'pear']
sample_298	PASS	PASS	same	'DST VAVF N DMV DFVM GAMCU DGCVB.'	'DST VAVF N DMV DFVM GAMCU DGCVB.'
sample_300	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_301	PASS	FAIL	broken	[0, 6, 2, -1, -2, 6, 6, -1, -1, -1, -1, -1, -1]	[0, 6, 2, -1, -2, 6, 6, -2, -2, -2, -2, -2, -2]
sample_303	PASS	PASS	same	'mJklbn'	'mJklbn'
sample_305	PASS	FAIL	broken	'o hose to'	'o hoseto'
sample_306	PASS	PASS	same	[0, 6, 1, 2, 0]	[0, 6, 1, 2, 0]
sample_307	PASS	FAIL	broken	'px\|\|\|\|\|\|pcznyf'	'px\|\|\|\|\|cznyf'
sample_310	PASS	FAIL	broken	'__1.00r____a6__6'	'__1.00r__j_a6__6'
sample_311	PASS	PASS	same	'no'	'no'
sample_316	PASS	PASS	same	'\| i am your father \|'	'\| i am your father \|'
sample_320	PASS	PASS	same	'usAr'	'usAr'
sample_322	PASS	PASS	same	['lsi', 's', 't', 't', 'd']	['lsi', 's', 't', 't', 'd']
sample_324	PASS	PASS	same	[]	[]
sample_327	PASS	PASS	same	[-3, 1, 7, -1]	[-3, 1, 7, -1]
sample_328	PASS	PASS	same	[1, 2, 3, 1, 2, 3]	[1, 2, 3, 1, 2, 3]
sample_330	PASS	PASS	same	' 4 2 '	' 4 2 '
sample_332	PASS	PASS	same	[]	[]
sample_335	PASS	PASS	same	'sjbrfqmw'	'sjbrfqmw'
sample_337	PASS	PASS	same	"LL"	'LL'
sample_340	PASS	PASS	same	' DEGHIVjkptx'	' DEGHIVjkptx'
sample_341	PASS	PASS	same	{}	{}
sample_343	PASS	PASS	same	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]
sample_344	PASS	PASS	same	[6, 4, 2, 8, 15]	[6, 4, 2, 8, 15]
sample_345	PASS	PASS	same	('mv', 'ml')	('mv', 'ml')
sample_347	PASS	PASS	same	'hhhhhzcw'	'hhhhhzcw'
sample_350	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_356	PASS	PASS	same	[2, 1]	[2, 1]
sample_357	PASS	PASS	same	'werc'	'werc'
sample_358	PASS	PASS	same	'tr'	'tr'
sample_359	PASS	PASS	same	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']
sample_360	PASS	PASS	same	'g'	'g'
sample_363	PASS	PASS	same	[1]	[1]
sample_364	PASS	PASS	same	[[3, False], [1, True]]	[[3, False], [1, True]]
sample_366	PASS	PASS	same	''	''
sample_367	PASS	PASS	same	[6, 2, 1, 1, 4, 1]	[6, 2, 1, 1, 4, 1]
sample_368	PASS	PASS	same	'4327 00004327 000004327 4327 0004327 4327'	'4327 00004327 000004327 4327 0004327 4327'
sample_372	PASS	PASS	same	['']	['']
sample_428	PASS	PASS	same	[]	[]
sample_429	PASS	PASS	same	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]
sample_431	PASS	PASS	same	[]	[]
sample_435	PASS	PASS	same	''	''
sample_445	PASS	PASS	same	'Carrot, Banana, and Strawberry'	'Carrot, Banana, and Strawberry'
sample_446	PASS	PASS	same	[]	[]
sample_457	PASS	PASS	same	[]	[]
sample_480	PASS	PASS	same	''	''
sample_483	PASS	PASS	same	' '	' '
sample_503	PASS	PASS	same	[]	[]
sample_510	PASS	PASS	same	'Wy'	'Wy'
sample_513	PASS	PASS	same	[]	[]
sample_516	PASS	PASS	same	[]	[]
sample_522	PASS	PASS	same	[]	[]
sample_572	FAIL	FAIL	both	[(2, 10), (3, 1), (3, 1), (2, 10)]	[(2, 10), (3, 1), ((3, 1), None), ((2, 10), None)]
sample_579	PASS	PASS	same	''	''
sample_583	PASS	PASS	same	't\nZA\nA'	't\nZA\nA'
sample_593	PASS	PASS	same	[]	[]
sample_600	PASS	PASS	same	[]	[]
sample_601	PASS	PASS	same	'ccccc sssss hhhhh AAAAA rrrrr ppppp'	'ccccc sssss hhhhh AAAAA rrrrr ppppp'
sample_623	PASS	FAIL	broken	'!~ih'	'HI~'
sample_663	PASS	PASS	same	[]	[]
sample_672	PASS	PASS	same	"1zd"	'1zd'
sample_673	PASS	PASS	same	"cA"	'cA'
sample_674	PASS	PASS	same	'qq'	'qq'
sample_682	PASS	PASS	same	'hy'	'hy'
sample_687	PASS	PASS	same	'R:j:u:g: :z:u:f:E:rjug nzufe'	'R:j:u:g: :z:u:f:E:rjug nzufe'
sample_692	PASS	PASS	same	[]	[]
sample_705	PASS	PASS	same	[]	[]
sample_714	PASS	PASS	same	[]	[]
sample_716	PASS	PASS	same	[]	[]
sample_728	PASS	PASS	same	''	''
sample_735	PASS	PASS	same	'Abb'	'Abb'
sample_759	PASS	PASS	same	[]	[]
sample_761	PASS	PASS	same	[]	[]
sample_767	PASS	PASS	same	'-'	'-'
sample_769	PASS	PASS	same	'AKa?RIU'	'AKa?RIU'
sample_780	PASS	PASS	same	'2'	'2'
sample_787	PASS	PASS	same	'Xzd'	'Xzd'
sample_797	PASS	PASS	same	[('a', 1), ('b', 2), ('c', 3)]	[('a', 1), ('b', 2), ('c', 3)]

Holo-3.1-35B-A3B-NVFP4

Baseline: 88 / 184
Mitigated: 141 / 184
Change: +53

Fixed: 54
Broken: 1
Wrong under both: 42
Both right: 87

Scroll sideways to read the full table.

Problem	baseline	mitigated	outcome	predicted (mitigated)	expected
sample_101	PASS	PASS	same	[-4, 4, 1, 0]	[-4, 4, 1, 0]
sample_102	PASS	PASS	same	[]	[]
sample_105	FAIL	FAIL	both	"PermissiOn Is Granted"	'Permission Is Granted'
sample_106	FAIL	PASS	fixed	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]
sample_107	FAIL	PASS	fixed	"UA6HAJQ"	'UA6HAJQ'
sample_109	PASS	PASS	same	[9, 1, 0, 1, 1]	[9, 1, 0, 1, 1]
sample_111	PASS	PASS	same	(89, 4)	(89, 4)
sample_113	FAIL	FAIL	both		'987YhnShAShD 93275yRgsgBgssHfBsFB'
sample_114	FAIL	FAIL	both	['a', '-', 'b']	['a', '', 'b']
sample_116	PASS	PASS	same	{}	{}
sample_119	FAIL	PASS	fixed	'VsNlYgLtAw'	'VsNlYgLtAw'
sample_120	PASS	PASS	same	{}	{}
sample_121	FAIL	PASS	fixed	'1001'	'1001'
sample_123	PASS	PASS	same	[1, 2, 3, 5, 6, 8]	[1, 2, 3, 5, 6, 8]
sample_128	FAIL	PASS	fixed	'Mmohamt'	'Mmohamt'
sample_129	FAIL	FAIL	both	[29, 19, 12, 6]	[28, 19, 12, 6]
sample_130	FAIL	FAIL	both	'l=4 h=6 o=9'	'h=l'
sample_132	PASS	PASS	same	'abc'	'abc'
sample_133	PASS	PASS	same	[7, 1, 2]	[7, 1, 2]
sample_135	PASS	PASS	same	['Russia', 'Kazakhstan']	['Russia', 'Kazakhstan']
sample_136	FAIL	FAIL	both	" a \n bc \n \n d \n ef "	' a \n bc \n \n d \n ef '
sample_137	PASS	PASS	same	[]	[]
sample_139	FAIL	PASS	fixed	'no'	'no'
sample_140	FAIL	FAIL	both	'Hi'	'Hey'
sample_141	FAIL	PASS	fixed	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]
sample_144	PASS	PASS	same	[]	[]
sample_146	PASS	PASS	same	[1, 2, 3, 4, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 6, 7, 8, 9, 10]
sample_147	PASS	PASS	same	[1, 1, 1]	[1, 1, 1]
sample_148	FAIL	PASS	fixed	'2io 12 tfiqr.-'	'2io 12 tfiqr.-'
sample_150	FAIL	PASS	fixed	[-2, 4, -4]	[-2, 4, -4]
sample_154	FAIL	PASS	fixed	'* There Hello'	'* There Hello'
sample_158	FAIL	PASS	fixed	[6, 4, -2, 6, 4, -2]	[6, 4, -2, 6, 4, -2]
sample_159	FAIL	PASS	fixed	'mgItr'	'mgItr'
sample_162	FAIL	PASS	fixed	'СBISHOPSWIFT'	'СBISHOPSWIFT'
sample_164	PASS	PASS	same	[0, 1, 3]	[0, 1, 3]
sample_166	PASS	PASS	same	{}	{}
sample_167	FAIL	FAIL	both		'aaQwQwQwbbQwQwQwccQwQwQwde'
sample_169	FAIL	PASS	fixed	'++++taole++++'	'++++taole++++'
sample_171	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_172	PASS	PASS	same	[]	[]
sample_173	PASS	PASS	same	[4, 8, 6, 8, 5]	[4, 8, 6, 8, 5]
sample_174	PASS	PASS	same	[1, 3, 2]	[1, 3, 2]
sample_175	FAIL	FAIL	both		' '
sample_177	FAIL	FAIL	both	'hEy dUdE tHis $nD^ &*&ThIs@#'	'HEy Dude tHIs $Nd^ &*&tHiS@#'
sample_178	PASS	PASS	same	[2, 2, 2]	[2, 2, 2]
sample_179	PASS	PASS	same	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]
sample_180	PASS	PASS	same	[-1, -2, -6, 8, 8]	[-1, -2, -6, 8, 8]
sample_181	FAIL	PASS	fixed	["3291223", 7]	['3291223', 7]
sample_182	PASS	PASS	same	[('a', 2), ('b', 1)]	[('a', 2), ('b', 1)]
sample_183	FAIL	FAIL	both	["e h! n!", "hello!!!"]	['echo']
sample_184	PASS	PASS	same	[1, 2]	[1, 2]
sample_185	FAIL	FAIL	both		[11, 14, 7, 12, 9, 16]
sample_186	FAIL	PASS	fixed	'pvtso'	'pvtso'
sample_188	PASS	PASS	same	['a']	['a']
sample_189	PASS	PASS	same	"{{{{}}}}"	'{{{{}}}}'
sample_193	FAIL	PASS	fixed	'1:1'	'1:1'
sample_194	PASS	PASS	same	[[] , [5, 6, 2, 3], [1, 9, 5, 6]]	[[], [5, 6, 2, 3], [1, 9, 5, 6]]
sample_201	FAIL	PASS	fixed	'641524'	'641524'
sample_202	PASS	PASS	same	[15, 15]	[15, 15]
sample_203	PASS	PASS	same	{}	{}
sample_204	FAIL	PASS	fixed	['m', 'a']	['m', 'a']
sample_206	FAIL	PASS	fixed	'h e l l o w o r l d!'	'h e l l o w o r l d!'
sample_208	FAIL	PASS	fixed	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']
sample_212	PASS	PASS	same	[1, -9, 7, 2, 6, -3, 3]	[1, -9, 7, 2, 6, -3, 3]
sample_215	FAIL	PASS	fixed	')'	')'
sample_224	PASS	PASS	same	{}	{}
sample_226	PASS	PASS	same	[1, 3, 3]	[1, 3, 3]
sample_227	FAIL	PASS	fixed	'Manolo'	'Manolo'
sample_228	FAIL	FAIL	both	3	'llthh#saflapkphtswp'
sample_229	FAIL	PASS	fixed	['9', 'm', 'm', 'A', '1', '0', 'L', 'k']	['9', 'm', 'm', 'A', '1', '0', 'L', 'k']
sample_233	FAIL	FAIL	both	[2, 1, 3]	[1, 2, 3]
sample_235	PASS	PASS	same	[]	[]
sample_236	FAIL	PASS	fixed	'ac8qk6qk6'	'ac8qk6qk6'
sample_238	PASS	PASS	same	[1, 9, 4]	[1, 9, 4]
sample_239	FAIL	PASS	fixed	'1co'	'1co'
sample_241	FAIL	PASS	fixed	'CW'	'CW'
sample_242	FAIL	PASS	fixed	"udhv zcvi nhtnfyd :erwuyawa pun"	'udhv zcvi nhtnfyd :erwuyawa pun'
sample_245	FAIL	FAIL	both	['v', 'z']	[]
sample_247	FAIL	FAIL	both		'yes'
sample_248	PASS	PASS	same	[666]	[666]
sample_251	FAIL	FAIL	both	'+353 Marie;Nelson;Oscar'	'Marie;Nelson;Oscar;+;3;5;3'
sample_254	FAIL	FAIL	both	'lower case'	'lwwer case'
sample_255	FAIL	PASS	fixed	'w'	'w'
sample_257	PASS	PASS	same	[['Hello World', '"I am String"']]	[['Hello World', '"I am String"']]
sample_258	FAIL	PASS	fixed	[1, 2, 7, 3, 9]	[1, 2, 7, 3, 9]
sample_259	FAIL	FAIL	both	'RSE'	'RES'
sample_260	PASS	PASS	same	[1, 2, 3, 4, 6, 5]	[1, 2, 3, 4, 6, 5]
sample_261	PASS	PASS	same	([], [12, 516, 5, 214, 51])	([], [12, 516, 5, 214, 51])
sample_262	FAIL	FAIL	both		'BA'
sample_263	PASS	PASS	same	["gloss", "banana", "barn", "lawn"]	['gloss', 'banana', 'barn', 'lawn']
sample_266	FAIL	FAIL	both	[2, 3, 3, 4, 6, -2, -2]	[2, 3, 3, 4, 6, -2]
sample_268	FAIL	FAIL	both	'h/grateful k'	'h / g r a t e f u l k'
sample_269	PASS	PASS	same	['0', 2]	['0', 2]
sample_270	PASS	PASS	same	{}	{}
sample_271	FAIL	PASS	fixed	'uufh'	'uufh'
sample_272	FAIL	FAIL	both	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 4, 6, 8, 0, 0]	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 2, 6, 0, 6, 6]
sample_273	PASS	PASS	same	'NET'	'NET'
sample_276	PASS	PASS	same	[0]	[0]
sample_277	PASS	PASS	same	[4, 3, 2, 1]	[4, 3, 2, 1]
sample_279	PASS	PASS	same	''	''
sample_284	PASS	PASS	same	''	''
sample_286	PASS	PASS	same	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]
sample_287	FAIL	FAIL	both		'pinneaple'
sample_288	FAIL	PASS	fixed	[(1, 3), (4, 555)]	[(1, 3), (4, 555)]
sample_290	FAIL	FAIL	both	'abixaa'	'ABIXAAAILY'
sample_293	FAIL	PASS	fixed	'no'	'no'
sample_295	FAIL	PASS	fixed	['pear', 'banana', 'pear']	['pear', 'banana', 'pear']
sample_298	FAIL	PASS	fixed	'DST VAVF N DMV DFVM GAMCU DGCVB.'	'DST VAVF N DMV DFVM GAMCU DGCVB.'
sample_300	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_301	FAIL	FAIL	both	[0, 6, 2, -1, -2, 6, 6, -1, -1, 6, 6, 6, 6]	[0, 6, 2, -1, -2, 6, 6, -2, -2, -2, -2, -2, -2]
sample_303	FAIL	FAIL	both	'mJkln'	'mJklbn'
sample_305	FAIL	FAIL	both	'o hosedto'	'o hoseto'
sample_306	PASS	PASS	same	[0, 6, 1, 2, 0]	[0, 6, 1, 2, 0]
sample_307	FAIL	FAIL	both	'p\|xcz\|nyf\|'	'px\|\|\|\|\|cznyf'
sample_310	FAIL	FAIL	both	'__1.00r__ _aj6__6'	'__1.00r__j_a6__6'
sample_311	FAIL	PASS	fixed	'no'	'no'
sample_316	FAIL	FAIL	both	' \| i am your father \|'	'\| i am your father \|'
sample_320	FAIL	FAIL	both	'usAR'	'usAr'
sample_322	PASS	PASS	same	['lsi', 's', 't', 't', 'd']	['lsi', 's', 't', 't', 'd']
sample_324	PASS	PASS	same	[]	[]
sample_327	FAIL	PASS	fixed	[-3, 1, 7, -1]	[-3, 1, 7, -1]
sample_328	PASS	PASS	same	[1, 2, 3, 1, 2, 3]	[1, 2, 3, 1, 2, 3]
sample_330	PASS	PASS	same	' 4 2 '	' 4 2 '
sample_332	PASS	PASS	same	[]	[]
sample_335	FAIL	FAIL	both	'sjbrlfqmw'	'sjbrfqmw'
sample_337	FAIL	PASS	fixed	'LL'	'LL'
sample_340	FAIL	FAIL	both	' DEGHIKPTVaddehikptvx'	' DEGHIVjkptx'
sample_341	PASS	PASS	same	{}	{}
sample_343	PASS	PASS	same	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]
sample_344	FAIL	PASS	fixed	[6, 4, 2, 8, 15]	[6, 4, 2, 8, 15]
sample_345	PASS	PASS	same	('mv', 'ml')	('mv', 'ml')
sample_347	FAIL	FAIL	both		'hhhhhzcw'
sample_350	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_356	PASS	PASS	same	[2, 1]	[2, 1]
sample_357	FAIL	PASS	fixed	'werc'	'werc'
sample_358	FAIL	PASS	fixed	'tr'	'tr'
sample_359	PASS	PASS	same	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']
sample_360	FAIL	PASS	fixed	'g'	'g'
sample_363	PASS	PASS	same	[1]	[1]
sample_364	PASS	PASS	same	[[3, False], [1, True]]	[[3, False], [1, True]]
sample_366	FAIL	FAIL	both	' [lo]+de,_i'	''
sample_367	PASS	PASS	same	[6, 2, 1, 1, 4, 1]	[6, 2, 1, 1, 4, 1]
sample_368	FAIL	PASS	fixed	'4327 00004327 000004327 4327 0004327 4327'	'4327 00004327 000004327 4327 0004327 4327'
sample_372	FAIL	PASS	fixed	['']	['']
sample_428	PASS	PASS	same	[]	[]
sample_429	PASS	PASS	same	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]
sample_431	PASS	PASS	same	[]	[]
sample_435	PASS	PASS	same	''	''
sample_445	FAIL	PASS	fixed	'Carrot, Banana, and Strawberry'	'Carrot, Banana, and Strawberry'
sample_446	PASS	PASS	same	[]	[]
sample_457	PASS	PASS	same	[]	[]
sample_480	PASS	PASS	same	''	''
sample_483	FAIL	FAIL	both	''	' '
sample_503	PASS	PASS	same	[]	[]
sample_510	PASS	PASS	same	'Wy'	'Wy'
sample_513	FAIL	PASS	fixed	[]	[]
sample_516	PASS	PASS	same	[]	[]
sample_522	PASS	PASS	same	[]	[]
sample_572	FAIL	FAIL	both	[(2, 10), (3, 1), (3, 1), (2, 10), (1, 9)]	[(2, 10), (3, 1), ((3, 1), None), ((2, 10), None)]
sample_579	PASS	PASS	same	''	''
sample_583	FAIL	PASS	fixed	"t\nZA\nA"	't\nZA\nA'
sample_593	PASS	PASS	same	[]	[]
sample_600	PASS	PASS	same	[]	[]
sample_601	FAIL	FAIL	both		'ccccc sssss hhhhh AAAAA rrrrr ppppp'
sample_623	FAIL	FAIL	both	'hi~'	'HI~'
sample_663	PASS	PASS	same	[]	[]
sample_672	FAIL	PASS	fixed	"1zd"	'1zd'
sample_673	FAIL	PASS	fixed	"cA"	'cA'
sample_674	PASS	PASS	same	'qq'	'qq'
sample_682	FAIL	PASS	fixed	'hy'	'hy'
sample_687	FAIL	PASS	fixed	'R:j:u:g: :z:u:f:E:rjug nzufe'	'R:j:u:g: :z:u:f:E:rjug nzufe'
sample_692	PASS	PASS	same	[]	[]
sample_705	PASS	PASS	same	[]	[]
sample_714	PASS	PASS	same	[]	[]
sample_716	PASS	PASS	same	[]	[]
sample_728	PASS	PASS	same	''	''
sample_735	FAIL	FAIL	both	'Ab'	'Abb'
sample_759	PASS	PASS	same	[]	[]
sample_761	PASS	PASS	same	[]	[]
sample_767	FAIL	FAIL	both		'-'
sample_769	FAIL	PASS	fixed	'AKa?RIU'	'AKa?RIU'
sample_780	FAIL	PASS	fixed	'2'	'2'
sample_787	FAIL	PASS	fixed	'Xzd'	'Xzd'
sample_797	PASS	FAIL	broken		[('a', 1), ('b', 2), ('c', 3)]

Qwen3.6-35B-A3B-NVFP4

Baseline: 163 / 184
Mitigated: 149 / 184
Change: -14

Fixed: 9
Broken: 23
Wrong under both: 12
Both right: 140

Scroll sideways to read the full table.

Problem	baseline	mitigated	outcome	predicted (mitigated)	expected
sample_101	PASS	PASS	same	[-4, 4, 1, 0]	[-4, 4, 1, 0]
sample_102	PASS	PASS	same	[]	[]
sample_105	PASS	PASS	same	'Permission Is Granted'	'Permission Is Granted'
sample_106	PASS	PASS	same	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]
sample_107	PASS	PASS	same	"UA6HAJQ"	'UA6HAJQ'
sample_109	PASS	PASS	same	[9, 1, 0, 1, 1]	[9, 1, 0, 1, 1]
sample_111	PASS	PASS	same	(89, 4)	(89, 4)
sample_113	FAIL	FAIL	both	9	'987YhnShAShD 93275yRgsgBgssHfBsFB'
sample_114	FAIL	FAIL	both	['a', '.', '.b']	['a', '', 'b']
sample_116	PASS	PASS	same	{}	{}
sample_119	PASS	PASS	same	"VsNlYgLtAw"	'VsNlYgLtAw'
sample_120	PASS	PASS	same	{}	{}
sample_121	FAIL	PASS	fixed	'1001'	'1001'
sample_123	PASS	FAIL	broken		[1, 2, 3, 5, 6, 8]
sample_128	PASS	PASS	same	'Mmohamt'	'Mmohamt'
sample_129	FAIL	FAIL	both		[28, 19, 12, 6]
sample_130	PASS	PASS	same	'h=l'	'h=l'
sample_132	PASS	PASS	same	'abc'	'abc'
sample_133	PASS	FAIL	broken		[7, 1, 2]
sample_135	PASS	PASS	same	['Russia', 'Kazakhstan']	['Russia', 'Kazakhstan']
sample_136	FAIL	FAIL	both		' a \n bc \n \n d \n ef '
sample_137	PASS	FAIL	broken		[]
sample_139	PASS	PASS	same	'no'	'no'
sample_140	PASS	PASS	same	'Hey'	'Hey'
sample_141	PASS	PASS	same	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]
sample_144	PASS	PASS	same	[]	[]
sample_146	PASS	PASS	same	[1, 2, 3, 4, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 6, 7, 8, 9, 10]
sample_147	PASS	PASS	same	[1, 1, 1]	[1, 1, 1]
sample_148	FAIL	FAIL	both	13	'2io 12 tfiqr.-'
sample_150	PASS	FAIL	broken		[-2, 4, -4]
sample_154	PASS	PASS	same	'* There Hello'	'* There Hello'
sample_158	PASS	PASS	same	[6, 4, -2, 6, 4, -2]	[6, 4, -2, 6, 4, -2]
sample_159	PASS	PASS	same	'mgItr'	'mgItr'
sample_162	FAIL	PASS	fixed	'СBISHOPSWIFT'	'СBISHOPSWIFT'
sample_164	PASS	PASS	same	[0, 1, 3]	[0, 1, 3]
sample_166	PASS	PASS	same	{}	{}
sample_167	PASS	PASS	same	'aaQwQwQwbbQwQwQwccQwQwQwde'	'aaQwQwQwbbQwQwQwccQwQwQwde'
sample_169	PASS	PASS	same	'++++taole++++'	'++++taole++++'
sample_171	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_172	PASS	PASS	same	[]	[]
sample_173	PASS	PASS	same	[4, 8, 6, 8, 5]	[4, 8, 6, 8, 5]
sample_174	PASS	PASS	same	[1, 3, 2]	[1, 3, 2]
sample_175	FAIL	FAIL	both	['', '']	' '
sample_177	PASS	FAIL	broken		'HEy Dude tHIs $Nd^ &*&tHiS@#'
sample_178	FAIL	PASS	fixed	[2, 2, 2]	[2, 2, 2]
sample_179	PASS	PASS	same	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]
sample_180	PASS	PASS	same	[-1, -2, -6, 8, 8]	[-1, -2, -6, 8, 8]
sample_181	PASS	PASS	same	["3291223", 7]	['3291223', 7]
sample_182	PASS	PASS	same	[('a', 2), ('b', 1)]	[('a', 2), ('b', 1)]
sample_183	PASS	PASS	same	['echo']	['echo']
sample_184	PASS	PASS	same	[1, 2]	[1, 2]
sample_185	PASS	PASS	same	[11, 14, 7, 12, 9, 16]	[11, 14, 7, 12, 9, 16]
sample_186	PASS	PASS	same	'pvtso'	'pvtso'
sample_188	PASS	PASS	same	["a"]	['a']
sample_189	PASS	FAIL	broken		'{{{{}}}}'
sample_193	PASS	PASS	same	'1:1'	'1:1'
sample_194	PASS	PASS	same	[[], [5, 6, 2, 3], [1, 9, 5, 6]]	[[], [5, 6, 2, 3], [1, 9, 5, 6]]
sample_201	PASS	PASS	same	'641524'	'641524'
sample_202	PASS	PASS	same	[15, 15]	[15, 15]
sample_203	PASS	PASS	same	{}	{}
sample_204	PASS	PASS	same	['m', 'a']	['m', 'a']
sample_206	PASS	PASS	same	'h e l l o w o r l d!'	'h e l l o w o r l d!'
sample_208	PASS	PASS	same	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']
sample_212	PASS	PASS	same	[1, -9, 7, 2, 6, -3, 3]	[1, -9, 7, 2, 6, -3, 3]
sample_215	PASS	FAIL	broken		')'
sample_224	PASS	PASS	same	{}	{}
sample_226	PASS	PASS	same	[1, 3, 3]	[1, 3, 3]
sample_227	PASS	PASS	same	'Manolo'	'Manolo'
sample_228	FAIL	FAIL	both		'llthh#saflapkphtswp'
sample_229	FAIL	PASS	fixed	['9', 'm', 'm', 'A', '1', '0', 'L', 'k']	['9', 'm', 'm', 'A', '1', '0', 'L', 'k']
sample_233	PASS	FAIL	broken		[1, 2, 3]
sample_235	PASS	PASS	same	[]	[]
sample_236	PASS	PASS	same	'ac8qk6qk6'	'ac8qk6qk6'
sample_238	PASS	PASS	same	[1, 9, 4]	[1, 9, 4]
sample_239	PASS	PASS	same	"1co"	'1co'
sample_241	PASS	PASS	same	'CW'	'CW'
sample_242	PASS	PASS	same	"udhv zcvi nhtnfyd :erwuyawa pun"	'udhv zcvi nhtnfyd :erwuyawa pun'
sample_245	PASS	PASS	same	[]	[]
sample_247	PASS	PASS	same	"yes"	'yes'
sample_248	PASS	PASS	same	[666]	[666]
sample_251	PASS	PASS	same	"Marie;Nelson;Oscar;+;3;5;3"	'Marie;Nelson;Oscar;+;3;5;3'
sample_254	FAIL	PASS	fixed	'lwwer case'	'lwwer case'
sample_255	PASS	PASS	same	'w'	'w'
sample_257	PASS	PASS	same	[['Hello World', '"I am String"']]	[['Hello World', '"I am String"']]
sample_258	PASS	FAIL	broken		[1, 2, 7, 3, 9]
sample_259	PASS	PASS	same	'RES'	'RES'
sample_260	PASS	FAIL	broken		[1, 2, 3, 4, 6, 5]
sample_261	PASS	PASS	same	([], [12, 516, 5, 214, 51])	([], [12, 516, 5, 214, 51])
sample_262	PASS	PASS	same	"BA"	'BA'
sample_263	PASS	PASS	same	["gloss", "banana", "barn", "lawn"]	['gloss', 'banana', 'barn', 'lawn']
sample_266	PASS	PASS	same	[2, 3, 3, 4, 6, -2]	[2, 3, 3, 4, 6, -2]
sample_268	FAIL	PASS	fixed	"h / g r a t e f u l k"	'h / g r a t e f u l k'
sample_269	PASS	PASS	same	['0', 2]	['0', 2]
sample_270	PASS	PASS	same	{}	{}
sample_271	PASS	PASS	same	'uufh'	'uufh'
sample_272	PASS	FAIL	broken		[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 2, 6, 0, 6, 6]
sample_273	PASS	PASS	same	'NET'	'NET'
sample_276	PASS	PASS	same	[0]	[0]
sample_277	PASS	PASS	same	[4, 3, 2, 1]	[4, 3, 2, 1]
sample_279	PASS	FAIL	broken		''
sample_284	PASS	PASS	same	''	''
sample_286	PASS	PASS	same	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]
sample_287	PASS	PASS	same	'pinneaple'	'pinneaple'
sample_288	FAIL	FAIL	both		[(1, 3), (4, 555)]
sample_290	FAIL	PASS	fixed	'ABIXAAAILY'	'ABIXAAAILY'
sample_293	PASS	PASS	same	'no'	'no'
sample_295	PASS	PASS	same	['pear', 'banana', 'pear']	['pear', 'banana', 'pear']
sample_298	FAIL	PASS	fixed	'DST VAVF N DMV DFVM GAMCU DGCVB.'	'DST VAVF N DMV DFVM GAMCU DGCVB.'
sample_300	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_301	FAIL	FAIL	both		[0, 6, 2, -1, -2, 6, 6, -2, -2, -2, -2, -2, -2]
sample_303	PASS	FAIL	broken	'mJkln'	'mJklbn'
sample_305	PASS	PASS	same	'o hoseto'	'o hoseto'
sample_306	PASS	PASS	same	[0, 6, 1, 2, 0]	[0, 6, 1, 2, 0]
sample_307	PASS	FAIL	broken		'px\|\|\|\|\|cznyf'
sample_310	FAIL	FAIL	both		'__1.00r__j_a6__6'
sample_311	PASS	PASS	same	'no'	'no'
sample_316	PASS	PASS	same	'\| i am your father \|'	'\| i am your father \|'
sample_320	PASS	PASS	same	'usAr'	'usAr'
sample_322	PASS	PASS	same	['lsi', 's', 't', 't', 'd']	['lsi', 's', 't', 't', 'd']
sample_324	PASS	PASS	same	[]	[]
sample_327	PASS	FAIL	broken		[-3, 1, 7, -1]
sample_328	PASS	PASS	same	[1, 2, 3, 1, 2, 3]	[1, 2, 3, 1, 2, 3]
sample_330	PASS	PASS	same	' 4 2 '	' 4 2 '
sample_332	PASS	PASS	same	[]	[]
sample_335	PASS	PASS	same	'sjbrfqmw'	'sjbrfqmw'
sample_337	PASS	PASS	same	'LL'	'LL'
sample_340	PASS	PASS	same	' DEGHIVjkptx'	' DEGHIVjkptx'
sample_341	PASS	PASS	same	{}	{}
sample_343	PASS	FAIL	broken		[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]
sample_344	PASS	PASS	same	[6, 4, 2, 8, 15]	[6, 4, 2, 8, 15]
sample_345	PASS	PASS	same	('mv', 'ml')	('mv', 'ml')
sample_347	FAIL	FAIL	both		'hhhhhzcw'
sample_350	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_356	PASS	PASS	same	[2, 1]	[2, 1]
sample_357	PASS	PASS	same	'werc'	'werc'
sample_358	PASS	PASS	same	'tr'	'tr'
sample_359	PASS	PASS	same	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']
sample_360	PASS	PASS	same	'g'	'g'
sample_363	PASS	PASS	same	[1]	[1]
sample_364	PASS	PASS	same	[[3, False], [1, True]]	[[3, False], [1, True]]
sample_366	PASS	PASS	same	''	''
sample_367	PASS	PASS	same	[6, 2, 1, 1, 4, 1]	[6, 2, 1, 1, 4, 1]
sample_368	FAIL	PASS	fixed	"4327 00004327 000004327 4327 0004327 4327"	'4327 00004327 000004327 4327 0004327 4327'
sample_372	PASS	PASS	same	['']	['']
sample_428	PASS	PASS	same	[]	[]
sample_429	PASS	PASS	same	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]
sample_431	PASS	PASS	same	[]	[]
sample_435	PASS	FAIL	broken		''
sample_445	PASS	FAIL	broken	' and Strawberry'	'Carrot, Banana, and Strawberry'
sample_446	PASS	PASS	same	[]	[]
sample_457	PASS	PASS	same	[]	[]
sample_480	PASS	PASS	same	''	''
sample_483	PASS	PASS	same	' '	' '
sample_503	PASS	PASS	same	[]	[]
sample_510	PASS	PASS	same	'Wy'	'Wy'
sample_513	PASS	PASS	same	[]	[]
sample_516	PASS	PASS	same	[]	[]
sample_522	PASS	PASS	same	[]	[]
sample_572	FAIL	FAIL	both		[(2, 10), (3, 1), ((3, 1), None), ((2, 10), None)]
sample_579	PASS	PASS	same	''	''
sample_583	PASS	PASS	same	"t\nZA\nA"	't\nZA\nA'
sample_593	PASS	PASS	same	[]	[]
sample_600	PASS	PASS	same	[]	[]
sample_601	PASS	PASS	same	"ccccc sssss hhhhh AAAAA rrrrr ppppp"	'ccccc sssss hhhhh AAAAA rrrrr ppppp'
sample_623	PASS	PASS	same	'HI~'	'HI~'
sample_663	PASS	PASS	same	[]	[]
sample_672	PASS	FAIL	broken		'1zd'
sample_673	PASS	PASS	same	"cA"	'cA'
sample_674	PASS	PASS	same	'qq'	'qq'
sample_682	PASS	PASS	same	'hy'	'hy'
sample_687	PASS	FAIL	broken		'R:j:u:g: :z:u:f:E:rjug nzufe'
sample_692	PASS	PASS	same	[]	[]
sample_705	PASS	FAIL	broken		[]
sample_714	PASS	PASS	same	[]	[]
sample_716	PASS	PASS	same	[]	[]
sample_728	PASS	PASS	same	''	''
sample_735	PASS	PASS	same	'Abb'	'Abb'
sample_759	PASS	FAIL	broken		[]
sample_761	PASS	PASS	same	[]	[]
sample_767	PASS	PASS	same	'-'	'-'
sample_769	PASS	PASS	same	'AKa?RIU'	'AKa?RIU'
sample_780	PASS	FAIL	broken		'2'
sample_787	PASS	PASS	same	'Xzd'	'Xzd'
sample_797	PASS	PASS	same	[('a', 1), ('b', 2), ('c', 3)]	[('a', 1), ('b', 2), ('c', 3)]

granite-4.0-h-small-FP8

Baseline: 75 / 184
Mitigated: 120 / 184
Change: +45

Fixed: 55
Broken: 10
Wrong under both: 54
Both right: 65

Scroll sideways to read the full table.

Problem	baseline	mitigated	outcome	predicted (mitigated)	expected
sample_101	PASS	PASS	same	[-4, 4, 1, 0]	[-4, 4, 1, 0]
sample_102	PASS	PASS	same	[]	[]
sample_105	FAIL	PASS	fixed	"Permission Is Granted"	'Permission Is Granted'
sample_106	FAIL	FAIL	both	[4, 16, -4, 18, 6, 6, 2, 8, -2, 9, 3, 3]	[4, 4, 4, 4, 4, 4, 2, 8, -2, 9, 3, 3]
sample_107	FAIL	PASS	fixed	'UA6HAJQ'	'UA6HAJQ'
sample_109	PASS	PASS	same	[9, 1, 0, 1, 1]	[9, 1, 0, 1, 1]
sample_111	PASS	PASS	same	(89, 4)	(89, 4)
sample_113	FAIL	FAIL	both	987	'987YhnShAShD 93275yRgsgBgssHfBsFB'
sample_114	FAIL	FAIL	both	['a', '.', 'b']	['a', '', 'b']
sample_116	PASS	PASS	same	{}	{}
sample_119	FAIL	PASS	fixed	"VsNlYgLtAw"	'VsNlYgLtAw'
sample_120	PASS	PASS	same	{}	{}
sample_121	FAIL	PASS	fixed	'1001'	'1001'
sample_123	PASS	PASS	same	[1, 2, 3, 5, 6, 8]	[1, 2, 3, 5, 6, 8]
sample_128	FAIL	FAIL	both	'Mmtoaoh t'	'Mmohamt'
sample_129	FAIL	FAIL	both	[17, 13, 9, 5, 1]	[28, 19, 12, 6]
sample_130	FAIL	FAIL	both	'h=6'	'h=l'
sample_132	FAIL	FAIL	both	'abcdabc'	'abc'
sample_133	FAIL	PASS	fixed	[7, 1, 2]	[7, 1, 2]
sample_135	PASS	PASS	same	['Russia', 'Kazakhstan']	['Russia', 'Kazakhstan']
sample_136	FAIL	FAIL	both	' a \n bc \n \n d \n ef '	' a \n bc \n \n d \n ef '
sample_137	FAIL	PASS	fixed	[]	[]
sample_139	FAIL	PASS	fixed	'no'	'no'
sample_140	FAIL	PASS	fixed	'Hey'	'Hey'
sample_141	FAIL	FAIL	both	[1, 3, 1, 1, 1, 1, 1, 1, 1]	[1, 3, 1, 3, 3, 1, 1, 1, 1, 1, 1]
sample_144	PASS	PASS	same	[]	[]
sample_146	PASS	PASS	same	[1, 2, 3, 4, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 6, 7, 8, 9, 10]
sample_147	PASS	PASS	same	[1, 1, 1]	[1, 1, 1]
sample_148	FAIL	FAIL	both	'2ioo 12 tfiqr.'	'2io 12 tfiqr.-'
sample_150	FAIL	PASS	fixed	[-2, 4, -4]	[-2, 4, -4]
sample_154	FAIL	PASS	fixed	'* There Hello'	'* There Hello'
sample_158	FAIL	PASS	fixed	[6, 4, -2, 6, 4, -2]	[6, 4, -2, 6, 4, -2]
sample_159	FAIL	FAIL	both	'mgITr'	'mgItr'
sample_162	FAIL	PASS	fixed	'СBISHOPSWIFT'	'СBISHOPSWIFT'
sample_164	PASS	PASS	same	[0, 1, 3]	[0, 1, 3]
sample_166	PASS	PASS	same	{}	{}
sample_167	FAIL	PASS	fixed	'aaQwQwQwbbQwQwQwccQwQwQwde'	'aaQwQwQwbbQwQwQwccQwQwQwde'
sample_169	FAIL	FAIL	both		'++++taole++++'
sample_171	FAIL	PASS	fixed	[1, 2, 3]	[1, 2, 3]
sample_172	PASS	PASS	same	[]	[]
sample_173	PASS	PASS	same	[4, 8, 6, 8, 5]	[4, 8, 6, 8, 5]
sample_174	PASS	PASS	same	[1, 3, 2]	[1, 3, 2]
sample_175	FAIL	FAIL	both	' '	' '
sample_177	FAIL	FAIL	both	'HeY dUdE tHiS $nD^ &*&tHiS@#'	'HEy Dude tHIs $Nd^ &*&tHiS@#'
sample_178	PASS	FAIL	broken	[2, 2, 2, 2]	[2, 2, 2]
sample_179	FAIL	FAIL	both	[2, 1, 2, 6, 0, 7, 1, 2, 6, 0, 2]	[2, 0, 6, 2, 1, 7, 1, 2, 6, 0, 2]
sample_180	PASS	FAIL	broken	[-1, 3, -2, -6, 8, 8]	[-1, -2, -6, 8, 8]
sample_181	FAIL	FAIL	both	["32912", 5]	['3291223', 7]
sample_182	PASS	PASS	same	[('a', 2), ('b', 1)]	[('a', 2), ('b', 1)]
sample_183	FAIL	PASS	fixed	['echo']	['echo']
sample_184	FAIL	PASS	fixed	[1, 2]	[1, 2]
sample_185	FAIL	FAIL	both	[11, 9, 7, 12, 14, 16]	[11, 14, 7, 12, 9, 16]
sample_186	FAIL	PASS	fixed	'pvtso'	'pvtso'
sample_188	PASS	PASS	same	["a"]	['a']
sample_189	FAIL	FAIL	both		'{{{{}}}}'
sample_193	FAIL	PASS	fixed	'1:1'	'1:1'
sample_194	PASS	PASS	same	[[], [5, 6, 2, 3], [1, 9, 5, 6]]	[[], [5, 6, 2, 3], [1, 9, 5, 6]]
sample_201	FAIL	FAIL	both	'645124'	'641524'
sample_202	PASS	PASS	same	[15, 15]	[15, 15]
sample_203	PASS	PASS	same	{}	{}
sample_204	FAIL	FAIL	both	['m', 'e']	['m', 'a']
sample_206	PASS	PASS	same	'h e l l o w o r l d!'	'h e l l o w o r l d!'
sample_208	PASS	FAIL	broken	['c', 'a', 't', ' ', 'd', 'e', 'e']	['c', 'a', 't', 'd', ' ', 'd', 'e', 'e']
sample_212	FAIL	PASS	fixed	[1, -9, 7, 2, 6, -3, 3]	[1, -9, 7, 2, 6, -3, 3]
sample_215	FAIL	PASS	fixed	')'	')'
sample_224	FAIL	PASS	fixed	{}	{}
sample_226	PASS	PASS	same	[1, 3, 3]	[1, 3, 3]
sample_227	FAIL	PASS	fixed	'Manolo'	'Manolo'
sample_228	FAIL	FAIL	both	'llthh#saf#lapkph#ts#wp'	'llthh#saflapkphtswp'
sample_229	FAIL	FAIL	both		['9', 'm', 'm', 'A', '1', '0', 'L', 'k']
sample_233	PASS	FAIL	broken	[2, 3, 1]	[1, 2, 3]
sample_235	PASS	PASS	same	[]	[]
sample_236	FAIL	FAIL	both	'ac8qk69wg'	'ac8qk6qk6'
sample_238	PASS	PASS	same	[1, 9, 4]	[1, 9, 4]
sample_239	FAIL	FAIL	both	't 1cos'	'1co'
sample_241	FAIL	PASS	fixed	'CW'	'CW'
sample_242	FAIL	FAIL	both		'udhv zcvi nhtnfyd :erwuyawa pun'
sample_245	FAIL	FAIL	both	['U', 'V', 'Z']	[]
sample_247	FAIL	PASS	fixed	"yes"	'yes'
sample_248	PASS	PASS	same	[666]	[666]
sample_251	FAIL	PASS	fixed	'Marie;Nelson;Oscar;+;3;5;3'	'Marie;Nelson;Oscar;+;3;5;3'
sample_254	FAIL	FAIL	both	'lower case'	'lwwer case'
sample_255	FAIL	PASS	fixed	'w'	'w'
sample_257	PASS	PASS	same	[['Hello World', '"I am String"']]	[['Hello World', '"I am String"']]
sample_258	FAIL	FAIL	both	[1, 7, 2, 3, 9]	[1, 2, 7, 3, 9]
sample_259	FAIL	FAIL	both	'SREXM'	'RES'
sample_260	PASS	PASS	same	[1, 2, 3, 4, 6, 5]	[1, 2, 3, 4, 6, 5]
sample_261	FAIL	FAIL	both	([], [12, 516, 5, 2, 3, 214, 51])	([], [12, 516, 5, 214, 51])
sample_262	FAIL	PASS	fixed	"BA"	'BA'
sample_263	PASS	PASS	same	['gloss', 'banana', 'barn', 'lawn']	['gloss', 'banana', 'barn', 'lawn']
sample_266	FAIL	PASS	fixed	[2, 3, 3, 4, 6, -2]	[2, 3, 3, 4, 6, -2]
sample_268	FAIL	FAIL	both	'h/grateful k'	'h / g r a t e f u l k'
sample_269	FAIL	PASS	fixed	['0', 2]	['0', 2]
sample_270	PASS	PASS	same	{}	{}
sample_271	FAIL	PASS	fixed	'uufh'	'uufh'
sample_272	FAIL	FAIL	both	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 2, 4, 6, 8, 0]	[9, 7, 5, 3, 1, 2, 4, 6, 8, 0, 2, 6, 0, 6, 6]
sample_273	FAIL	FAIL	both	'ET.'	'NET'
sample_276	PASS	PASS	same	[0]	[0]
sample_277	PASS	PASS	same	[4, 3, 2, 1]	[4, 3, 2, 1]
sample_279	PASS	PASS	same	''	''
sample_284	PASS	FAIL	broken	'est'	''
sample_286	PASS	PASS	same	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]	[1, 2, 3, 4, 11, 6, 7, 8, 9, 10]
sample_287	PASS	PASS	same	'pinneaple'	'pinneaple'
sample_288	FAIL	FAIL	both	[(1, 3), (12, 6), (71, 7)]	[(1, 3), (4, 555)]
sample_290	FAIL	FAIL	both	'bixaaaily'	'ABIXAAAILY'
sample_293	FAIL	PASS	fixed	'no'	'no'
sample_295	FAIL	PASS	fixed	['pear', 'banana', 'pear']	['pear', 'banana', 'pear']
sample_298	FAIL	FAIL	both		'DST VAVF N DMV DFVM GAMCU DGCVB.'
sample_300	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_301	FAIL	FAIL	both	[0, 6, 2, -1, -2, 0, 0, 6, 6, 2, 2, -1, -1, -2, -2]	[0, 6, 2, -1, -2, 6, 6, -2, -2, -2, -2, -2, -2]
sample_303	FAIL	FAIL	both	'mJkLbn'	'mJklbn'
sample_305	FAIL	FAIL	both		'o hoseto'
sample_306	PASS	PASS	same	[0, 6, 1, 2, 0]	[0, 6, 1, 2, 0]
sample_307	FAIL	FAIL	both	'p\|x\|c\|z\|n\|y\|f'	'px\|\|\|\|\|cznyf'
sample_310	FAIL	FAIL	both	'__10r0__jaa66'	'__1.00r__j_a6__6'
sample_311	FAIL	PASS	fixed	'no'	'no'
sample_316	PASS	PASS	same	'\| i am your father \|'	'\| i am your father \|'
sample_320	FAIL	PASS	fixed	'usAr'	'usAr'
sample_322	PASS	PASS	same	['lsi', 's', 't', 't', 'd']	['lsi', 's', 't', 't', 'd']
sample_324	PASS	PASS	same	[]	[]
sample_327	FAIL	PASS	fixed	[-3, 1, 7, -1]	[-3, 1, 7, -1]
sample_328	FAIL	PASS	fixed	[1, 2, 3, 1, 2, 3]	[1, 2, 3, 1, 2, 3]
sample_330	FAIL	PASS	fixed	' 4 2 '	' 4 2 '
sample_332	PASS	PASS	same	[]	[]
sample_335	FAIL	PASS	fixed	'sjbrfqmw'	'sjbrfqmw'
sample_337	FAIL	PASS	fixed	"LL"	'LL'
sample_340	FAIL	FAIL	both	'G'	' DEGHIVjkptx'
sample_341	PASS	PASS	same	{}	{}
sample_343	PASS	PASS	same	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]	[[1, 2, 3], [1, 2], 1, [1, 2, 3], 3, [2, 1]]
sample_344	PASS	PASS	same	[6, 4, 2, 8, 15]	[6, 4, 2, 8, 15]
sample_345	PASS	PASS	same	('mv', 'ml')	('mv', 'ml')
sample_347	FAIL	PASS	fixed	'hhhhhzcw'	'hhhhhzcw'
sample_350	PASS	PASS	same	[1, 2, 3]	[1, 2, 3]
sample_356	FAIL	FAIL	both	[1, 2]	[2, 1]
sample_357	FAIL	PASS	fixed	'werc'	'werc'
sample_358	FAIL	PASS	fixed	'tr'	'tr'
sample_359	FAIL	FAIL	both	[' dZwbSR ', ' wijHeq ', ' qluVok ', ' dxjxbF ']	['dZwbSR', 'wijHeq', 'qluVok', 'dxjxbF']
sample_360	FAIL	FAIL	both	'ggggggggggggggg'	'g'
sample_363	PASS	PASS	same	[1]	[1]
sample_364	PASS	PASS	same	[[3, False], [1, True]]	[[3, False], [1, True]]
sample_366	PASS	FAIL	broken	']'	''
sample_367	PASS	PASS	same	[6, 2, 1, 1, 4, 1]	[6, 2, 1, 1, 4, 1]
sample_368	FAIL	FAIL	both	'43 4327 4327 43 4327 4327'	'4327 00004327 000004327 4327 0004327 4327'
sample_372	FAIL	FAIL	both	['v,']	['']
sample_428	PASS	PASS	same	[]	[]
sample_429	PASS	FAIL	broken	[(5, 1), ('abc', 2), ('defghi', 2), (87.29, 3)]	[(87.29, 3), ('defghi', 2), ('abc', 2), (5, 1)]
sample_431	PASS	PASS	same	[]	[]
sample_435	FAIL	FAIL	both	'1'	''
sample_445	FAIL	FAIL	both	"Carrot, Banana, And Strawberry"	'Carrot, Banana, and Strawberry'
sample_446	PASS	PASS	same	[]	[]
sample_457	PASS	PASS	same	[]	[]
sample_480	PASS	PASS	same	''	''
sample_483	PASS	FAIL	broken	' '	' '
sample_503	PASS	PASS	same	[]	[]
sample_510	FAIL	PASS	fixed	'Wy'	'Wy'
sample_513	FAIL	PASS	fixed	[]	[]
sample_516	FAIL	FAIL	both	['condor']	[]
sample_522	PASS	PASS	same	[]	[]
sample_572	FAIL	FAIL	both	[(1, 9), (2, 10)]	[(2, 10), (3, 1), ((3, 1), None), ((2, 10), None)]
sample_579	PASS	PASS	same	''	''
sample_583	FAIL	PASS	fixed	't\nZA\nA'	't\nZA\nA'
sample_593	PASS	PASS	same	[]	[]
sample_600	PASS	PASS	same	[]	[]
sample_601	FAIL	FAIL	both	'ccccc sssss hhhhh rrrrr AAAAA ppppp'	'ccccc sssss hhhhh AAAAA rrrrr ppppp'
sample_623	FAIL	PASS	fixed	'HI~'	'HI~'
sample_663	PASS	PASS	same	[]	[]
sample_672	FAIL	PASS	fixed	'1zd'	'1zd'
sample_673	FAIL	PASS	fixed	"cA"	'cA'
sample_674	PASS	FAIL	broken	''	'qq'
sample_682	FAIL	PASS	fixed	'hy'	'hy'
sample_687	FAIL	FAIL	both	'R:j:u:g::z:u:f:E:r:j:u:g::n:z:u:f:e'	'R:j:u:g: :z:u:f:E:rjug nzufe'
sample_692	PASS	PASS	same	[]	[]
sample_705	PASS	FAIL	broken	['Somewhere Sydney', 'Somewhere Hong Kong', 'Somewhere Melbourne', 'Somewhere Sao Paolo', …	[]
sample_714	FAIL	PASS	fixed	[]	[]
sample_716	PASS	PASS	same	[]	[]
sample_728	FAIL	PASS	fixed	''	''
sample_735	FAIL	PASS	fixed	'Abb'	'Abb'
sample_759	PASS	PASS	same	[]	[]
sample_761	PASS	PASS	same	[]	[]
sample_767	FAIL	FAIL	both		'-'
sample_769	FAIL	PASS	fixed	'AKa?RIU'	'AKa?RIU'
sample_780	FAIL	PASS	fixed	'2'	'2'
sample_787	FAIL	PASS	fixed	'Xzd'	'Xzd'
sample_797	PASS	PASS	same	[('a', 1), ('b', 2), ('c', 3)]	[('a', 1), ('b', 2), ('c', 3)]