CRUXEval-O · Fieldnote

Can local models work out what Python code returns?

Seven models over 100 CRUXEval-O problems, with prompt, parser and infrastructure failures kept visible.

Julen Gamboa · 8 July 2026

What needed fixing before I trusted the scores

  • Gemma's recorded score changed from 80 to 98/100. The first run had 16 prompt failures where it reproduced def f(...) rather than an answer literal. A rerun used an explicit ###ANSWER### system message and a stronger prompt. The same message recovered answers buried in reasoning text for Ornith and Holo. These were changed prompt conditions, not a silent parser-only correction.
  • Qwen changed from 84 to 97/100 after a prompt repair. The original INSTR included a <python_literal> placeholder and a worked-example answer that Qwen copied. On roughly 6 difficult problems, its response could follow whichever answer-like text was most salient. The adopted prompt removed the metavariable, standalone answer line and worked result in answer position. Some residual failures still changed across prompt variants. That instability matters if the intended role is reviewing code.
  • The parser also needed a repair. parse_answer now removes trailing code-fence backticks and sentence punctuation appended after an otherwise correct literal by vLLM-served Qwen and Ornith. Re-evaluation changed only failures whose cleaned prediction actually equalled the expected answer. That is distinct from rerunning a model with a different prompt.
  • How to read the denominator. A score is ok / model_n. Prompt, infrastructure and harness failures are excluded from that denominator; genuine model-answer failures count against it. Each of the 7 model cards records per-problem fail_kind, and every card in this saved report has 0 exclusions. These are 100-problem Pass@1 results, not a repeated-run reliability estimate.
Nemotron-3-Super-120B-A12B-NVFP4
id: super-120b
98 / 100
98% · 100/100 processed · 13.8 t/s
show
Scroll sideways to read the full table.
ProblemResultPredictedExpected
sample_0PASS[(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)][(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)]
sample_1PASS{1: None, 2: None}{1: None, 2: None}
sample_2PASS'hbtofdeiequ''hbtofdeiequ'
sample_3PASSbcksrutq'bcksrutq'
sample_4PASS' '' '
sample_5PASS(0, 'xxxxxxxxxxxxxxxxxx')(0, 'xxxxxxxxxxxxxxxxxx')
sample_6PASS[('74', 31)][('74', 31)]
sample_7PASS[][]
sample_8PASS'UppEr''UppEr'
sample_9PASSFalseFalse
sample_10PASS''''
sample_11PASS{'foo': ['bar']}{'foo': ['bar']}
sample_12PASS'If you want to live a happy life! Daniel''If you want to live a happy life! Daniel'
sample_13PASS22
sample_14PASS'POO''POO'
sample_15PASS"ZN KGD JW LNT"'ZN KGD JW LNT'
sample_16PASS'zejrohaj''zejrohaj'
sample_17PASS99
sample_18PASS[3, 5, 4, 3, 2, 1, 0][3, 5, 4, 3, 2, 1, 0]
sample_19PASS''''
sample_20PASS',saw'',saw'
sample_21PASS[1, 1, 2, 2, 2][1, 1, 2, 2, 2]
sample_22PASS[0][0]
sample_23PASS'new-medium-performing-application - XQuery 2.''new-medium-performing-application - XQuery 2.'
sample_24PASS[45, 3, 61, 39, 27, 47][45, 3, 61, 39, 27, 47]
sample_25PASS{'l': 1, 't': 2}{'l': 1, 't': 2}
sample_26PASS'error''error'
sample_27PASSFalseFalse
sample_28PASSTrueTrue
sample_29PASS'123314''123314'
sample_30PASS['a', 'b', 'c']['a', 'b', 'c']
sample_31PASS88
sample_32PASS'*ume;*vo''*ume;*vo'
sample_33FAIL"{5:[5,5],2:[5,2,2],7:[5,2,2]}"'{5:[5,5],2:[5,2,2],7:[5,2,7]}'
sample_34PASS[2, 7, 7, 6, 8, 4, 2, 5, 21][2, 7, 7, 6, 8, 4, 2, 5, 21]
sample_35PASS[][]
sample_36PASS'ha''ha'
sample_37PASS['123', '23', '3']['123', '23', '3']
sample_38PASS'1Oe-ErrBzz-Bmm''1Oe-ErrBzz-Bmm'
sample_39PASS00
sample_40PASS"the cow goes moo#"'the cow goes moo#'
sample_41PASS[58, 92, 21][58, 92, 21]
sample_42PASS[][]
sample_43PASS-1-1
sample_44PASS'*+++n+z+o+h''*+++n+z+o+h'
sample_45PASS22
sample_46PASS'manylettersasvszhelloman''manylettersasvszhelloman'
sample_47PASSFalseFalse
sample_48PASS""''
sample_49PASS'816''816'
sample_50PASS[1][1]
sample_51PASS2020
sample_52PASS"seiq d"'seiq d'
sample_53PASS[1, 1, 1, 1, 1, 1, 1][1, 1, 1, 1, 1, 1, 1]
sample_54PASS11
sample_55PASS[89, 43, 17, 14, 8, 4][89, 43, 17, 14, 8, 4]
sample_56PASSTrueTrue
sample_57PASS11
sample_58PASS[-1, 0, 0, 1, 1, -1, 0, -1, 0, -1][-1, 0, 0, 1, 1, -1, 0, -1, 0, -1]
sample_59PASS'hi''hi'
sample_60PASS'R''R'
sample_61PASS'nada''nada'
sample_62PASS('ja', 'nee', 'coke', 'zoo')('ja', 'nee', 'coke', 'zoo')
sample_63PASS'dbtdabdahesyehu''dbtdabdahesyehu'
sample_64PASS' 7 '' 7 '
sample_65PASS99
sample_66FAIL'h'''
sample_67PASS'6,8,8''6,8,8'
sample_68PASS'dq''dq'
sample_69PASS'Name unknown''Name unknown'
sample_70PASS3030
sample_71PASS{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}
sample_72PASSTrueTrue
sample_73PASS(3, 6)(3, 6)
sample_74PASS[44, 34, 23, 82, 15, 24, 11, 63, 99][44, 34, 23, 82, 15, 24, 11, 63, 99]
sample_75PASS-22-22
sample_76PASS[6, 2, 0, 0, 0, 0, 0, 2, 3, 10][6, 2, 0, 0, 0, 0, 0, 2, 3, 10]
sample_77PASS''''
sample_78PASS'mty''mty'
sample_79PASS'1,2,3,4''1,2,3,4'
sample_80PASS'ba''ba'
sample_81PASS[('Bulls', 'bulls'), ('White Sox', 45)][('Bulls', 'bulls'), ('White Sox', 45)]
sample_82PASS'BFS''BFS'
sample_83PASS'2:0''2:0'
sample_84PASS"nwvday mefday ofmeday bdrylday"'nwvday mefday ofmeday bdrylday'
sample_85PASS[3, 4.5][3, 4.5]
sample_86PASS['sdfs', 'drcr', '2e']['sdfs', 'drcr', '2e']
sample_87PASS'-2139-1''-2139-1'
sample_88PASS"hello"'hello'
sample_89PASS'O''O'
sample_90PASS[[1, 2, 3], [], [1, 2, 3]][[1, 2, 3], [], [1, 2, 3]]
sample_91PASS['1', '2', 'a', 'b', '3', 'x', 'y']['1', '2', 'a', 'b', '3', 'x', 'y']
sample_92PASSFalseFalse
sample_93PASS'iq!!!!''iq!!!!'
sample_94PASS{'w': 3, 'wi': 10}{'w': 3, 'wi': 10}
sample_95PASS{'fr': 'AAA'}{'fr': 'AAA'}
sample_96PASSTrueTrue
sample_97PASSTrueTrue
sample_98PASS11
sample_99PASS'aa++___bb''aa++___bb'
Nemotron-3-Nano-30B-A3B-NVFP4
id: nemotron-30b
96 / 100
96% · 100/100 processed · 51.9 t/s
show
Scroll sideways to read the full table.
ProblemResultPredictedExpected
sample_0PASS[(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)][(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)]
sample_1PASS{1: None, 2: None}{1: None, 2: None}
sample_2PASS'hbtofdeiequ''hbtofdeiequ'
sample_3FAIL'bckrutq''bcksrutq'
sample_4PASS' '' '
sample_5PASS(0, 'xxxxxxxxxxxxxxxxxx')(0, 'xxxxxxxxxxxxxxxxxx')
sample_6PASS[('74', 31)][('74', 31)]
sample_7PASS[][]
sample_8PASS'UppEr''UppEr'
sample_9PASSFalseFalse
sample_10PASS''''
sample_11PASS{'foo': ['bar']}{'foo': ['bar']}
sample_12PASS'If you want to live a happy life! Daniel''If you want to live a happy life! Daniel'
sample_13PASS22
sample_14PASS'POO''POO'
sample_15PASS"ZN KGD JW LNT"'ZN KGD JW LNT'
sample_16PASS'zejrohaj''zejrohaj'
sample_17PASS99
sample_18PASS[3, 5, 4, 3, 2, 1, 0][3, 5, 4, 3, 2, 1, 0]
sample_19PASS""''
sample_20PASS',saw'',saw'
sample_21PASS[1, 1, 2, 2, 2][1, 1, 2, 2, 2]
sample_22PASS[0][0]
sample_23PASS'new-medium-performing-application - XQuery 2.''new-medium-performing-application - XQuery 2.'
sample_24PASS[45, 3, 61, 39, 27, 47][45, 3, 61, 39, 27, 47]
sample_25PASS{'l': 1, 't': 2}{'l': 1, 't': 2}
sample_26PASS'error''error'
sample_27PASSFalseFalse
sample_28PASSTrueTrue
sample_29PASS'123314''123314'
sample_30PASS["a", "b", "c"]['a', 'b', 'c']
sample_31PASS88
sample_32FAIL"' *ume;*vo'"'*ume;*vo'
sample_33PASS'{5:[5,5],2:[5,2,2],7:[5,2,7]}''{5:[5,5],2:[5,2,2],7:[5,2,7]}'
sample_34PASS[2, 7, 7, 6, 8, 4, 2, 5, 21][2, 7, 7, 6, 8, 4, 2, 5, 21]
sample_35PASS[][]
sample_36PASS'ha''ha'
sample_37PASS['123', '23', '3']['123', '23', '3']
sample_38FAIL'1oe-errBzz-bmm''1Oe-ErrBzz-Bmm'
sample_39PASS00
sample_40PASS"the cow goes moo#"'the cow goes moo#'
sample_41PASS[58, 92, 21][58, 92, 21]
sample_42PASS[][]
sample_43PASS-1-1
sample_44PASS"*+++n+z+o+h"'*+++n+z+o+h'
sample_45PASS22
sample_46PASS'manylettersasvszhelloman''manylettersasvszhelloman'
sample_47PASSFalseFalse
sample_48PASS""''
sample_49PASS'816''816'
sample_50PASS[1][1]
sample_51PASS2020
sample_52PASS"seiq d"'seiq d'
sample_53PASS[1, 1, 1, 1, 1, 1, 1][1, 1, 1, 1, 1, 1, 1]
sample_54PASS11
sample_55PASS[89, 43, 17, 14, 8, 4][89, 43, 17, 14, 8, 4]
sample_56PASSTrueTrue
sample_57PASS11
sample_58PASS[-1, 0, 0, 1, 1, -1, 0, -1, 0, -1][-1, 0, 0, 1, 1, -1, 0, -1, 0, -1]
sample_59PASS'hi''hi'
sample_60PASS'R''R'
sample_61PASS'nada''nada'
sample_62PASS('ja', 'nee', 'coke', 'zoo')('ja', 'nee', 'coke', 'zoo')
sample_63PASS'dbtdabdahesyehu''dbtdabdahesyehu'
sample_64PASS' 7 '' 7 '
sample_65PASS99
sample_66PASS''''
sample_67PASS'6,8,8''6,8,8'
sample_68PASS'dq''dq'
sample_69PASS'Name unknown''Name unknown'
sample_70PASS3030
sample_71PASS{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}
sample_72PASSTrueTrue
sample_73PASS(3, 6)(3, 6)
sample_74PASS[44, 34, 23, 82, 15, 24, 11, 63, 99][44, 34, 23, 82, 15, 24, 11, 63, 99]
sample_75PASS-22-22
sample_76PASS[6, 2, 0, 0, 0, 0, 0, 2, 3, 10][6, 2, 0, 0, 0, 0, 0, 2, 3, 10]
sample_77PASS''''
sample_78PASSmty'mty'
sample_79PASS'1,2,3,4''1,2,3,4'
sample_80FAILBetter approach: The string literal 'ab ' includes 'ab' then eight spaces? Let's co…'ba'
sample_81PASS[('Bulls', 'bulls'), ('White Sox', 45)][('Bulls', 'bulls'), ('White Sox', 45)]
sample_82PASS'BFS''BFS'
sample_83PASS'2:0''2:0'
sample_84PASS"nwvday mefday ofmeday bdrylday"'nwvday mefday ofmeday bdrylday'
sample_85PASS[3, 4.5][3, 4.5]
sample_86PASS['sdfs', 'drcr', '2e']['sdfs', 'drcr', '2e']
sample_87PASS'-2139-1''-2139-1'
sample_88PASS"hello"'hello'
sample_89PASS'O''O'
sample_90PASS[[1, 2, 3], [], [1, 2, 3]][[1, 2, 3], [], [1, 2, 3]]
sample_91PASS['1', '2', 'a', 'b', '3', 'x', 'y']['1', '2', 'a', 'b', '3', 'x', 'y']
sample_92PASSFalseFalse
sample_93PASS'iq!!!!''iq!!!!'
sample_94PASS{'w': 3, 'wi': 10}{'w': 3, 'wi': 10}
sample_95PASS{'fr':'AAA'}{'fr': 'AAA'}
sample_96PASSTrueTrue
sample_97PASSTrueTrue
sample_98PASS11
sample_99PASS'aa++___bb''aa++___bb'
gemma-4-26B-A4B-it
id: gemma-4-26b
98 / 100
98% · 100/100 processed · 23.1 t/s
show
Scroll sideways to read the full table.
ProblemResultPredictedExpected
sample_0PASS[(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)][(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)]
sample_1PASS{1: None, 2: None}{1: None, 2: None}
sample_2PASS'hbtofdeiequ''hbtofdeiequ'
sample_3PASS'bcksrutq''bcksrutq'
sample_4PASS' '' '
sample_5PASS(0, 'xxxxxxxxxxxxxxxxxx')(0, 'xxxxxxxxxxxxxxxxxx')
sample_6PASS[('74', 31)][('74', 31)]
sample_7PASS[][]
sample_8PASS'UppEr''UppEr'
sample_9PASSFalseFalse
sample_10PASS''''
sample_11PASS{'foo': ['bar']}{'foo': ['bar']}
sample_12PASS'If you want to live a happy life! Daniel''If you want to live a happy life! Daniel'
sample_13PASS22
sample_14PASS'POO''POO'
sample_15PASS'ZN KGD JW LNT''ZN KGD JW LNT'
sample_16PASS'zejrohaj''zejrohaj'
sample_17PASS99
sample_18PASS[3, 5, 4, 3, 2, 1, 0][3, 5, 4, 3, 2, 1, 0]
sample_19PASS""''
sample_20PASS',saw'',saw'
sample_21PASS[1, 1, 2, 2, 2][1, 1, 2, 2, 2]
sample_22PASS[0][0]
sample_23PASS'new-medium-performing-application - XQuery 2.''new-medium-performing-application - XQuery 2.'
sample_24PASS[45, 3, 61, 39, 27, 47][45, 3, 61, 39, 27, 47]
sample_25PASS{'l': 1, 't': 2}{'l': 1, 't': 2}
sample_26PASS'error''error'
sample_27PASSFalseFalse
sample_28PASSTrueTrue
sample_29PASS'123314''123314'
sample_30PASS["a", "b", "c"]['a', 'b', 'c']
sample_31PASS88
sample_32PASS'*ume;*vo''*ume;*vo'
sample_33FAIL'{5:[5,5],2:[5,2,2],7:[5,2,7],3:[5,7,2,3]}''{5:[5,5],2:[5,2,2],7:[5,2,7]}'
sample_34PASS[2, 7, 7, 6, 8, 4, 2, 5, 21][2, 7, 7, 6, 8, 4, 2, 5, 21]
sample_35PASS[][]
sample_36PASS'ha''ha'
sample_37PASS['123', '23', '3']['123', '23', '3']
sample_38PASS'1Oe-ErrBzz-Bmm''1Oe-ErrBzz-Bmm'
sample_39PASS00
sample_40PASS'the cow goes moo#''the cow goes moo#'
sample_41PASS[58, 92, 21][58, 92, 21]
sample_42PASS[][]
sample_43PASS-1-1
sample_44PASS'*+++n+z+o+h''*+++n+z+o+h'
sample_45PASS22
sample_46PASS'manylettersasvszhelloman''manylettersasvszhelloman'
sample_47PASSFalseFalse
sample_48PASS""''
sample_49PASS'816''816'
sample_50PASS[1][1]
sample_51PASS2020
sample_52PASS'seiq d''seiq d'
sample_53PASS[1, 1, 1, 1, 1, 1, 1][1, 1, 1, 1, 1, 1, 1]
sample_54PASS11
sample_55PASS[89, 43, 17, 14, 8, 4][89, 43, 17, 14, 8, 4]
sample_56PASSTrueTrue
sample_57PASS11
sample_58PASS[-1, 0, 0, 1, 1, -1, 0, -1, 0, -1][-1, 0, 0, 1, 1, -1, 0, -1, 0, -1]
sample_59PASS'hi''hi'
sample_60PASS'R''R'
sample_61PASS'nada''nada'
sample_62PASS('ja', 'nee', 'coke', 'zoo')('ja', 'nee', 'coke', 'zoo')
sample_63PASS'dbtdabdahesyehu''dbtdabdahesyehu'
sample_64PASS' 7 '' 7 '
sample_65PASS99
sample_66FAIL'happ'''
sample_67PASS'6,8,8''6,8,8'
sample_68PASS'dq''dq'
sample_69PASS'Name unknown''Name unknown'
sample_70PASS3030
sample_71PASS{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}
sample_72PASSTrueTrue
sample_73PASS(3, 6)(3, 6)
sample_74PASS[44, 34, 23, 82, 15, 24, 11, 63, 99][44, 34, 23, 82, 15, 24, 11, 63, 99]
sample_75PASS-22-22
sample_76PASS[6, 2, 0, 0, 0, 0, 0, 2, 3, 10][6, 2, 0, 0, 0, 0, 0, 2, 3, 10]
sample_77PASS''''
sample_78PASS'mty''mty'
sample_79PASS'1,2,3,4''1,2,3,4'
sample_80PASS'ba''ba'
sample_81PASS[('Bulls', 'bulls'), ('White Sox', 45)][('Bulls', 'bulls'), ('White Sox', 45)]
sample_82PASS'BFS''BFS'
sample_83PASS'2:0''2:0'
sample_84PASS'nwvday mefday ofmeday bdrylday''nwvday mefday ofmeday bdrylday'
sample_85PASS[3, 4.5][3, 4.5]
sample_86PASS['sdfs', 'drcr', '2e']['sdfs', 'drcr', '2e']
sample_87PASS'-2139-1''-2139-1'
sample_88PASS'hello''hello'
sample_89PASS'O''O'
sample_90PASS[[1, 2, 3], [], [1, 2, 3]][[1, 2, 3], [], [1, 2, 3]]
sample_91PASS['1', '2', 'a', 'b', '3', 'x', 'y']['1', '2', 'a', 'b', '3', 'x', 'y']
sample_92PASSFalseFalse
sample_93PASS'iq!!!!''iq!!!!'
sample_94PASS{'w': 3, 'wi': 10}{'w': 3, 'wi': 10}
sample_95PASS{'fr': 'AAA'}{'fr': 'AAA'}
sample_96PASSTrueTrue
sample_97PASSTrueTrue
sample_98PASS11
sample_99PASS'aa++___bb''aa++___bb'
Ornith-1.0-35B-FP8
id: ornith-fp8
94 / 100
94% · 100/100 processed · 34.3 t/s
show
Scroll sideways to read the full table.
ProblemResultPredictedExpected
sample_0PASS[(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)][(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)]
sample_1PASS{1: None, 2: None}{1: None, 2: None}
sample_2PASS'hbtofdeiequ''hbtofdeiequ'
sample_3PASS'bcksrutq''bcksrutq'
sample_4PASS' '' '
sample_5PASS(0, 'xxxxxxxxxxxxxxxxxx')(0, 'xxxxxxxxxxxxxxxxxx')
sample_6PASS[('74', 31)][('74', 31)]
sample_7PASS[][]
sample_8PASS'UppEr''UppEr'
sample_9PASSFalseFalse
sample_10PASS''''
sample_11PASS{'foo': ['bar']}{'foo': ['bar']}
sample_12PASS'If you want to live a happy life! Daniel''If you want to live a happy life! Daniel'
sample_13PASS22
sample_14PASS'POO''POO'
sample_15PASS"ZN KGD JW LNT"'ZN KGD JW LNT'
sample_16PASS'zejrohaj''zejrohaj'
sample_17PASS99
sample_18PASS[3, 5, 4, 3, 2, 1, 0][3, 5, 4, 3, 2, 1, 0]
sample_19PASS""''
sample_20PASS',saw'',saw'
sample_21PASS[1, 1, 2, 2, 2][1, 1, 2, 2, 2]
sample_22PASS[0][0]
sample_23PASS'new-medium-performing-application - XQuery 2.''new-medium-performing-application - XQuery 2.'
sample_24PASS[45, 3, 61, 39, 27, 47][45, 3, 61, 39, 27, 47]
sample_25PASS{"l": 1, "t": 2}{'l': 1, 't': 2}
sample_26PASS'error''error'
sample_27PASSFalseFalse
sample_28PASSTrueTrue
sample_29PASS'123314''123314'
sample_30PASS["a", "b", "c"]['a', 'b', 'c']
sample_31PASS88
sample_32PASS'*ume;*vo''*ume;*vo'
sample_33FAIL1). dic[5]=[5,5,5]'{5:[5,5],2:[5,2,2],7:[5,2,7]}'
sample_34PASS[2, 7, 7, 6, 8, 4, 2, 5, 21][2, 7, 7, 6, 8, 4, 2, 5, 21]
sample_35PASS[][]
sample_36PASS'ha''ha'
sample_37PASS['123', '23', '3']['123', '23', '3']
sample_38FAIL'1Oe-ErBzz-Bmm''1Oe-ErrBzz-Bmm'
sample_39PASS00
sample_40PASS"the cow goes moo#"'the cow goes moo#'
sample_41PASS[58, 92, 21][58, 92, 21]
sample_42PASS[][]
sample_43PASS-1-1
sample_44FAIL'*++n+z+o+h''*+++n+z+o+h'
sample_45PASS22
sample_46PASS'manylettersasvszhelloman''manylettersasvszhelloman'
sample_47PASSFalseFalse
sample_48PASS""''
sample_49PASS'816''816'
sample_50PASS[1][1]
sample_51PASS2020
sample_52PASS'seiq d''seiq d'
sample_53PASS[1, 1, 1, 1, 1, 1, 1][1, 1, 1, 1, 1, 1, 1]
sample_54PASS11
sample_55PASS[89, 43, 17, 14, 8, 4][89, 43, 17, 14, 8, 4]
sample_56PASSTrueTrue
sample_57PASS11
sample_58PASS[-1, 0, 0, 1, 1, -1, 0, -1, 0, -1][-1, 0, 0, 1, 1, -1, 0, -1, 0, -1]
sample_59PASS'hi''hi'
sample_60PASS'R''R'
sample_61PASS'nada''nada'
sample_62PASS('ja', 'nee', 'coke', 'zoo')('ja', 'nee', 'coke', 'zoo')
sample_63PASS'dbtdabdahesyehu''dbtdabdahesyehu'
sample_64PASS' 7 '' 7 '
sample_65PASS99
sample_66FAIL'h'''
sample_67PASS'6,8,8''6,8,8'
sample_68FAIL"'dq'
sample_69PASS'Name unknown''Name unknown'
sample_70PASS3030
sample_71PASS{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}
sample_72PASSTrueTrue
sample_73PASS(3, 6)(3, 6)
sample_74PASS[44, 34, 23, 82, 15, 24, 11, 63, 99][44, 34, 23, 82, 15, 24, 11, 63, 99]
sample_75PASS-22-22
sample_76PASS[6, 2, 0, 0, 0, 0, 0, 2, 3, 10][6, 2, 0, 0, 0, 0, 0, 2, 3, 10]
sample_77PASS''''
sample_78PASS'mty''mty'
sample_79PASS'1,2,3,4''1,2,3,4'
sample_80PASS'ba''ba'
sample_81PASS[('Bulls', 'bulls'), ('White Sox', 45)][('Bulls', 'bulls'), ('White Sox', 45)]
sample_82PASS'BFS''BFS'
sample_83PASS'2:0'`.'2:0'
sample_84PASS"nwvday mefday ofmeday bdrylday"'nwvday mefday ofmeday bdrylday'
sample_85PASS[3, 4.5][3, 4.5]
sample_86PASS['sdfs', 'drcr', '2e']['sdfs', 'drcr', '2e']
sample_87PASS'-2139-1''-2139-1'
sample_88PASS"hello"'hello'
sample_89PASS'O''O'
sample_90PASS[[1, 2, 3], [], [1, 2, 3]][[1, 2, 3], [], [1, 2, 3]]
sample_91PASS['1', '2', 'a', 'b', '3', 'x', 'y']['1', '2', 'a', 'b', '3', 'x', 'y']
sample_92PASSFalseFalse
sample_93PASS'iq!!!!''iq!!!!'
sample_94PASS{'w': 3, 'wi': 10}{'w': 3, 'wi': 10}
sample_95PASS{'fr': 'AAA'}{'fr': 'AAA'}
sample_96PASSTrueTrue
sample_97PASSTrueTrue
sample_98PASS11
sample_99FAIL'aa+++___bb''aa++___bb'
Holo-3.1-35B-A3B-NVFP4
id: holo-3.1-35b
88 / 100
88% · 100/100 processed · 66.3 t/s
show
Scroll sideways to read the full table.
ProblemResultPredictedExpected
sample_0PASS[(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)][(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)]
sample_1PASS{1: None, 2: None}{1: None, 2: None}
sample_2PASShbtofdeiequ'hbtofdeiequ'
sample_3PASS'bcksrutq''bcksrutq'
sample_4PASS' '' '
sample_5FAIL(3, 'xxxxxxxxxxxxxxxxxxxx')(0, 'xxxxxxxxxxxxxxxxxx')
sample_6PASS[('74', 31)][('74', 31)]
sample_7PASS[][]
sample_8PASSUppEr'UppEr'
sample_9PASSFalseFalse
sample_10PASS''''
sample_11FAIL[{'foo': ['bar']}]{'foo': ['bar']}
sample_12PASS'If you want to live a happy life! Daniel''If you want to live a happy life! Daniel'
sample_13PASS22
sample_14PASS'POO''POO'
sample_15PASS"ZN KGD JW LNT"'ZN KGD JW LNT'
sample_16PASS'zejrohaj''zejrohaj'
sample_17PASS99
sample_18PASS[3, 5, 4, 3, 2, 1, 0][3, 5, 4, 3, 2, 1, 0]
sample_19PASS''
sample_20PASS',saw'',saw'
sample_21PASS[1, 1, 2, 2, 2][1, 1, 2, 2, 2]
sample_22FAIL0[0]
sample_23FAIL'new-medium-performing-application - XQuery 2''new-medium-performing-application - XQuery 2.'
sample_24PASS[45, 3, 61, 39, 27, 47][45, 3, 61, 39, 27, 47]
sample_25PASS{"l": 1, "t": 2}{'l': 1, 't': 2}
sample_26PASSerror'error'
sample_27PASSFalseFalse
sample_28PASSTrueTrue
sample_29PASS'123314''123314'
sample_30PASS['a', 'b', 'c']['a', 'b', 'c']
sample_31PASS88
sample_32PASS'*ume;*vo''*ume;*vo'
sample_33FAIL{5:[5,5,5],2:[5,2,2],7:[5,2,7],3:[5,7,3]}'{5:[5,5],2:[5,2,2],7:[5,2,7]}'
sample_34FAIL [2, 7, 7, 6, 8, 4, 5, 21][2, 7, 7, 6, 8, 4, 2, 5, 21]
sample_35PASS[][]
sample_36PASS'ha''ha'
sample_37PASS['123', '23', '3']['123', '23', '3']
sample_38FAIL'1Oe-errBzz-bmm''1Oe-ErrBzz-Bmm'
sample_39PASS00
sample_40PASS"the cow goes moo#"'the cow goes moo#'
sample_41PASS[58, 92, 21][58, 92, 21]
sample_42PASS[][]
sample_43PASS-1-1
sample_44FAIL'+*n+z+o+h''*+++n+z+o+h'
sample_45PASS22
sample_46PASSmanylettersasvszhelloman'manylettersasvszhelloman'
sample_47PASSFalseFalse
sample_48PASS""''
sample_49PASS'816''816'
sample_50PASS[1][1]
sample_51PASS2020
sample_52PASS"seiq d"'seiq d'
sample_53PASS[1, 1, 1, 1, 1, 1, 1][1, 1, 1, 1, 1, 1, 1]
sample_54PASS11
sample_55PASS[89, 43, 17, 14, 8, 4][89, 43, 17, 14, 8, 4]
sample_56PASSTrueTrue
sample_57PASS11
sample_58PASS[-1, 0, 0, 1, 1, -1, 0, -1, 0, -1][-1, 0, 0, 1, 1, -1, 0, -1, 0, -1]
sample_59PASS'hi''hi'
sample_60PASSR'R'
sample_61PASS'nada''nada'
sample_62PASS('ja', 'nee', 'coke', 'zoo')('ja', 'nee', 'coke', 'zoo')
sample_63PASS'dbtdabdahesyehu''dbtdabdahesyehu'
sample_64PASS' 7 '' 7 '
sample_65PASS99
sample_66PASS''''
sample_67PASS'6,8,8''6,8,8'
sample_68FAIL'wpvr.dq.omeunh''dq'
sample_69PASS 'Name unknown''Name unknown'
sample_70PASS3030
sample_71PASS{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}
sample_72PASSTrueTrue
sample_73PASS(3, 6)(3, 6)
sample_74PASS[44, 34, 23, 82, 15, 24, 11, 63, 99][44, 34, 23, 82, 15, 24, 11, 63, 99]
sample_75PASS-22-22
sample_76PASS[6, 2, 0, 0, 0, 0, 0, 2, 3, 10][6, 2, 0, 0, 0, 0, 0, 2, 3, 10]
sample_77PASS''''
sample_78FAIL'mtw''mty'
sample_79PASS'1,2,3,4''1,2,3,4'
sample_80PASS'ba''ba'
sample_81FAIL[('bulls', 23), ('White Sox', 45)][('Bulls', 'bulls'), ('White Sox', 45)]
sample_82PASS'BFS''BFS'
sample_83PASS'2:0''2:0'
sample_84PASS"nwvday mefday ofmeday bdrylday"'nwvday mefday ofmeday bdrylday'
sample_85PASS[3, 4.5][3, 4.5]
sample_86PASS['sdfs', 'drcr', '2e']['sdfs', 'drcr', '2e']
sample_87PASS'-2139-1''-2139-1'
sample_88FAIL"hel"'hello'
sample_89PASS'O''O'
sample_90PASS[[1, 2, 3], [], [1, 2, 3]][[1, 2, 3], [], [1, 2, 3]]
sample_91PASS['1', '2', 'a', 'b', '3', 'x', 'y']['1', '2', 'a', 'b', '3', 'x', 'y']
sample_92PASSFalseFalse
sample_93PASS'iq!!!!''iq!!!!'
sample_94PASS{'w': 3, 'wi': 10}{'w': 3, 'wi': 10}
sample_95PASS{'fr': 'AAA'}{'fr': 'AAA'}
sample_96PASSTrueTrue
sample_97PASSTrueTrue
sample_98PASS11
sample_99PASS'aa++___bb''aa++___bb'
Qwen3.6-35B-A3B-NVFP4
id: qwen3.6-35b
97 / 100
97% · 100/100 processed · 74.6 t/s
show
Scroll sideways to read the full table.
ProblemResultPredictedExpected
sample_0PASS[(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)][(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)]
sample_1PASS{1: None, 2: None}{1: None, 2: None}
sample_2PASS'hbtofdeiequ''hbtofdeiequ'
sample_3PASS'bcksrutq''bcksrutq'
sample_4PASS' '' '
sample_5PASS(0, 'xxxxxxxxxxxxxxxxxx')(0, 'xxxxxxxxxxxxxxxxxx')
sample_6PASS[('74', 31)]`.[('74', 31)]
sample_7PASS[][]
sample_8PASS'UppEr''UppEr'
sample_9PASSFalseFalse
sample_10PASS''''
sample_11PASS{'foo': ['bar']}{'foo': ['bar']}
sample_12PASSIf you want to live a happy life! Daniel'If you want to live a happy life! Daniel'
sample_13PASS22
sample_14PASS'POO''POO'
sample_15PASS"ZN KGD JW LNT"'ZN KGD JW LNT'
sample_16PASS'zejrohaj''zejrohaj'
sample_17PASS99
sample_18PASS[3, 5, 4, 3, 2, 1, 0][3, 5, 4, 3, 2, 1, 0]
sample_19PASS""''
sample_20PASS',saw'',saw'
sample_21PASS[1, 1, 2, 2, 2][1, 1, 2, 2, 2]
sample_22PASS[0]`.[0]
sample_23PASS'new-medium-performing-application - XQuery 2.''new-medium-performing-application - XQuery 2.'
sample_24PASS[45, 3, 61, 39, 27, 47][45, 3, 61, 39, 27, 47]
sample_25PASS{'l': 1, 't': 2}{'l': 1, 't': 2}
sample_26FAILand is followed by the exact Python literal..."'error'
sample_27PASSFalseFalse
sample_28PASSTrueTrue
sample_29PASS'123314''123314'
sample_30PASS['a', 'b', 'c']['a', 'b', 'c']
sample_31PASS88
sample_32PASS'*ume;*vo''*ume;*vo'
sample_33PASS"{5:[5,5],2:[5,2,2],7:[5,2,7]}"'{5:[5,5],2:[5,2,2],7:[5,2,7]}'
sample_34PASS[2, 7, 7, 6, 8, 4, 2, 5, 21][2, 7, 7, 6, 8, 4, 2, 5, 21]
sample_35PASS[]`.[]
sample_36PASS'ha''ha'
sample_37PASS['123', '23', '3']['123', '23', '3']
sample_38PASS'1Oe-ErrBzz-Bmm''1Oe-ErrBzz-Bmm'
sample_39PASS00
sample_40PASS"the cow goes moo#"'the cow goes moo#'
sample_41PASS[58, 92, 21][58, 92, 21]
sample_42PASS[][]
sample_43PASS-1-1
sample_44PASS"*+++n+z+o+h"'*+++n+z+o+h'
sample_45PASS22
sample_46PASS'manylettersasvszhelloman''manylettersasvszhelloman'
sample_47PASSFalseFalse
sample_48PASS""''
sample_49PASS'816''816'
sample_50PASS[1][1]
sample_51PASS2020
sample_52PASS"seiq d"'seiq d'
sample_53FAILand is followed by the exact Python literal..."[1, 1, 1, 1, 1, 1, 1]
sample_54PASS11
sample_55PASS[89, 43, 17, 14, 8, 4][89, 43, 17, 14, 8, 4]
sample_56PASSTrueTrue
sample_57PASS11
sample_58PASS[-1, 0, 0, 1, 1, -1, 0, -1, 0, -1][-1, 0, 0, 1, 1, -1, 0, -1, 0, -1]
sample_59PASS'hi''hi'
sample_60PASS'R''R'
sample_61PASS'nada''nada'
sample_62PASS('ja', 'nee', 'coke', 'zoo')('ja', 'nee', 'coke', 'zoo')
sample_63PASS'dbtdabdahesyehu''dbtdabdahesyehu'
sample_64PASS" 7 "' 7 '
sample_65PASS99
sample_66PASS''''
sample_67PASS'6,8,8''6,8,8'
sample_68PASS'dq''dq'
sample_69PASS'Name unknown''Name unknown'
sample_70PASS3030
sample_71PASS{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}
sample_72PASSTrueTrue
sample_73PASS(3, 6)(3, 6)
sample_74PASS[44, 34, 23, 82, 15, 24, 11, 63, 99][44, 34, 23, 82, 15, 24, 11, 63, 99]
sample_75PASS-22-22
sample_76PASS[6, 2, 0, 0, 0, 0, 0, 2, 3, 10][6, 2, 0, 0, 0, 0, 0, 2, 3, 10]
sample_77PASS''''
sample_78PASS'mty''mty'
sample_79PASS'1,2,3,4''1,2,3,4'
sample_80PASS'ba''ba'
sample_81PASS[('Bulls', 'bulls'), ('White Sox', 45)][('Bulls', 'bulls'), ('White Sox', 45)]
sample_82PASS'BFS''BFS'
sample_83PASS'2:0'`.'2:0'
sample_84PASS"nwvday mefday ofmeday bdrylday"'nwvday mefday ofmeday bdrylday'
sample_85PASS[3, 4.5][3, 4.5]
sample_86PASS['sdfs', 'drcr', '2e']['sdfs', 'drcr', '2e']
sample_87PASS'-2139-1''-2139-1'
sample_88PASS"hello"'hello'
sample_89PASS'O''O'
sample_90PASS[[1, 2, 3], [], [1, 2, 3]][[1, 2, 3], [], [1, 2, 3]]
sample_91PASS['1', '2', 'a', 'b', '3', 'x', 'y']['1', '2', 'a', 'b', '3', 'x', 'y']
sample_92PASSFalseFalse
sample_93PASS'iq!!!!''iq!!!!'
sample_94PASS{'w': 3, 'wi': 10}{'w': 3, 'wi': 10}
sample_95PASS{'fr': 'AAA'}{'fr': 'AAA'}
sample_96PASSTrueTrue
sample_97PASSTrueTrue
sample_98PASS11
sample_99FAIL'aa+++___bb'`? No.'aa++___bb'
granite-4.0-h-small-FP8
id: granite-small
81 / 100
81% · 100/100 processed · 19.6 t/s
show
Scroll sideways to read the full table.
ProblemResultPredictedExpected
sample_0PASS[(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)][(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)]
sample_1PASS{1: None, 2: None}{1: None, 2: None}
sample_2PASS'hbtofdeiequ''hbtofdeiequ'
sample_3PASS'bcksrutq''bcksrutq'
sample_4FAIL' '' '
sample_5FAIL(2, 'xxxxxxxxxxxxxxxxxxxx')(0, 'xxxxxxxxxxxxxxxxxx')
sample_6FAIL[('65', 34)][('74', 31)]
sample_7PASS[][]
sample_8PASS'UppEr''UppEr'
sample_9PASSFalseFalse
sample_10PASS''''
sample_11PASS{'foo': ['bar']}{'foo': ['bar']}
sample_12PASS'If you want to live a happy life! Daniel''If you want to live a happy life! Daniel'
sample_13PASS22
sample_14PASS'POO''POO'
sample_15PASS'ZN KGD JW LNT''ZN KGD JW LNT'
sample_16PASS'zejrohaj''zejrohaj'
sample_17FAIL89
sample_18PASS[3, 5, 4, 3, 2, 1, 0][3, 5, 4, 3, 2, 1, 0]
sample_19PASS""''
sample_20PASS',saw'',saw'
sample_21FAIL[1, 1, 2, 2, 2, 2][1, 1, 2, 2, 2]
sample_22PASS[0][0]
sample_23FAIL'new-medium-performing-application - XQuery ''new-medium-performing-application - XQuery 2.'
sample_24PASS[45, 3, 61, 39, 27, 47][45, 3, 61, 39, 27, 47]
sample_25PASS{"l": 1, "t": 2}{'l': 1, 't': 2}
sample_26PASS'error''error'
sample_27PASSFalseFalse
sample_28PASSTrueTrue
sample_29PASS'123314''123314'
sample_30PASS["a", "b", "c"]['a', 'b', 'c']
sample_31PASS88
sample_32FAIL'*ume;*v''*ume;*vo'
sample_33FAIL'{5:[5,5],2:[5,2,2],7:[5,2,7],3:[5,7,2,3]}''{5:[5,5],2:[5,2,2],7:[5,2,7]}'
sample_34PASS[2, 7, 7, 6, 8, 4, 2, 5, 21][2, 7, 7, 6, 8, 4, 2, 5, 21]
sample_35FAIL[2][]
sample_36PASS'ha''ha'
sample_37PASS['123', '23', '3']['123', '23', '3']
sample_38FAIL'1oe-errBzz-bmm''1Oe-ErrBzz-Bmm'
sample_39PASS00
sample_40PASS"the cow goes moo#"'the cow goes moo#'
sample_41PASS[58, 92, 21][58, 92, 21]
sample_42PASS[][]
sample_43PASS-1-1
sample_44FAIL'*+n+z+o+h''*+++n+z+o+h'
sample_45PASS22
sample_46PASS'manylettersasvszhelloman''manylettersasvszhelloman'
sample_47PASSFalseFalse
sample_48PASS""''
sample_49PASS'816''816'
sample_50PASS[1][1]
sample_51PASS2020
sample_52PASS'seiq d''seiq d'
sample_53PASS[1, 1, 1, 1, 1, 1, 1][1, 1, 1, 1, 1, 1, 1]
sample_54PASS11
sample_55PASS[89, 43, 17, 14, 8, 4][89, 43, 17, 14, 8, 4]
sample_56PASSTrueTrue
sample_57PASS11
sample_58PASS[-1, 0, 0, 1, 1, -1, 0, -1, 0, -1][-1, 0, 0, 1, 1, -1, 0, -1, 0, -1]
sample_59PASS'hi''hi'
sample_60PASS'R''R'
sample_61PASS'nada''nada'
sample_62PASS('ja', 'nee', 'coke', 'zoo')('ja', 'nee', 'coke', 'zoo')
sample_63PASS'dbtdabdahesyehu''dbtdabdahesyehu'
sample_64PASS' 7 '' 7 '
sample_65PASS99
sample_66FAIL'h'''
sample_67PASS'6,8,8''6,8,8'
sample_68FAIL'wpvr.dq''dq'
sample_69PASS'Name unknown''Name unknown'
sample_70PASS3030
sample_71PASS{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}{1: 2, 3: 4, 5: 6, 7: 8, 10: 9}
sample_72PASSTrueTrue
sample_73PASS(3, 6)(3, 6)
sample_74PASS[44, 34, 23, 82, 15, 24, 11, 63, 99][44, 34, 23, 82, 15, 24, 11, 63, 99]
sample_75FAIL8-22
sample_76FAIL[6, 2, 0, 0, 0, 0, 0, 3, 10][6, 2, 0, 0, 0, 0, 0, 2, 3, 10]
sample_77PASS''''
sample_78PASS'mty''mty'
sample_79PASS'1,2,3,4''1,2,3,4'
sample_80PASS'ba''ba'
sample_81FAIL[("Bulls", 23), ("White Sox", 45)][('Bulls', 'bulls'), ('White Sox', 45)]
sample_82PASS'BFS''BFS'
sample_83PASS'2:0''2:0'
sample_84FAIL'nwdvday mefday ofmeday bdrylday''nwvday mefday ofmeday bdrylday'
sample_85FAILpython[3, 4.5]
sample_86PASS['sdfs', 'drcr', '2e']['sdfs', 'drcr', '2e']
sample_87PASS'-2139-1''-2139-1'
sample_88PASS"hello"'hello'
sample_89PASS'O''O'
sample_90PASS[[1, 2, 3], [], [1, 2, 3]][[1, 2, 3], [], [1, 2, 3]]
sample_91PASS['1', '2', 'a', 'b', '3', 'x', 'y']['1', '2', 'a', 'b', '3', 'x', 'y']
sample_92PASSFalseFalse
sample_93PASS'iq!!!!''iq!!!!'
sample_94PASS{'w': 3, 'wi': 10}{'w': 3, 'wi': 10}
sample_95PASS{'fr': 'AAA'}{'fr': 'AAA'}
sample_96PASSTrueTrue
sample_97PASSTrueTrue
sample_98PASS11
sample_99FAIL'aa+++___bb''aa++___bb'