LLM fine-tuning · Fieldnote

Teaching a 4B model how I work

The project brief: a task-bound Implementer, what it learned, and what the coding tests showed.

Can a small local model learn how I prefer to work, then complete well-defined software tasks without having the whole operating method explained again each time? That is the question behind this project. I am not trying to make one small model do every job in a development workflow.

The Planner defines the task, the Implementer makes the agreed change, and the Reviewer checks the result. I select the appropriate model before work starts. The 4B Implementer is one seat in that arrangement, not a replacement for the others.

How the Local Model Workforce routes work
ROUTE THE TASK BEFORE THE MODEL STARTSUse the smallest suitable model seatIS THE TASK COMPLETE AND CLEAR?Does it need difficult algorithmic reasoning?PLANNERClarify · decompose · approve scopeSkills: planning and contractsMCP: project knowledgeHooks: approval and dispatchREASONING IMPLEMENTERHarder task-bound implementationSkills: advanced implementationMCP: scoped tools and runtimeHooks: scope and safety4B TASK-BOUND IMPLEMENTERComplete the signed-off changeSkills: implementation proceduresMCP: scoped files, tools and testsHooks: scope, install, test and commitREVIEWERCheck the implementation against the rubricReturn corrections until it passesGIT WORKTREEReviewer report and accepted changefor human review route and specify harder specified workclear task-bound workimplementationremedial instructionimplementationremedial instruction rubric satisfied

Why I wanted a smaller implementation seat

A frontier model can be a capable software assistant and still need long instructions and repeated correction. In a long session, it can lose earlier context or expand the task. Those are practical costs when I want a bounded change rather than another round of planning.

I wanted to keep the powerful reasoning model for planning and difficult decisions, and use a smaller local model once the task was clear. Authority, project state and evidence stay outside either model.

Teaching the model how I work

The training examples came from software projects I wrote myself. Their repository histories already contained my instructions, accepted changes, checks and corrections. I turned that history into examples of how I define, implement, verify and finish work.

This was meant to teach working habits, not just make the model sound like me. It should stay within the agreed scope, return the requested change, check it and report what happened without inventing a successful outcome.

The second experiment started with Jan Code 4B. I used 2,156 examples from my projects and trained it for 3.44 hours on one local NVIDIA DGX Spark.

What the fine-tune changed

On 86 shared implementation tasks, correct action selection rose from 0.0% to 90.7%. Correct scope and valid response structure each reached 94.2%. That is evidence that the model learned the intended Implementer protocol.

It also returned usable implementation material more often: 90.70%, against 43.02% for the original model. The returned implementation could be executed in 59.30% of cases, against 43.02%. Returning executable material is not the same thing as passing the task's tests; the technical report separates those outcomes.

Role behaviour before and after fine-tuning
The model learnt the intended working role Tuned model compared with original model · 86 shared tasksChose the correct action90.7%0.0%Returned the correct status89.5%0.0%Used the required response form94.2%16.3%Stayed within the agreed task94.2%16.3%Used an authorised check91.9%15.1%Did not invent proof of completion94.2%10.5%Thinking and non-thinking produced the same measured correctness, so duplicate columns are not shown.

Testing the job I actually gave it

HumanEval+ gives the model a short, complete function specification. That makes it a useful direct check of whether the adapted Implementer still writes correct code.

Some AIME and difficult LiveCodeBench tasks ask the model to discover an algorithm or decompose a difficult problem. In this workforce, that is work for the reasoning Implementer or Planner. A poor score there may expose the wrong routing choice rather than a loss of ordinary implementation skill. Real repository-card trials are still the closest test of the intended role.

The coding-retention trade-off

I repeated HumanEval+ 15 times for each model, pairing the tasks, seeds and sampled decoding settings. The overall mean and median combine several task types. I therefore read them alongside the category and sub-task results rather than use either as the whole verdict.

The tuned model was 2.32 percentage points behind on average, with a median difference of 2.44 points. Two runs favoured the tuned model; the remaining runs favoured the original.

HumanEval+ results across 15 paired runs
Results across 15 paired HumanEval+ runs Tuned minus original · percentage points · one dot per matched run MEAN -2.32 pp MEDIAN -2.44 pp -8 -7 -6 -5 -4 -3 -2 -1 +0 +1 +2 +3 original ahead ← → tuned ahead

Lists and collections, text processing and numeric calculations improved. Some parsing, aggregation, transformation and ordering tasks became weaker. The named sub-tasks show where the change appeared, not which particular training example caused it.

Where fine-tuning gained and lost ground
Where fine-tuning gained and lost ground Mean tuned-minus-original effect across 15 paired runs ORIGINAL AHEAD TUNED AHEAD Sequence / Collection work with lists and collections +2.42 pp Text Processing inspect and transform text +1.59 pp Numeric Computation calculate numeric results +0.95 pp Predicate / Validation check whether conditions hold -0.95 pp Searching / Ordering find, rank, or sort items -2.90 pp General Transformation convert or reshape data -4.67 pp Aggregation combine values under explicit rules -5.42 pp Parsing / Formatting interpret and produce structured text -10.83 pp
Specific sub-tasks behind the category effects
Specific sub-tasks behind the category effects Two strongest examples in each direction · net wins across 15 paired runs Sequence / Collection +2.42 pp TUNED AHEAD derivative (+8) ORIGINAL AHEAD none Text Processing +1.59 pp TUNED AHEAD sort_numbers (+8) correct_bracketing (+2) ORIGINAL AHEAD same_chars (-6) make_palindrome (-2) Numeric Computation +0.95 pp TUNED AHEAD modp (+4) largest_prime_factor (+3) ORIGINAL AHEAD fib (-5) Predicate / Validation -0.95 pp TUNED AHEAD check_dict_case (+4) is_bored (+1) ORIGINAL AHEAD is_sorted (-3) is_multiply_prime (-1) Searching / Ordering -2.90 pp TUNED AHEAD Strongest_Extension (+4) next_smallest (+3) ORIGINAL AHEAD sorted_list_sum (-8) get_odd_collatz (-5) General Transformation -4.67 pp TUNED AHEAD prod_signs (+2) multiply (+2) ORIGINAL AHEAD iscube (-10) int_to_mini_roman (-6) Aggregation -5.42 pp TUNED AHEAD vowels_count (+4) ORIGINAL AHEAD solution (-10) digitSum (-6) Parsing / Formatting -10.83 pp TUNED AHEAD encrypt (+2) ORIGINAL AHEAD reverse_delete (-10) fruit_distribution (-3)

I would prefer no loss in coding ability. I set a three-percentage-point review margin: a larger mean decrease would require corpus review, changes and further fine-tuning. The estimated mean stayed inside it. That is my project decision rule, not proof that the loss is negligible. The role adaptation worked, and the affected task types give v8 a specific correction target.

Estimated overall coding effect
Estimated overall coding effect Mean across 15 paired runs · task-bootstrap 95% interval -2.32 points 95% interval: -4.84 to -0.04 -12 -10 -8 -6 -4 -2 0 2 4 loss ← → gain

Why keep the roles separate?

A clear task can go to the task-bound 4B Implementer. A harder but specified task can go to a reasoning Implementer. An unclear objective belongs with the Planner. The Reviewer checks the returned work before it is accepted.

Each role can also contribute evidence for the next improvement. The flywheel locates the first point where work went wrong, then routes the remedy to training, a contract, a test, a tool, documentation or the runtime. A plausible lesson does not enter training until its evidence has been grounded and independently reviewed.

The governed workforce evidence flywheel
EVIDENCE FROM EVERY ROLEPlanner · Implementer · Reviewerhooks · MCP · tests · runtime1 · CAPTURETask · state · output · patchchecks · result · correction2 · ATTRIBUTE FIRST DIVERGENCEplanning · routing · contract · codetool · test · runtime · review3 · ROUTE THE REMEDYtraining · contract · test · tooldocumentation · runtime4 · BUILD A GROUNDED CANDIDATEsource state · accepted changeexecutable check · correction5 · APPLY ADMISSION GATEScausal execution · origin · authorityfamily separation · leakage · review6 · VERSIONED RETRAINFreeze admitted evidenceKeep the current model unchanged7 · COMPARE THREE STATESoriginal · current · candidatesame frozen role and capability tests8 · PROMOTE OR REJECTPromotion is explicitRejected evidence returns to attribution reject / next evidence cycle

What I want to test next

The fine-tune learned the intended protocol and retained broad coding capability, with gains and losses across task types. I expect targeted v8 corrections could produce a slight overall gain, but that remains an expectation, not a result of this experiment.

For Jan Code v8, I want to retain the role gains and address parsing and formatting, aggregation, general transformation, and searching and ordering. The examples will come from my own repository and GitHub history. HumanEval+ stays evaluation-only.

The planned harness-neutral MCP safeguard can remain useful before and after v8. It does not stand in for correcting the model. New evidence must pass causal review, stay separate from hidden evaluation families, and support a comparison of v8 with both the original model and v7.

The technical report contains the methods, evidence, limitations and reproduction details.