Teaching a 4B model how I work: the technical report

Jan Code 4B Implementer v7 · technical case study · 15-seed update

I wanted a small local model that could take a well-specified software task and implement it the way I work. That model belongs in a larger workforce: specialised seats, explicit routing, durable project state, mediated tools and checks that actually run. It is not a group of models deciding their own responsibilities and authority.

This report follows the 4B Implementer through executable corpus construction, two starting checkpoints, evidence repair, controlled evaluation and external coding tests. The useful question is what the adaptation changed, and which of those changes survived a test of the work rather than the response format.

Why this work began

The promise is familiar: describe an objective and let a frontier model plan, implement, test and revise the solution. Making that useful and safe in my own projects took considerably more engineering than the promise suggests.

I used CLAUDE.md, AGENTS.md, plugins, skills, hooks, MCP servers and orchestration frameworks. They helped, but the coordination problem remained. Instruction bundles consumed startup context, compaction could weaken earlier instructions, and long sessions drifted from the agreed task. An autonomous pool could generate unrequested code, optimise a target I had not chosen or stall on an unimportant detail.

Those are practitioner observations, not a controlled failure-rate estimate for frontier models. The posts below show other developers describing related problems across different model ecosystems.

A few examples of the operational friction:

Josh (@jjpcodes) on frontier-agent friction

Josh describes frontier-agent friction

John Myles White on Claude Code agents remaining idle

John Myles White describes Claude Code agents remaining idle

Pengz (@penguinspecz) on unattended scope drift

Pengz describes an unattended agent optimising an invented target

I did not think the answer was to tell users they were using Claude incorrectly, or add another general framework above the harness. I split the work into roles, used my repositories' Git histories to build evidence for each role, and moved the first routing decision outside the model. The 4B Implementer gets a signed-off task. A larger reasoning seat gets ambiguity or algorithmic complexity. Approval, scoped worktrees and commits limit the effect of a failure.

Others are describing related architecture choices. Alex Verem highlights Andrew Ng's emphasis on the workflow rather than treating the model as the whole system. Alex Verem on agentic architecture

Satya Nadella makes a related enterprise argument: retain the data, prompts and metadata; keep context, memory and the harness separate from any one model; and use multiple models rather than outsource the whole intelligence layer. That supports local control of orchestration and model choice. It does not require every workload to run on one local model. TechCrunch: Satya Nadella says companies that trust one AI for everything may not survive

What the experiments taught me

The target was one small, portable implementation seat. It does not replace the Planner, reasoning Implementer or Reviewer. It needs a signed-off contract, enough repository context, explicit authority and executable acceptance checks.

I first full-fine-tuned dense Qwen3.5-4B. That established the training method and four-condition comparison, but exposed the main distinction in this report: a correctly formatted response can still contain code that fails its fixture. Reviewing individual cases also exposed missing artefacts, ambiguous contracts, faulty fixtures and evaluators, and runtime limits. They were not all model failures.

I froze v6 and repaired the evidence into a new v7 corpus. I also changed the starting checkpoint to janhq/Jan-code-4b, whose instruct and coding prior was closer to the intended work. The second experiment asked whether I could teach the Implementer contract while retaining that coding ability.

On the common 86-task implementation set, correct role events rose from 0% to 90.70%. The model returned more implementation material too. Fixture pass did not improve. It had learned the delivery contract much more clearly than it had learned to produce correct implementations on this project holdout.

The first LiveCodeBench v6 diagnostic raised a different problem. Many AtCoder and LeetCode Hard-style tasks asked for algorithm selection, decomposition and long monolithic solutions. In this system those jobs belong with a reasoning Implementer or Planner, not directly with the task-bound 4B seat.

I therefore used HumanEval+ as the primary coding-retention check: 164 compact, specified Python tasks with stronger tests. Across 15 paired sampled runs, the mean tuned-minus-base effect was -2.32 percentage points, the median -2.44 points, and the task-bootstrap interval -4.84 to -0.04 points.

Sequence and collection, text processing and numeric computation improved. Parsing and formatting, aggregation, general transformation, and searching and ordering had persistent deficits. Those are the skills I would target in the next corpus, rather than respond to the overall mean with an undirected retraining run.

The conclusion stays scoped to the role. Jan Code v7 learned the Implementer contract and retained broad compact coding ability, with measurable trade-offs. The plan for v8 is to retain those role gains, add grounded examples for the affected skills, and test on family-disjoint cases against both the original model and v7.

1. What each model is responsible for

I keep project state, authority and execution outside the model conversation. The Planner defines work and selects the implementation seat. The Implementer stays within the authorised scope. The Reviewer checks the result, tests, receipts and assumptions. The tools enforce access and retain the evidence.

The execution layer binds a signed-off task to a model, repository scope, permitted tools, isolated runtime and executable checks. It keeps receipts and a rollback path. The model proposes a change within that arrangement; it does not decide its own authority or deployment scope.

I aim for the 4B seat to complete a well-scoped change in one task-bound turn. A Reviewer can ask for a focused correction or escalate. If a task needs significant algorithmic or semantic reasoning, a higher seat should resolve or decompose it first. A sequence of bounded implementation turns is expected for some work; needing several turns is not automatically a failure.

This is external routing, not a claim to control a model's internal experts. A complete task can go to the dense 4B Implementer. A harder task goes to a more capable MoE Implementer or Planner, which still uses its own internal routing. I choose the seat and the task before that inference begins.

The architecture now allows two Implementer seats:

The adaptation method also has to fit the hardware and the evidence available.

I completed a 4B BF16 full fine-tune on one 128 GB DGX Spark. NVIDIA documents an 8B full-SFT workflow on the same class of machine. Beyond that directly evidenced range, LoRA or QLoRA are the practical single-Spark routes for this project; I plan LoRA for the 27B–35B reasoning and review seats. That is my evidence boundary, not a claim that larger full fine-tunes are impossible.

A monolithic algorithmic task can be a routing failure rather than a failure of the 4B Implementer. The Planner can split the objective into independently testable changes, or send a difficult but complete step to the reasoning Implementer.

The integration plan extends tools I already have. Hillstar provides orchestration and is intended to take a stronger routing role. Nuthatch supplies retrieval and the GraphRAG knowledge base. Existing hooks mediate particular repository actions; a harness-neutral MCP safeguard remains planned. Testudo supplies containerised execution, with Firecracker microVMs available as an additional boundary. Not every integration path is complete. Hillstar Nuthatch Testudo Firecracker microVMs

Why instructions are not an execution boundary

A descriptive study of 253 CLAUDE.md files found that build and run instructions appeared in 77.1%, implementation guidance in 71.9%, architecture in 64.8%, and security guidance in 8.7%. It did not test instruction compliance, enforcement reliability, code quality, or task outcomes; direct performance evaluation was future work. I use it only to describe manifest practice. Chatlatanagulchai et al., *On the Use of Agentic Coding Manifests*

Gloaguen et al. provide the controlled performance evidence. Across SWE-bench Lite and their 138-task CTXbench, LLM-generated context files changed resolution rates by −0.5 and −2.0 percentage points on average; neither change was statistically significant. Developer-provided files improved performance by 2.4 points on average, also without statistical significance. Context files increased steps and inference cost by about 20–23%. Trace analysis showed that agents generally followed the instructions, ran more tests, and explored more files. The problem was not general disobedience: additional and sometimes redundant requirements changed behaviour without reliably improving success. The study covered Python tasks and measured patches that passed all tests, so its findings should not be treated as universal across languages or outcomes. Gloaguen et al., *Evaluating AGENTS.md*

Zhang et al. found a different aggregate result in more than 5,000 Claude Code and Claude Opus 4.6 runs on a discriminative subset of SWE-bench Verified: curated and random rule sets both improved task performance by 13.8 points. Because random, shuffled, mismatched-domain, and unconverted rules matched curated rules, the authors infer a context-priming effect rather than reliable content-specific instruction. In their per-rule analysis, beneficial rules were negative constraints and harmful rules were positive directives. This result is scoped to one agent, one model, and one benchmark configuration. Zhang et al., *Guardrails Beat Guidance*

The studies do not show that models generally ignore AGENTS.md or CLAUDE.md. They show that natural-language guidance can alter behaviour while its effect on success remains model-, task-, rule-, and context-dependent. An observational analysis of 20,574 real coding-agent sessions reinforces the enforcement boundary: rule following and constraint violations were recurring forms of visible misalignment, and 91.49% of visible resolutions required explicit user correction. Because this was an observational study of sessions that exposed developer pushback, it characterises visible failures rather than estimating a controlled failure rate for all agent use. Tang et al., *How Coding Agents Fail Their Users*

I read these studies as a boundary on advisory control. Context files can change behaviour, but they do not guarantee the right action or contain its consequences. Those jobs need action mediation and outcome checks outside the model.

Skills can package a useful procedure, and script-based hooks can enforce deterministic checks. Neither is free of trade-offs. An over-broad rule can block legitimate work because it does not fully understand the request.

That leaves three distinct parts of the control stack:

1. Fine-tuning changes how likely the model is to follow the intended role and delivery conventions. It does not turn them into an absolute rule.
2. MCP supplies a defined interface for the task's tools and data. Hooks enforce deterministic checks, while skills supply reusable procedures around it.
3. Isolated worktrees, explicit authority, executable checks, durable artefacts and independent review limit the consequences of a wrong answer.

Internal expert routing does not replace those controls. A frontier MoE model may activate only some experts for a token, but its serving system still has to hold the model and pay that infrastructure cost. More reasoning can also add scope, and a long context can lose earlier instructions. None of those internal mechanisms is an external authority boundary.

I choose the task-bound model before inference: a complete specification goes to the 4B seat; ambiguity or difficult algorithmic work goes to a reasoning seat. The aim is lower inference cost and less repeated prompting, correction and role negotiation, not a guarantee that routing alone makes the implementation correct.

A fixed response style can be another source of trouble. The third-party i-have-adhd skill says to "cap lists at 5 items" and omit preamble, recap and closing pleasantries. That may suit a user-selected presentation mode. Encoding it as universal behaviour would also restrict the space available for exploration, qualification and longer explanations. I would not make that decision without testing the tasks it might affect. ayghri, *i-have-adhd* skill specification

Anthropic's April 2026 Claude Code postmortem is relevant here. It describes lower default reasoning effort, a thinking-retention defect and a verbosity-reduction instruction that harmed care and coding quality. That is direct evidence about those particular changes. It gives me a reason to test restrictive style rules, but not a controlled result for permanently training the separate i-have-adhd skill into a model. Anthropic, *An update on recent Claude Code quality reports*

I separate reasoning from presentation instead. The runtime keeps the native reasoning trace and evidence as durable artefacts. A later step renders the user-facing result in ASD-STE100. The user or Reviewer can inspect the detail without requiring the model to do its reasoning in a permanently restricted format.

The role split serves the same purpose. The Implementer returns an artefact for a signed-off task. The Reviewer checks it against the acceptance condition and scope, then requests a focused repair or escalates. No one model has to carry every responsibility through an indefinitely long session.

The proposed Atomic Task Graph framework describes a compatible decomposition principle. It exposes subtask dependencies in a directed graph, localises context, preserves validated regions, and repairs only the affected subgraph. The paper reports improvements with 7B–8B backbones. Its interactive benchmarks do not directly validate this workforce, but the framework supports the decision to separate planning, execution, and local repair. Zhang et al., *Atomic Task Graph: A Unified Framework for Agentic Planning and Execution*

Decomposition makes a routing decision possible before execution. This differs from Fireworks' *oracle routing* analysis, which runs multiple models and retrospectively selects the cheapest correct result. That is a useful counterfactual upper bound, but it does not select a model before work begins. This project uses an inspectable operational rule: a complete signed-off specification routes to the 4B Implementer; ambiguity or algorithmic complexity routes to a reasoning seat for decomposition or resolution. Fireworks AI, *Kimi K3 is competitive with Fable; Kimi K3 + Fable is SoTA*

Pre-inference routing also reduces the need to use reasoning effort as a project-control mechanism. Cheng-Yuan (Sam) Lee of Cognition reports a FrontierCode result in which stronger reasoning effort increased unrequested refactoring even when the requested change was identical. The proposed verifier penalises changes beyond scope. This project adds an earlier control: route and decompose first, then bind each implementation to a signed-off task, isolated worktree, and individual commit. The practitioner evidence supports this design choice; it does not replace the project's measured evaluation. Cheng-Yuan Lee, reasoning effort and scope expansion

Reasoning effort and scope expansion

Problem decomposition also makes codebase understanding reviewable. The Planner must identify dependencies, interfaces and acceptance conditions before work is dispatched. That is a different learning path from allowing one long-running model session to accumulate changes faster than a human can inspect them. Jeff Wang, President of New Enterprise at Cognition, describes this relationship through Codemaps in Windsurf. Cognition is the company behind Devin and Windsurf; Windsurf is the agentic development environment discussed in the post. Jeff Wang of Cognition, codebase understanding and coding ability

Codebase understanding and decomposition

2. Building training examples from executable evidence

The training unit is a causal family, not a collection of paraphrased prompts. It links a repository state before a change, the accepted implementation, the deterministic check, the execution receipt, any correction and a hidden family-disjoint analogue.

By family I mean the evidence around one underlying change. The hidden analogue exercises the capability through a different instance: it must not reuse the training failure, accepted patch, fixture, identifiers or repository state. That is how I try to distinguish transfer from remembering an example.

Each seat receives a different projection of the same family. The Implementer sees the contract, relevant state, permitted action and expected delivery. The Reviewer sees the patch, checks, receipts and decision. The Planner sees ambiguity, decomposition, authority and routing. The projections remain linked to the same Git and execution evidence.

For each family, I worked through the following sequence:

1. preserve a parent state that fails for the intended reason;
2. define one well-scoped accepted change;
3. run a deterministic oracle;
4. retain the implementation and execution receipt;
5. preserve real correction paths;
6. create a family-disjoint hidden analogue;
7. project linked evidence into role-specific records;
8. keep the complete family in one split.

I rejected examples with fabricated completion, missing artefacts, unexecuted implementations, unsupported authority, ambiguous requirements, defective contracts or graders, stale evidence, target leakage, repeated causal shapes or family overlap.

Paraphrases can teach phrasing. An executable family can also teach a decision, a permitted change and the evidence that separates success from a convincing-looking failure. That is the distinction I wanted the corpus to preserve.

The starting material was my own operating documentation: working methods, authority, delivery conventions and safety boundaries. I mapped those preferences to real Git states, accepted changes, tests, receipts and corrections instead of treating the prose itself as evidence of a successful task.

3. Training and checkpoint selection

I kept the process in four phases, following the order of the general fine-tuning reference.

For Jan Code v7, the target was the task-bound Implementer. Entire causal families stayed in one split. The training projection had 2,156 records, validation 265 records, and the frozen evaluation 222 cases.

The Jan Code v7 run used BF16 full fine-tuning, a maximum sequence length of 2,560 tokens, gradient accumulation over 16 micro-batches, AdamW, a learning rate of 5 × 10⁻⁶, weight decay of 0.01, and seed 3407.

I completed 225 optimiser steps, validating every 12 steps. From 18 validation measurements, I selected checkpoint 204 at validation loss 0.33107.

The frozen evaluation used 222 behavioural cases and the common 86-task implementation set. Before external coding tests, I bound the selected checkpoint to its recipe, recorded hashes, evaluation evidence and runtime configuration.

4. First experiment: dense Qwen3.5-4B

Starting with a dense base checkpoint made the first experiment demanding. It had to learn general assistant behaviour as well as my Implementer contract.

That run established the BF16 recipe, validation, checkpoint selection, telemetry and frozen tuned/base × thinking/non-thinking comparison. Behavioural learning worked. The executable results showed why that was not the same as learning to implement reliably.

Qwen v6 training and validation loss

On 86 implementation-required cases, the selected v6 checkpoint delivered a repository patch in 85 cases and a parseable patch in 81. Only 16 patches applied and passed their fixture. Fifty-two applied but failed. Thirteen could not be applied, four responses could not be parsed, and one proposed no implementation.

A better response contract did not guarantee correct code. I also could not attribute a failed check to the model until I found the first material divergence. Both lessons shaped the v7 corpus repair and the decision to change the starting checkpoint.

Dense-model outcomes

6. Why I changed the starting checkpoint

The dense model could learn the decisions and protocol while still implementing poorly. Further attempts to constrain its reasoning risked getting more of the delivery contract at the expense of the code. I changed the starting model rather than keep adding restrictions.

Jan Code already had a prior for generation, editing, refactoring, debugging and bounded coding-agent work. A lightweight worker below a larger Planner was close to the role I wanted. That made it a better-grounded second experiment than asking the dense base to learn both jobs at once. Jan Code 4B model card

The benchmark material available when the model was selected reported LiveCodeBench v6 at 51.0 for Jan Code and 35.1 for Qwen3-4B-Instruct-2507, and AIME25 at 53.0 versus 47.0. These are self-reported upstream results, not measurements reproduced by this project. They informed checkpoint selection but did not replace independent coding-retention evaluation.

For v7, I wanted to test four things:

• preserve the Jan Code coding prior;
• teach deterministic role and protocol conventions;
• improve delivery against signed-off tasks;
• retain general coding ability within a reasonable margin of tolerance.

The starting checkpoint changed because of the evidence from the first experiment, not because a second model promised a more attractive headline.

7. Jan Code v7 training

The run used a full BF16 fine-tune of janhq/Jan-code-4b, 2,156 training records, and 265 validation records. The learning rate was 5e-06; the seed was 3407.

I selected step 204 at held-out validation loss 0.33107 before running the frozen evaluation. Training loss alone did not choose the checkpoint.

The run lasted 3.44 hours. Sampled GPU-board energy was 210.27 Wh. Average and maximum board power were 60.97 W and 68.89 W. Average and maximum temperature were 75.3 °C and 83 °C.

The auxiliary fan estimate used the Noctua NF-A14 industrialPPC-3000 PWM at its 6.6 W rated draw. Over this run, estimated fan energy was 22.71 Wh. The combined sampled board-plus-fan estimate was 232.99 Wh, or about 0.233 kWh. At an illustrative electricity price of 25–30 pence per kWh, this measured boundary costs about 5.8–7.0 pence. It is equivalent to roughly 46 minutes of the average power implied by Ofgem's 2,700 kWh annual typical domestic consumption value.

This comparison has a strict boundary. It includes sampled GPU-board energy and the fan estimate. It excludes the rest of the DGX Spark, host peripherals, and conversion losses between the wall socket and components. Public figures for training very large frontier models can describe a genuinely large energy load, but they do not represent every adaptation class. Parameter count, precision, optimiser state, sequence length, hardware, and training method change the energy requirement materially. Ofgem, typical domestic consumption values used for the April–June 2026 price cap

8. Did it learn the delivery contract?

I compared the tuned and untouched BF16 models with thinking enabled and disabled. The full non-thinking behavioural set contained 222 cases. The directly comparable implementation analysis used the same 86 implementation-required task IDs in all four conditions.

Those 86 tasks came from my corpus and repository history, not HumanEval+. They test contracts, delivery format, events and authority as well as code. The tuned model had seen examples of that way of working; the base had not. An advantage here can establish role learning without establishing better general coding. HumanEval+ asks the latter question separately.

Implementation recovery measures the delivery, not its correctness. I reviewed what the model could see for all 86 tasks against each hidden fixture's exact interface. Every task had a usable behavioural specification, but only 74/86 disclosed the full fixture interface. The other 12/86, across seven families, omitted an exact field name, argument order, added function argument or output schema that the grader required.

This was after an earlier repair: 26 of the 86 tasks already included a block disclosing previously missing or implicit contract details. Behavioural completeness and fixture-interface completeness were still not the same thing.

The report records a 12/86 fixture-pass result here. It would be wrong to call the other 74 cases straightforward coding failures. At least 12 had specification-oracle ambiguity; 27 failed during patch application, eight at the protocol or implementation-proposal stage, and 39 ran but failed their fixture. Those are different problems to diagnose.

In ff-honest-subprocess-failure, for example, the request said to retain a command's "exit status". The hidden fixture required the key status; the model returned exit_status. That is a reasonable interpretation of an incomplete interface contract, not clean evidence of an inability to write the code.

v7 learned the delivery protocol strongly while executable correctness stayed weak. To explain a fixture failure, I still have to separate specification gaps, patch delivery, runtime behaviour and implementation defects.

The tuned model produced the correct event on 90.70% of the common set and the correct return state on 89.53%. Parseable JSON, scope compliance, and absence of fabricated receipts each reached 94.19%. Authorised verification reached 91.86%. The base model produced no correct role event or return state on this set.

The tuned model returned recoverable implementations in 90.70% of cases and executed them in 59.30%, against 43.02% for both measures in the base. The reported fixture-pass percentages went the other way: 18.60% tuned versus 26.74% base.

Thinking changed output volume and runtime but not measured correctness on these 86 tasks. All four conditions used greedy decoding (do_sample=False); the flag changed the chat template, but I did not repeat the comparison under stochastic decoding. Equal scores in this condition do not make thinking and non-thinking generally equivalent.

This internal benchmark is not a pure coding test. It also depends on contract interpretation, available artefacts, response extraction, patch application, fixture and oracle quality, and runtime compatibility. I need an external benchmark precisely because those effects are present.

9. What happened to coding ability?

Complete HumanEval+

I ran 15 paired HumanEval+ comparisons. Each pair gave the tuned and base models the same 164 Python tasks and seed, using temperature 0.2, top-p 0.95, top-k 20 and a 1,024-token limit.

The mean tuned-minus-base effect was -2.32 percentage points and the median -2.44. Individual paired runs ranged from -6.71 to +1.22 points. The task-bootstrap 95% interval for the mean was -4.84 to -0.04 points.

The full run-by-run figure and bound result tables remain in the analysis bundle.

Expressed as a loss, the mean was 2.32 points, inside my three-point overall margin. The interval extends beyond that margin, so the mean alone does not settle whether the loss is acceptable. I keep the uncertainty alongside the project decision rule.

LiveCodeBench v6

The first 50-task LiveCodeBench v6 tranche was diagnostic. Many competitive-programming problems were monolithic and needed more reasoning or context than I intended the task-bound 4B seat to supply.

The completed diagnostic retained 37 tasks below the agreed 100-second threshold and excluded 13 scope-mismatch candidates. It remains preliminary. The larger role-scoped LCB v6 projection contains 702 retained tasks from the six official release files. It excludes 350 hard-tier tasks and three empirically evidenced non-hard scope mismatches. No full-run outcome is claimed in this report.

Within this hierarchy, a higher seat should decompose a reasoning-heavy problem before the 4B seat implements the resulting steps. I do not want an evaluation to call missing decomposition an implementation failure without checking which job was actually assigned.

I keep LiveCodeBench and HumanEval+ separate. Their scores are not pooled.

10. What changed in coding behaviour

Of 164 tasks, 21 had a positive 15-run net effect for the tuned model, 30 a negative effect and 113 were neutral. That is a mixed result. Neither 'the fine-tune improved coding' nor 'the fine-tune damaged every skill' describes it.

11. Uneven task-type effects

Sequence and collection improved by 2.42 points, text processing by 1.59 and numeric computation by 0.95. Parsing and formatting had a 10.83-point deficit, aggregation 5.42 and general transformation 4.67. Those task groups are more useful correction targets than the overall average alone.

The largest positive drivers included derivative, sort_numbers, check_dict_case, vowels_count and Strongest_Extension. Negative drivers included reverse_delete, solution, iscube, sorted_list_sum and same_chars. Those are particular executable tasks, not proof that an entire faculty appeared or disappeared.

12. Was the fine-tune responsible?

There is evidence consistent with an adaptation effect: eleven outcomes favoured base, four favoured tuned, and all five aggregation discordances favoured base. The fine-tune demonstrably changed response behaviour towards the contract. That can coexist with uneven changes across task families.

But I cannot attribute every failure categorically. The base model exhibits the same two failure-mechanism families. Neither mechanism-specific directional result, nor the overall exact result, is significant. This part of the analysis has one generation per HumanEval task; sampling and decoding variance remain alternatives.

Leon Chlon reports a possible alternative mitigation through NTK-Mirror, a forward-pass activation controller fitted while the underlying model remains frozen. In the published Qwen2.5-7B GSM8K comparison, Chlon reports a 16.3-percentage-point decrease on held-out code generation after LoRA adaptation and a 4.7-point decrease with a 50,000-scalar NTK-Mirror controller. The repository implements signed log-gates, controller composition, benchmark runners, and diagnostics, and it calls for matched baselines, controls, manifests, model revisions, and dataset hashes when making benchmark claims.

If independently reproduced under matched controls, NTK-Mirror could be worth testing as a mitigation. I did not evaluate it or LoRA here, and my repeated full-fine-tuning result does not show a loss of the magnitude reported in that comparison. It is a possible future experiment, not an explanation already established for v7.

Leon Chlon, NTK-Mirror publication

NTK-Mirror repository and method notes

What I am prepared to conclude is narrower:

v7 learned the task-bound role and retained broad HumanEval+ implementation ability. Some task families improved or stayed stable; others need targeted correction. That gives me a validation and routing plan, not a blanket claim of readiness.

13. Real-project evaluation

Benchmarks do not fully measure the intended role. The next test needs actual repository cards.

For retrospective trials, I would select completed cards with accepted implementations and execution evidence, hide the historical solution, and give each model the same starting repository state, contract, tools and context. Fresh tests would compare the returned change with the accepted outcome.

For prospective trials, I would freeze new cards and their starting state before generation. I would record first-attempt success, success after a focused correction, project tests, unnecessary edits, regressions, Reviewer effort, latency, tokens, throughput and energy. The tasks must contribute to the same project goal without an existing solution.

Those repository-card trials remain the next evidence stage in this report. I do not present them as completed work.

14. Proportionate assurance

A benchmark does not make a model compliant. ISO/IEC 42001 concerns organisational AI management; the NIST AI Risk Management Framework puts testing, evaluation, verification and validation within continuing risk management. I draw on those practices without claiming ISO certification, legal compliance or unrestricted production readiness.

The assurance should match the authority. This is a small local model under supervision, receiving a defined card, limited context, mediated tools, executable checks and review. Worktrees, per-card commits and rollback limit the effect of a wrong result. Broader tools or weaker supervision would need stronger evidence and controls.

The scope and security cases will cover authority boundaries, missing artefacts, ambiguity, prompt injection, secret handling, unsafe commands, and tool-output poisoning. The repository-card trials will measure executable correctness, unnecessary changes, escalation, reversibility, and Reviewer effort. These tests evaluate the deployed role and controls rather than the model in isolation.

This is a staged approach for a single-developer project, not a reason to pretend a supervised pilot is either a certified product or impossible to try. Legal review is outside the present evaluation plan unless a concrete commercial, regulatory, contractual, privacy or distribution issue triggers it. The approach draws on the NIST AI Risk Management Framework, Generative AI Profile and Secure Software Development Framework, and ISO/IEC 42001 and ISO/IEC 23894. NIST AI Risk Management Framework NIST Generative AI Profile NIST Secure Software Development Framework ISO/IEC 42001 ISO/IEC 23894

I want the pilot's evidence and assurance to grow together, without confusing either with permission for broader use.

15. Jan Code v8 remediation and retesting

For v8, I plan to address the observed deficits without discarding the role gains. The new examples will come from work in my own repositories.

That work needs to:

1. attribute each failure at its first divergence;
2. search authored repository and GitHub evidence for matching mechanisms;
3. add grounded correction cases for parsing and formatting, aggregation, general transformation, and searching and ordering where causal evidence exists;
4. keep deterministic contracts, tests, routing, tools, documentation, and runtime controls outside the weights;
5. reserve family-disjoint hidden analogues;
6. compare the original model, v7, and the v8 candidate under the same frozen conditions; and
7. promote or reject v8 explicitly.

A planned harness-neutral MCP safeguard can prepare a deterministic checklist, route risk and return an accept, review or reject result. It can help before and after retraining, but does not prove the model deficit is fixed. Its production placement remains a later integration decision. After v8, the warning families are planned to receive three targeted confirmation runs alongside the retained 15-seed baseline.

16. Evidence flywheel

The flywheel is already callable as a skill. The Planner can invoke it with task and repository context, and a human can trigger it after reviewing an outcome. A fine-tuned Reviewer would add another capture point when an implementation fails or a correction reveals a reusable lesson.

Each capture keeps the contract, repository state, model output, patch, tool activity, execution receipt, evaluation and accepted correction. The next integration work is to bring the existing scripts into a single pipeline for attribution, remedy routing, candidate construction, admission and versioned training evidence.

Admitting a case does not authorise training. Provenance, execution, authority, ambiguity, leakage, family separation and independent review remain separate checks. A lesson might improve a fine-tune, Nuthatch, a contract, a test, an MCP safeguard, documentation or the runtime. Training is not the default remedy for every failure.

17. What this work supports

I take four conclusions from the project:

1. Executable corpus engineering taught the 4B model a strong task-bound Implementer role and protocol.
2. Correct response shape, implementation recovery, execution, and fixture pass are distinct measures.
3. Across 15 paired sampled HumanEval+ runs, the tuned model was 2.32 percentage points behind on average. The effect was task-dependent: some capability groups improved, while parsing and formatting, aggregation, general transformation, and searching and ordering identify targeted remediation work.
4. The model workforce should route by task envelope and keep authority, durable state, deterministic checks, and risk controls outside the model.

A small local model can learn this task-bound implementation role without losing all of its broader coding ability. That is useful, but it is not the end of the evaluation. For v8, the evidence points to preserving the role gains, correcting specific task types and retesting against the same retained baselines.

18. Limitations and reproducibility

The coding-retention conclusion uses 15 complete paired sampled HumanEval+ runs per model with matched recorded seeds, do_sample=True, temperature 0.2, top-p 0.95, top-k 20, and max_new_tokens=1024. The mean tuned-minus-base effect was -2.32 percentage points; the task-bootstrap 95% interval was -4.84 to -0.04 points. Task-type labels remain exploratory, and the interval is close to zero at its upper bound. The four-condition internal benchmark includes evaluator and runtime effects. Training energy is sampled GPU-board energy, not wall energy. The larger role-scoped LiveCodeBench comparison and repository-card trials are the next evidence stage.

Key bindings: