Fieldnote · models · evaluation

Is fast decoding the be all and end all?

Four ways to serve GLM-5.3-Flash. A high decode rate can be true and still tell you very little about how an agent feels to use. Serving the same model on the same two DGX Sparks under four recipes and measuring each one the same way produced a trade-off, not a winner: the recipe that answers fastest is not the one that generates fastest, and none of the four escapes the constraints that matter for real agent work.

Julen Gamboa · 9 October 2026

What was compared

Four independently guarded serving recipes on the same hardware: TensorFold EXL3 with IncoAI's DFlash2 drafter under two policies, fc5:0.3 and fnc7:0.3; TensorFold MTP, the only DFlash2-free recipe of the four; and SparkGLM NVFP4 on its upstream 4×512k profile, which also uses DFlash2.

Each recipe was measured at one active request (C=1); fc5, fnc7 and SparkGLM also at four (C=4). MTP's fixed recipe supports C=1 only, and SparkGLM's C=4/256k cell was intentionally not run: four distinct 256k prompts unfortunately exceed its ~600k shared KV pool, I might revisit this and increase the KV pool but I was getting OOM errors when I first tried. In any case, that cell is unavailable, it's not zero, and nothing else was extrapolated from this. Every number on this page is traceable to a raw condition in the explorable sections below; the measurement families (sustained decode, prefill estimate, finite request, Tool Eval, 500k+ needle) are defined in the glossary and never mixed.

The fastest answer is not the fastest generator

At 32k context, SparkGLM's client-observed prefill estimate was 2,508 prompt tokens/s against roughly 1,780–1,810 for all three TensorFold recipes, and its first output token arrived at 13.1 s versus about 17.4–17.8 s. If your workload is "send a long prompt, get moving quickly," SparkGLM wins this setup.

But once decoding is under way, TensorFold pulls ahead. Sustained decode at 32k/C=1 measured 50.6–52.8 tok/s for the TensorFold recipes against SparkGLM's 41.3. At C=4 the gap widens: fc5 and fnc7 reached 162.8 and 182.6 aggregate tok/s against SparkGLM's 71.3, though those are timed active-generation windows, not completed-request throughput. On the same finite C=4 workload measured on a shared clock, the rates were far lower (15.5 / 17.0 / 9.9 tok/s) because the shared span includes staggered prefill and decode gaps. These numbers answer different questions, which is exactly why the sections below keep the measurement families separate.

SparkGLM's whole-request wall times also look better partly because it frequently produced shorter responses (28 output tokens versus 259–339 in one 32k finite cell). A faster stop is less work, not faster completion of an equivalent answer.

Inference results

Models
All models

Verified MTP serving configuration and memory

Mia EXL3 checkpoint revision 078455ffe6472f9a52fbc1139f58b9db2881b25c, TensorFold image content sha256:7c988ce66b0f1c045bee1d538b9dd8fbd3cfa2202ed4006d4c10fccac373b7d3 on both Sparks (patch label 330b3568fc33). One active MTP stream, FP8 KV, 1,048,576-token prompt-plus-reply window, default output budget 32,768, thinking and vision enabled, loopback API. The paired and both local guards enforced 1,024 MiB MemAvailable; post-start head/worker readings were 18,722/22,387 MiB. After the 256k probe they were 17,644/21,225 MiB; these are not measured prefill minima. The model was stopped cleanly afterwards. The startup and shutdown preserved rank logs on their respective Sparks at ~/.cache/tensorfold-glm53/logs/20261007-210138-rank0.log.gz and ~/.cache/tensorfold-glm53/logs/20261007-210138-rank1.log.gz. Both long prompts used fresh nonces; token counts came from this service’s /tokenize endpoint; first-token timing stops at the first nonempty streamed reasoning or content delta.

What is included, and what is not?

Only independently measured C=1 and C=4 conditions appear in the Concurrency menu; C=2 and C=3 cannot be inferred from C=4 requests. Start with C=1 and MTP selected to see measured results. C2–C4 are unsupported by the fixed MTP profile, not unmeasured tests that can be filled by rerunning it. The first MTP 128k/256k admission checks used tiny output caps. Subsequent C1 sustained-decode, 400-token finite-workload, Tool Eval and 500k+ needle checks are separate measured runs. At 256k, temp 0 and omitted finite replies hit the 400-token cap; those are not completed answers. MTP cannot serve active C2–C4 under this recipe. fc5 and fnc7 have gated finite C1/C4 evidence. SparkGLM has gated finite C1 and C4 through 128k; four distinct 256k prompts exceed its approximately 600k shared KV pool, so its C4/256k cell is unavailable rather than zero. A dash means unmeasured or unsupported, not zero. The server's no-temperature-parameter default for the MTP profile is temperature 1.0, top-p 0.95. Historical TensorFold ~80–89 tok/s headlines used full completion usage tokens over a shorter post-warmup interval; those inflated rates are excluded from this comparison (see `docs/reconciliation/glm-tensorfold-throughput-drift.md`). TensorFold fc5/fnc7 and the measured SparkGLM profile all use IncoAI DFlash2 (CC BY-NC-ND 4.0); only TensorFold MTP is DFlash2-free among these four, and its fixed profile supports C=1 only. SparkGLM's engine is AGPL-3.0-only. The NVFP4 and EXL3 recipes also differ in checkpoint and serving runtime, so a speed difference cannot isolate NVFP4's effect.

Metric glossary

Sustained decode (tok/s)
Output-token rate during a timed generation window after warmup. Prefill time is excluded. At C=4, aggregate rate counts all four active streams; it does not show completed-request throughput.
Prefill (tok/s)
Here estimated as prompt tokens divided by client-observed time to first token, from one C=1 request per prompt length. This includes request overhead, so it is not server-only prefill or a C=4 prefill measurement.
First streamed token / TTFT (s)
Time from sending a request until the first nonempty reasoning or answer delta. Lower is faster; long prompts spend much of this time in prefill.
Request wall time (s)
Time from request start to completion, including prefill, generation and response overhead.
Finite aggregate decode (tok/s)
For the gated fc5 run, sum of each stream’s output tokens after its first divided by the shared monotonic interval from the earliest first output to the latest last output. Interleaved prefill or pauses between streams affect this span; it is not the timed sustained-decode rate. MTP C=1 uses its per-request first-to-last output interval. fnc7’s gated C=4 rerun and SparkGLM’s applicable gated cells supply the same shared-clock evidence; earlier approximate fnc7 files remain superseded.
End-to-end output (tok/s)
Output tokens divided by the entire request wall time. fc5, fnc7 and applicable SparkGLM gated finite C=4 values use the actual common monotonic start-to-last-end batch interval. Historical fnc7 approximate wall-time files are superseded. This can look low at 128k–256k because processing the long input takes tens or hundreds of seconds before output begins. It does not mean that generation itself runs at that low rate: compare with finite decode and TTFT for the same request. Shorter replies also lower this whole-request average.
Sampling
Explicit temperature 0 and 1 are separate from a request with no temperature field. An omitted field uses the server's own defaults.

Speed did not buy correctness

The 69-scenario Tool Eval ranked the TensorFold recipes 93/94/93 (omitted/temperature 0/1) against SparkGLM's 88/88/87. SparkGLM's higher numeric deployability composite comes from faster tool turns (median 1,403 ms versus 2,757–2,964 ms), not from better tool behaviour. And the finding that matters most: all twelve C=1 runs failed TC-51's planning-order safety gate, issuing a send_email before receiving the result of a dependent create_calendar_event. No recipe, drafter, or quantization escaped it.

The benchmark's own framing applies: it tests mock-tool scenarios; a passing score is not a safety qualification. The orchestrator must still enforce dependency order, validate schemas and required arguments, and gate consequential side effects behind approval. A benchmark score cannot, to my knowledge, easily do those jobs.

Long context: everyone retrieved the needle, slowly

All four recipes retrieved an exact fresh code from a 500k+ prompt with zero cached tokens. But the first-token waits are the story: 225.9 s for SparkGLM (on its shorter ~520k prompt, fitting its 524,288 limit) versus 419–439 s for the TensorFold recipes on ~527k prompts. Speculative decoding improves generation, not cold prefill: a half-million-token cold prompt costs six to seven minutes before the first token regardless of drafter. No C=4 500k+ retrieval was measured, and none should be inferred.

What this means for real work

For an interactive workload, the input length, output length, concurrency, and how often tool turns interrupt generation decide which recipe fits, and one bar cannot stand in for that workload. A fast-decoding recipe with a slow first token suits queued analysis and long generations. A fast-first-token recipe with lower throughput and shorter answers suits interactive probes. Neither is "the fastest"; they are fast at different things.

Meme of Jim Carrey replying to a child: slow is smooth and smooth is fast is just something slow people say

None of the four measured configurations simultaneously provides proven C=4 throughput, the stronger Tool Eval scores, and freedom from DFlash2 restrictions. For an open-source, DFlash2-free project, TensorFold MTP is the best research starting point: it scores with fc5/fnc7 and avoids the drafter licence, but its fixed profile serves only C=1, and enabling concurrency needs implementation, validation, and rebenchmarking. SparkGLM's measured advantages do not solve the DFlash2 constraint (its engine is additionally AGPL-3.0-only), and no speed difference here isolates NVFP4 versus EXL3, because the checkpoints and runtimes differ too.

On licensing: DFlash2 is CC BY-NC-ND 4.0. The licensor's written reply to me covers individual personal experimentation at this stage, not products, hosted services, or company deployment. Audit runtime, checkpoint, drafter and dependency rights separately before any release or service. Compulsory disclaimer blurb: This is not legal advice.

Limits

One observation per cell, no randomization, no error bars, no matched answer-length controls. Finite replies may stop early or hit the 400-token cap; neither outcome measures answer quality. Temperature handling differs between the two APIs (omitted defaults are not guaranteed identical sampling pipelines). The full provenance, exceptions and metric contract are listed in the glossary below.

How this report was composed

This report was composed with the help of the MTP quant, served by Ash Hart's TensorFold engine.

Credits

Everything measured on this page rests on other people's work. Per recipe:

TensorFold EXL3, fc5:0.3 / fnc7:0.3 / MTP

SparkGLM NVFP4

Benchmarks