From June 29 to July 2 we ran a public experiment on code.umans.ai to figure out how good NVFP4 really is for coding: an NVFP4-quantized build of GLM-5.2 on our own GPUs, open to community volunteers on rotating seats. We served it at 200+ tokens per second, peaking near 250, with sub-2 s median time to first token. The whole thing was live, public, and a lot of fun. It is exactly the kind of test you can only run with a community willing to throw real work at a model and tell you bluntly what happened.

We went in skeptical, and we came out with a clear answer: the speed was real, the quality was not. So we closed the experiment and unlisted the model.

This post is the wrap-up, but it is also an explainer. “NVFP4” gets thrown around as a magic word, so let’s take the time to understand what it actually is, and then use the experiment to see what it does and does not buy you.

What NVFP4 actually is

A weight is a number, and 4 bits is a very small number. A model is billions of weights. BF16 gives each one 16 bits; FP8 gives 8; NVFP4 gives 4. Four bits means sixteen bit patterns. NVFP4’s element format is E2M1 (1 sign bit, 2 exponent bits, 1 mantissa bit), and here is the entire menu of values a weight can take: 0, 0.5, 1, 1.5, 2, 3, 4, 6, and their negatives. That is the whole grid. Every weight in the layer has to become one of those.

Scaling is what makes that survivable. Real weights do not live between -6 and 6 in convenient steps; they span wildly different magnitudes. So quantization stores each number as (4-bit grid value) × (shared scale factor): pick a scale that stretches the grid to fit the numbers it has to represent. The catch: everything that shares one scale gets one stretched grid. Share a scale across a whole tensor and one outlier stretches the grid so far that ordinary weights collapse onto the same few points.

Micro-blocks are NVFP4’s actual innovation. NVFP4 gives every block of 16 consecutive values its own FP8 (E4M3) scale, plus one FP32 scale for the whole tensor. Small blocks mean each little neighborhood of weights gets a grid stretched just for it. The older MXFP4 format uses 32-value blocks and power-of-two scales; NVFP4’s fractional scales fit each block noticeably better.

The difference is visible at the number level. Take the same full-precision input and quantize it both ways: MXFP4’s power-of-two scale snaps to coarse steps (0.00, 2.0, 4.0…), while NVFP4’s fractional scale lands much closer to the original values. NVIDIA’s own worked example shows about 9x lower rounding error, an MSE of 0.08 vs 0.72, from the scale format alone.

The scales cost ~0.5 extra bits per value, so NVFP4 is effectively ~4.5 bits per weight: about 1.8x smaller than FP8, ~3.5x smaller than BF16. Both diagrams are from NVIDIA’s NVFP4 introduction.

The hardware is why this format exists. Blackwell’s fifth-generation tensor cores execute microscaled FP4 natively: the grouping, the per-block scaling, the 4-bit matrix math, all in silicon. No dequantize-then-compute tax. That is what makes NVFP4 a serving format rather than just a compression trick: the weights take half the memory of FP8 AND the matmuls get faster. It is why 200+ tok/s on a 744B model was even on the table.

A “4-bit model” is really a mostly-4-bit model. In the actual checkpoints, quantization covers the linear layers inside the MoE experts, which is where the bulk of the parameters are. The GLM-5.2 NVFP4 card quantizes those weights and activations and explicitly leaves the shared expert alone; attention, embeddings, and the router stay in higher precision, and so does the KV cache. Kimi’s 4-bit release is weight-only on the MoE components. Read the fine print on any low-bit model; the headline format never covers the whole network.

PTQ vs QAT is the line that decides everything. Rounding every weight to its nearest grid point injects noise. There are two ways to deal with that:

• Post-training quantization (PTQ): quantize the finished model, calibrate the scales on a small dataset, ship. The model never gets a chance to adapt to the grid. Cheap, fast, and often fine for short interactions. GLM-5.2 NVFP4 is this: a third-party PTQ conversion of the FP8 release.
• Quantization-aware training (QAT): keep training with the quantizer in the loop, so the weights learn to sit where the grid can actually hold them. The released model IS the quantized model. Kimi does this: K2-Thinking’s card states they adopt Quantization-Aware Training (QAT) during the post-training phase, and adds a sentence we wish every lab would copy: All benchmark results are reported under INT4 precision.

There is a subtlety in that last point that matters more than it sounds. QAT is not a single switch. GLM-5 applies INT4 QAT, but only during supervised fine-tuning (GLM-5 technical report, §2.4.3); the reinforcement learning stages that follow run without the quantizer in the loop, so the released weights are not the weights that were aligned. Kimi and DeepSeek V4 do it the hard way: QAT runs through the entire post-training phase, including RL. DeepSeek V4 even uses real FP4 weights during RL rollouts, so the model that gets sampled during training is the model that ships (DeepSeek-V4 technical report, §5.2.1). That is the difference between a model that has seen the grid and a model that has learned to live on it.

Keep that distinction in mind. It is the whole story of what follows.

What that buys at GLM-5.2 scale

For GLM-5.2 (744B MoE, 40B activated, served from the publisher’s FP8 release), the back-of-the-envelope from the micro-block and hardware points above (worked out in the appendix):

• Weights: ~744 GB in FP8 vs ~420 GB in NVFP4 per copy. Freed GPU memory is KV cache, and KV cache is concurrency. We walked through that math in host-claude-code and GLM-5 vs Kimi-K2.5.
• Decode: ~40 GB moved per generated token in FP8 vs ~22 GB in NVFP4. Roughly 1.8x throughput headroom where it hurts most, under load.

GLM-5.2 is our busiest model. Anyone who has hit it at peak hours knows the pressure is real. A checkpoint that halves both terms deserved a real test, not a dismissal. And it cost zero new GPU spend: we ran the experiment on capacity we already had.

What we told users going in

We were skeptical before serving the first token, and the model’s public description said so:

The published NVFP4 results look very flattering, but we’re skeptical of its real quality and performance: this checkpoint was NOT QAT post-trained for NVFP4 (the QAT-on-NVFP4 models are where we’ve seen the best quality), so treat the benchmarks with caution. […] for production work we recommend the fp8 umans-glm-5.2.

Why we had to run it ourselves

When the GLM-5.2 NVFP4 checkpoint landed, the model card had five benchmarks, all within about a point of the FP8 baseline (full table in the appendix). What it did not have was anything that resembles how our users actually work: no SWE-bench, no DeepSWE-style long-horizon agentic evals, nothing that chains hundreds of dependent tool calls.

Benchmarks are a useful basis. We rely on them. But I am also cautious when a model is evaluated only on short, one-shot tasks, because that is exactly where quantization noise stays invisible. A one-shot benchmark samples the noise once and averages it away. An agent is an autoregressive chain: every token conditions on everything before it, so a slightly worse tool call feeds the next hundred. A three-hour session does not sample the noise, it integrates it.

So the only way to answer the question we cared about was to put the model in front of people doing real work and watch what happened. Real users, real repos, four days. And have some fun with it.

How it went

The first hours sounded like this, verbatim:

Man its so fast its like im in heaven

glm at this speed is intoxicating though […] a near-frontier model running this fast is ridiculous

it’s almost too fast to believe it’s doing anything meaningful but looking at the quality it produced, it’s actually comparable/better in some cases

Participants posted side-by-side outputs against fp8, and one even ran a quick benchmark of their own and reported it looked really good. If we had stopped there, NVFP4 wins.

Then the day-to-day verdicts arrived:

my workflow seems dumb with nvfp4 but was good with fp8 🤔 same code changed nothing

unfortunately from my preliminary probing there is COT collapse, and not even at deep context.

i went back to fp8 because nvfp4 was just making enough mistakes to keep setting off TTSR rules in omp, whereas fp8 seems to rarely do so […] nvfp4 is a nice option to have for the raw speed, but idk if it can replace fp8, at least in my experience so far

The second and the last quotes are the same person, separated by about three hours of real work. That arc, from speed euphoria to quietly switching back, is the measurement no benchmark table gives you. Others caught concrete artifacts along the way: chain-of-thought collapsing well before deep context. Exactly the compounding-noise signature you expect from a PTQ checkpoint that was never trained for the grid.

Our wrap-up in Discord said it plainly: feedback was consistent, speed was great, but for serious work the gap with fp8 was clear, so we are keeping quality first and closing this one out.

QAT is the difference, not the format

Mid-experiment, one participant asked exactly the right question: Still no QAT, right? What needs to happen for that?

NVFP4 is not the villain here. The same format, applied to a model that was post-trained for 4-bit precision, is essentially lossless.

Kimi-K2.7-Code, our current default coder, ships from Moonshot natively quantization-aware trained at 4 bits (same method as K2-Thinking, QAT through the full post-training phase), and we serve the released weights as-is. DeepSeek V4 takes the same route with FP4. Our rule, straight from our FAQ: quality first; when a model is released with QAT weights, we serve those directly rather than quantize on top.

And when NVIDIA converted those QAT weights to NVFP4, they could publish the benchmarks that matter. The K2.7-Code NVFP4 card includes SWE-bench Verified and Terminal Bench 2.1:

NVFP4 scores a hair above the baseline, well within the confidence interval, so the honest claim is parity. A 4-bit model re-encoded into a 4-bit format loses nothing, because the post-training already taught the weights to live on that grid.

Same format, opposite outcomes. The variable is whether the lab trained for it.

What we ship

NVFP4 the format is good enough to serve, and we proved it can: 200+ tok/s on a 744B model, with headroom to go faster and serve more concurrent users than fp8 ever could. The format is not the reason we unlisted the model. The checkpoint was. GLM-5.2 was never trained for 4-bit, so the tokens it produced at that speed were not useful enough for real work.

If the tokens are not useful, they are not worth serving, no matter how efficiently we can produce them.

The serving recipe stays on the shelf for checkpoints that earn it: post-trained at the precision we serve, benchmarked on long-horizon software engineering. K2.7 proves that path exists, and we are already serving it.

Thanks to everyone who volunteered their real workloads and their patience, including the ones who disagreed with the verdict. The fastest way to a trustworthy model lineup is users who tell us bluntly whether the tokens are useful.

Appendix

A. Weights and decode math

NVFP4 stores one 4-bit E2M1 element plus one FP8 scale per 16-value block, ~4.5 bits per value, so bytes per parameter is ~0.56 vs 1.0 for FP8.

• Weights: $744\text{B} \times 0.56 \approx 420\ \text{GB}$ vs $744\ \text{GB}$ in FP8, per copy of the weights.
• Decode bytes per token (MoE proxy, activated params times bytes per weight): $40\text{B} \times 0.56 \approx 22\ \text{GB}$ vs $40\ \text{GB}$.

The real ratio is slightly worse than the ideal: the NVFP4 checkpoint quantizes the linear operators inside the MoE experts and leaves the shared expert unquantized (per the model card).

B. The published GLM-5.2 NVFP4 benchmarks

From the model card, NVFP4 vs FP8 baseline:

Near-parity across the board. τ²-Bench Telecom is agentic, but its episodes are short-horizon customer-service flows. Nothing here measures hundreds of dependent coding steps, which is where our users live and where the gap showed.