From July 14 to July 16 we ran a public experiment on umans.ai: DeepSeek V4 Pro with DSpark speculative decoding, open to community volunteers on rotating Labs seats.

We were there for the architecture. DeepSeek V4 changes two things that decide how many people a GPU can serve at the same time, and both of them are things we expect the rest of the open-weight field to adopt. We wanted our hands on them, on real traffic, before that happens.

Attention that scales almost linearly

Attention is the part of a transformer that scales badly. Every new token attends to every token before it, so the work per token grows with the length of the conversation, and the work over a whole session grows with the square of it. The memory behaves the same way: the KV cache holds an entry per token per layer, it grows linearly with context, and it stays resident until the session ends.

That memory is what decides how many people we can serve. A GPU’s HBM holds one copy of the weights, and everything left over is KV cache. Divide the leftover by the KV footprint of a single session and you have your concurrency ceiling. Agentic coding is close to the worst possible workload for this arithmetic: sessions run for hours, contexts run into the hundreds of thousands of tokens, and every one of those tokens occupies memory for the entire session. This is the same math we walked through in host-claude-code and GLM-5 vs Kimi-K2.5, and it is the ceiling we keep running into.

DeepSeek V4 attacks it in the architecture rather than in the serving stack. Instead of each token attending to every previous token, each token attends to a compressed representation of them. The product that used to be token × token becomes token × rep(token), and a bounded representation on the right-hand side turns a quadratic curve into a nearly linear one.

Two mechanisms do the compressing (DeepSeek-V4 technical report, §2.3):

• Compressed Sparse Attention. Collapse every m tokens into a single KV entry, then use a lightweight indexer to select only the k most relevant of those compressed entries to attend to, with a sliding window kept alongside for local detail. Compression and sparsity stack.
• Heavily Compressed Attention. Compress far harder still, then attend densely over the result. No selection needed, because there is so little left to attend to.

The numbers are the reason this is interesting. At a 1M-token context, V4’s KV cache is around 2% of a conventional BF16 baseline. Against DeepSeek’s own previous generation at that same context length, V4 Pro runs at 27% of the single-token FLOPs and 10% of the KV cache of V3.2; V4 Flash lands at 10% of the FLOPs and 7% of the KV cache.

Read those as concurrency multipliers rather than efficiency percentages. Ten times less KV cache per session is roughly ten times as many sessions resident in the same memory, and the FLOPs figure means the sessions you do serve stop paying a steadily growing attention tax as their context fills up. The long session, which has always been the expensive customer, becomes an ordinary one.

This is where the whole field is heading. A recent survey of cross-datacenter serving (Prefill-as-a-Service, Qin et al., April 2026) measures the current generation of hybrid-attention models against their full-attention predecessors and finds KV cache reductions of 4x to 36x: 13x for MiMo-V2-Flash over MiniMax-M2.5, 4x for Qwen3.5-397B over Qwen3-235B, roughly 36x for Ring-2.5-1T. Their point is that the footprint has now fallen far enough to change what is possible at the systems layer, and they use it to run prefill in a different datacenter and ship the resulting cache over commodity Ethernet. Our interest is more boring than theirs. Every gigabyte a session stops holding is a gigabyte the next person in the queue can have.

That is the memory half. The speed half is DSpark.

The frontier

Here is the plot that made us want to run this experiment. It is Figure 7 from DeepSeek’s DSpark paper, measured on their own live production traffic.

Read the axes slowly, because this picture is the entire economics of serving a model.

• The x-axis is tokens per second per user. How fast one session feels. It is the number you experience when you watch tokens land in your terminal.
• The y-axis is aggregate throughput per GPU. How much total work the machine does per second, which is how many people it can serve at the same time.

Those two fight each other on every serving stack ever built. A GPU generating tokens spends most of its time waiting on memory, so the way you extract work from it is batching: run many users’ requests through the same forward pass and amortize the weight reads across all of them. Bigger batch, more total throughput, and each individual user waits a little longer for every token. Smaller batch, snappier sessions, idle silicon. You slide along the curve.

The curve is the constraint. Where you sit on it is a policy choice. Every capacity lever a serving stack has picks a point on it: concurrency limits, priority tiers, a separate lane for background work. They all decide who gets served how fast when the curve is too small for everybody. That is allocation, and allocation is zero sum.

Pushing the curve outward is the move that gives everybody something. That is what the green line is. At a fixed 35 tokens per second per user, DSpark carries 52% more aggregate throughput than the configuration DeepSeek ran before it. Read the other way, at matched throughput, each user’s tokens arrive 57% to 78% faster.

The paper also marks a 406% figure at a stricter 50 tok/s/user target, and it is admirably careful about what that means: at that interactivity level the old configuration collapses to a very small batch, so the ratio inflates. DeepSeek’s own reading is that the frontier reached an interactivity level the baseline could not serve at all. We read it the same way, and for us that reading is the more useful one. “The regime where the baseline collapses” is a fair description of our peak hours.

Speculative decoding, briefly

Generating a token requires a full forward pass over the model, and that pass is dominated by reading weights out of memory. One token’s worth of arithmetic comes nowhere near filling the tensor cores while that read happens, so the machine spends the decode phase largely waiting.

Speculative decoding fills the gap with a bet. A small, cheap draft model proposes a block of candidate tokens. The real model then verifies the whole block in a single forward pass, because verification parallelizes in exactly the way generation does not: checking twelve tokens costs roughly what checking one costs, since the expensive part was reading the weights either way. Whatever prefix the big model agrees with gets accepted, plus a bonus token it generates itself. The rest is discarded and the cycle repeats.

The acceptance rule makes this lossless. It is rejection sampling, constructed so that accepted tokens follow the target model’s distribution exactly. Any token the big model would not have produced gets thrown away and replaced with one it would. DeepSeek says it plainly on the release: DeepSeek-V4-Pro-DSpark is not a new model. It is the same checkpoint with an additional speculative decoding module attached. We have turned down fast checkpoints before for degrading the tokens they produced; this class of acceleration removes that question entirely, which is why we were willing to put it in front of users at all.

The economics reduce to one fraction. If a cycle costs T_draft + T_verify and yields τ accepted tokens, per-token latency is (T_draft + T_verify) / τ. Three levers: draft faster, draft better so τ rises, or verify smarter so T_verify falls.

The catch is that rejected tokens cost real compute. Under light load that compute would have idled anyway, so the waste is free and you should bet as much as you can afford. Under heavy load, every token you verify and then reject occupied batch capacity another user needed, and the bet gets funded out of somebody else’s session.

That single fact explains the state of the art. DeepSeek’s production configuration before DSpark was MTP-1: one drafted token per cycle. Longer draft blocks help an individual user perfectly well, but a static longer block, in the paper’s words, strictly degrades aggregate throughput under high concurrency due to excessive verification overhead. Everyone serving at scale has been deliberately leaving single-user speed on the table, because the static version of this trade takes capacity from everybody to give speed to one.

What DSpark adds

Two mechanisms. Only the second one moves the frontier.

1. Draft better: a little autoregression goes a long way

Drafters come in two families. Autoregressive drafters generate the draft one token at a time, conditioning each on the last. Good quality, but drafting time grows with block size, which forces tiny blocks and shallow networks. Parallel drafters emit the entire block in one forward pass, so drafting cost barely depends on block size, which buys a deep drafter and a long block.

Parallel drafting has one structural flaw: each position is predicted independently, so no position knows what the others chose. The paper’s example is the clearest way to see it. If the continuation could plausibly be “of course” or “no problem”, an independent drafter can happily emit “of problem”, because position 1 averages over all futures while position 2 averages over all pasts. Acceptance decays as you move down the block.

DSpark keeps the heavy backbone fully parallel and bolts on a tiny sequential head that walks the block left to right, nudging each position’s logits based on what the previous position actually sampled. In the shipped configuration that head is a low-rank table of token-to-token transitions. Once position 1 has sampled “of”, it boosts “course” and suppresses “problem”.

The trade is lopsided in its favour. Accepted length rises 16% to 18% over the parallel baseline and 27% to 31% over the autoregressive one, across math, code and chat, for 0.2% to 1.3% added round latency. A two-layer DSpark beats a five-layer pure-parallel drafter. Injecting a little sequential structure turns out to be far cheaper than stacking more parallel capacity.

2. Verify smarter: a throttle that costs the user nothing

Longer draft blocks on their own would make the throughput problem worse. This is the half that fixes it, and the half we care about.

DSpark attaches a confidence head that outputs, for each draft position, the probability that this token survives verification given that everything before it survived. Multiply along the block and you get the survival probability of every possible prefix length. Those numbers have to be genuinely calibrated rather than merely correctly ranked, because they feed an arithmetic decision, so raw confidence (overconfident by 3% to 8% expected calibration error, as neural confidence always is) gets a per-position temperature fitted on held-out data to bring it to about 1%.

Then the hardware-aware scheduler uses them. Profile the engine once at startup to learn steps per second as a function of batch size. Now, every step: expected accepted tokens is the sum of survival probabilities across all admitted candidates, batch size is how many tokens you are sending the target model, and expected system throughput is the product of the two. Sort every candidate token from every active request by survival probability, admit them one at a time, and stop the moment the product stops rising.

Here is what that looks like on live traffic.

The bottom row is the one to look at. The flat blue line at 2 is the old configuration: a static budget, at every load level, forever. The green is DSpark: roughly 5 to 6 tokens per request when the system is quiet, decaying smoothly toward 3 as concurrency climbs past 150. The scheduler answers one question every step, “does one more speculative token still pay for itself at this batch size”, and the answer moves on its own as the room fills up.

That is a throttle, and it is the good kind. It has the shape of every other scheduling lever in a serving stack, and it is a better instrument for one reason: it costs the user nothing. A priority tier slows somebody down so somebody else can be fast. DSpark’s scheduler withdraws a bet that was going to lose. The tokens it declines to verify are the ones the model was likely to reject.

Truncating the block based on confidence is exactly the kind of thing that quietly breaks the losslessness guarantee, because an admission decision that peeks at a token it has not committed to leaks information into the sampling. The paper carries a counterexample for this in an appendix, and the production version sizes the truncation from confidence measured two steps earlier, building a causal barrier between the decision and the tokens it could have cheated on. So even the adaptive, load-dependent version reproduces the target distribution exactly. The care taken there is part of why we took the result seriously.

Why any of this is our problem

We process a trillion tokens a week. That runs on our own money and time rather than a venture round, at pricing we have kept deliberately generous, so how much work we get out of each GPU is what makes the combination possible at all. Every architecture that fits more sessions on the same silicon is room we would otherwise have to find in the price.

This is DeepSeek’s architecture and their serving stack, and we cannot apply it to GLM 5.2 by flipping a switch. It travels, though, and that is the actual bet. A DSpark drafter trains against a frozen target model, and DeepSeek open-sourced DeepSpec, the training repository, with Eagle3, DFlash and DSpark in it. The paper itself trains DSpark drafters for Qwen3 at 4B, 8B and 14B and for Gemma4-12B. The compressed-attention half is spreading just as fast, across Kimi, Qwen, MiMo and Ring. The real question is how quickly the models we already serve arrive with this shape built in, and we wanted the operational answers in hand before they do.

What we served

The publisher’s own MIT-licensed weights (DeepSeek V4 Pro: 1.6T total parameters, 49B activated), unmodified, plus the DSpark draft module. A 384K context window, three reasoning modes (none, high, max) defaulting to high, text only. Seat-gated through the Labs page, 50 seats with most of them rotating so more people got a turn.

The model’s public description set the terms:

DeepSeek V4 Pro as a Labs experiment, open for a short test window: temporary, not a permanent model. […] It is offered at limited capacity and availability, so expect it to be flaky and to go down under load: crash it, give it a moment, and try again. For production work we recommend umans-coder or umans-glm-5.2.

How it went

The speed was real and it arrived in the shape people actually care about. One participant ran a long agentic task through it, 153 tool calls end to end, finishing in 12 minutes against roughly 20 to 25 for the same shape of task on our Kimi and 50-plus on GLM 5.2 in production. Time to first token came in at p50 3.9s, p90 12.0s, p99 15.3s, with zero timeouts against a 20 second cutoff. They were the first to point out that the run failed the test it was aiming at, which was beside the point of the measurement. Treat it as one anecdote with numbers attached, taken on a quiet experimental node. The p99 with no timeouts was still what we cared about, because a healthy tail is precisely what we have been failing to deliver on our busiest model.

The day-to-day reports were consistent with that:

it’s working well for me, fast, thank you!

Deepseek followed instructions, didn’t loop or hallucinate at all. Isn’t nearly as verbose as GLM 5.2. Overall, extremely happy.

Fast. Not too deliberative. Seems to be making good decisions in the codebase I’m working in.

And one participant put their finger on why a fast lane earns its place even when it is not the smartest model in the lineup:

theres a clear place for it, but thats not because “deepseek is fast”, its because the other options are not nearly as fast in terms as ttft and tps […] this is a great model to switch to for walking some steps where glm 5.2 will sit for minutes trying to connect

Then the failures arrived, and every one of them lives in the checkpoint.

Tool calls leaking as prose. DeepSeek V4 emits tool calls in its own DSML format, and testers repeatedly got raw DSML in the response body or buried inside the reasoning stream instead of a structured tool call:

The provider returned a tool call in the wrong channel and format: raw DSML inside reasoning_content instead of structured tool_calls. That violates the API contract the harness expects.

Language bleed. Chinese appearing in output, well outside the reasoning:

Hoping I’m not the only one seeing DeepSeek constantly trying to talk to you in Chinese, and not just in its reasoning either.

Precision decaying deep in context. At roughly 200k tokens, one tester watched it write SameSite: '/lax' where sameSite: 'lax' was meant, then catch its own mistake in the next breath. Another run produced untipdated for untyped. Others hit file edits failing often enough to eat the speed advantage:

feels like the speed of the model has been undermined by how many mistakes it’s made / its lack of capability, I’m pretty sure another model would have finished this by now

A head-to-head one tester ran told the same story from the other side: on the same “build me a Block Blast clone” task, GLM 5.2 got it in one shot while DeepSeek needed three iterations with console errors pasted back in, though it got there faster in wall-clock time and with a cleaner layout once it did.

These are the signatures of a preview build: a malformed tool-call boundary, a language drift, edit precision degrading with context length. We were serving DeepSeek’s released weights with a distribution-preserving accelerator, so the checkpoint is the only place they could have come from.

The tool-call one is half ours to fix. DeepSeek’s format expects the reasoning block to close before a tool block opens, the model sometimes forgets to close it, and the parser then swallows the entire tool call as reasoning so the harness never sees a structured call. vLLM merged a recovery for exactly this on July 4, treating the tool-start marker as an implicit end of reasoning. The equivalent fix in the engine we run is on our list, and we intend to send it upstream rather than carry it privately. It is a real bug with a real fix, and it is also a tell: a production-ready checkpoint would trip it far less often.

What we took from it

• DSpark lets us serve more users at once while keeping each session fast enough. The frontier moved, on live traffic.
• The DeepSeek V4 architecture is now mature enough to serve at scale. It runs, it holds a tail, and the serving-side unknowns we went in with came back answered.
• The model itself is solid. On its good runs it followed instructions, stayed on task, and did not pad.

V4 Pro stays out of the lineup. It is a preview build, the issues testers hit are model-side, and DeepSeek has confirmed a better version this month. We would rather carry the learnings into the version that ships than spend the effort patching a preview.

What we ship

The compressed-attention idea is the one to watch, because it changes the arithmetic that has capped us all year. Attention that costs token × rep(token) instead of token × token means long sessions stop being expensive customers, and long sessions are what agentic coding is made of. DSpark sits on top of that and hands back the other half: it fills the idle time between tokens with work that is guaranteed to produce the model’s own output, and it knows to stop betting when the machine gets busy.

So the lineup stays as it is for now, and our expectations for the next model worth serving have moved. We know what to look for, we know what it buys, and we know what to do with it when it arrives.

Given the choice between deciding which of you gets to be fast and making the machine serve more of you at once, we will pull the second lever every time. This is us learning how.

Thanks to everyone who spent two days throwing real work at a preview model and telling us bluntly where it broke. The reports in that thread were sharper than anything we would have found on our own, and they are the reason we can say with confidence which of these problems are ours to fix and which are not.

Appendix

A. The production frontier anchors

From the DSpark paper (Figure 7), DSpark against the MTP-1 production baseline on live traffic:

The two nominal figures are the anchors where MTP-1 collapses to a very small batch, so the ratio inflates. DeepSeek’s own reading, which we share, is that they show the frontier reaching interactivity levels the baseline could not serve at all.

B. Attention cost at 1M tokens

From the DeepSeek-V4 technical report, measured against DeepSeek-V3.2 at a 1M-token context:

Against a conventional BF16 GQA baseline at the same context length, V4’s KV cache lands at roughly 2%.

C. Draft quality, offline

Accepted tokens per decoding round, macro-averaged across math, code and chat, with the confidence scheduler disabled so this isolates draft quality:

The gap widens with block size: at a draft length of 15 the improvement over DFlash reaches 30% on math, 26% on code and 22% on chat, because pure parallel drafting decays fastest exactly where the block is longest. Added round latency across that whole range stays between 0.2% and 1.3%. These targets are Qwen and Gemma models, which is what makes the technique portable.