← logs.beer

From 1,060 to 4,016 tok/s on one B300, and why it wasn't the kernels

2026-10-13

We serve DeepSeek-V4-Flash on a single NVIDIA B300 with vLLM, behind an OpenAI Responses API shim. The model is a 284B-total / 13B-active MoE with MXFP4 experts and block-FP8, about 165 GB of weights. Our prompts are long: rendered instructions plus about 93 tool schemas come to 60–90k tokens per request. The target was 5,000 tok/s aggregate decode on one GPU, and we started at about 1,060.

This post is about the profiling. The short version is that the GPU was mostly idle, and the fixes were configuration changes, not new kernels.

The roofline

At large batch, each decode step has to stream the weights it touches from HBM. With about 165 GB resident and about 8 TB/s of bandwidth, a full sweep costs about 20 ms. That means the engine can do about 48 steps per second. Throughput is roughly:

tok/s ≈ concurrent sequences × accepted tokens per step × steps/s

Speculative decoding accepts about 1–2 tokens per step. At that rate, 5,000 tok/s needs more than 100 sequences decoding at the full step rate. So the real question was whether the engine keeps 100+ sequences on the GPU every 20 ms.

Finding 1: CUDA graph capture size counts tokens, not sequences

We had set max_cudagraph_capture_size=64 to match max_num_seqs=64, so vLLM wouldn't capture hundreds of unused graphs at boot. That was wrong. With speculative decoding, a step carries seqs × (1 + spec depth) tokens. At depth 1, a capture size of 64 covers only about 32 sequences, and anything above that falls back to eager mode.

Doubling the batch cut throughput by 60%. The rule: set the capture size to at least 2 × max_num_seqs, and higher for deeper speculation.

Finding 2: the MoE autotuner only saw decode shapes

vLLM passes tune_max_num_tokens = max_cudagraph_capture_size to FlashInfer's TRT-LLM MXFP4 MoE autotuner. With a capture size of 64, every prefill chunk (up to 8,192 tokens) was clamped to the 64-token bucket. Prefill was running on tile configs tuned for decode.

We added a small runtime patch that widens the tuning range to the maximum batched tokens. Results:

• MoE FC1 went from 4.63 ms to 0.83 ms, and FC2 from 2.43 ms to 0.40 ms.
• MoE's share of prefill time dropped from about 65% to 21%.
• Priming the cold prefix went from 4.9 s to 2.7 s.

We didn't write a kernel. The autotuner just got to see the shapes it actually runs.

Finding 3: at c=160, the GPU was busy 27% of the time

With the graph size fixed, c=160 reached about 1,570 tok/s, still far below the roofline. nsys showed why, once we read it correctly.

There's a trap here. With --cuda-graph-trace=graph, the standard cuda_gpu_kern_sum report hides every kernel that runs inside a graph. The kernel table showed only a handful of small non-graph kernels. We exported the trace to SQLite and took the union of the GRAPH_TRACE, kernel, and memcpy intervals. That gave the real picture:

• Over a 10 s window, the GPU was busy 27% of the time.
• Each step took about 72 ms, of which about 19 ms was GPU work. That 19 ms is right at the weight-streaming floor.
• There were 3.4 s of idle gaps longer than 1 ms each.

The kernels were fine. The GPU was waiting on the host.

Finding 4: one API server can't feed a B300

We logged the scheduler's running and waiting counts every 0.5 s. waiting stayed at 0 while running climbed by only about 4.5 requests per second. Requests weren't queueing in the engine. They were stuck before it, in a single API server process that renders chat templates and tokenizes 60–90k-token prompts.

Two gotchas:

• Our serving container had no explicit CPU reservation, so it got a fractional default. More API server processes need real cores.
• We launch vLLM through a wrapper that monkey-patches request parsing. The extra API servers are spawned processes that re-import the entry file as __mp_main__. Patches must run at import time in every process, not only under if __name__ == "__main__", or the extra servers silently run unpatched.

Also, with multiple API servers, vLLM's /metrics double-counts steps. Measure tok/s on the client side.

Where it stands

We went from 1,060 to 4,016 tok/s on one B300. Almost all of it came from configuration changes we found by profiling: a capture size set in tokens, a wider MoE autotune range, 4 API servers, and 32 CPU cores. These changes are now rolling out to production.

The remaining ~1,000 tok/s to reach 5,000 is next. We'll run steady-state tests with longer outputs, 256 concurrent sequences, and deeper speculation at high batch. Kernel work is on the list too. One example is the fused RoPE + FP8 quantization step in prefill, which runs about 4× slower than its roofline. That comes only once the profile says the GPU is the bottleneck again.

Lesson: before writing a kernel, check whether the GPU is even busy.