I spent a while trying to make an LLM serving stack faster. The bill barely moved. Then I looked at what the GPUs were actually doing and found the real problem in about ten minutes.
Six B300s were live at 17:06 IST. Each was averaging 0 to 0.2 running requests.
Not 0.2 requests per second. 0.2 requests in flight. They were sitting there, warm, costing money, doing essentially nothing. The engine was never the constraint. The autoscaler configuration was.
Before touching anything, I instrumented what the day actually looked like:
That is a workload with one narrow peak and twenty-three hours of near-idle. It had been provisioned as though the peak were the steady state.
The peak floor — the capacity held whether or not anyone was using it — was roughly 4.7x higher than it needed to be. Rewriting the scaling config so the floor tracked actual concurrency collapsed that gap, and off-peak fell further still.
Daily spend dropped about 70%, with a hard ceiling replacing what had been an open-ended range. Measured quality did not change. I checked, because a cost win that quietly degrades output is not a win.
Cutting idle capacity is arithmetic. The interesting problem was speculative decoding.
We ran a flat speculative setting — 5 draft tokens per step, always. At low batch this is great: the draft model runs ahead, the target model verifies cheaply, latency drops. At high batch it collapses. Every speculative token you propose is verification work multiplied across the batch, and past some concurrency the verification cost exceeds what the speculation saves.
So the schedule got tapered by batch size:
num_speculative_tokens_per_batch_size = [[1, 4, 5], [5, 12, 3], [13, 64, 1]]
Batches of 1–4 get 5 draft tokens. Batches of 5–12 get 3. Batches of 13–64 get 1, which is effectively off. Speculation where it pays, nothing where it does not.
Result on 60k-context requests: +57% throughput at 16 concurrent, +40% at 32, with single-stream latency untouched. Production settled at P50 1.58s, P90 4.47s.
If 5 draft tokens beat 1, surely 7 beats 5.
It does not. P90 regressed to 7.29s. The extra draft tokens are wasted whenever the target model rejects them, and the rejection rate climbs faster than the speedup. Reverted, and written down, so nobody on the team re-runs it in three months.
I think recording negative results is underrated. The cost of re-learning something is paid every time somebody new has the same reasonable idea.
Measure concurrency, not throughput. Throughput answers "how fast can this thing go." Your bill is set by how much capacity sits warm while nothing arrives. Those are different questions and only one of them is on the invoice.
Tune against your real traffic shape. Every default in a serving stack is picked for a steady, saturated load. Almost no production workload is steady and saturated. Ours had one 40-minute peak and a long flat tail, and nearly every useful setting followed from that fact.
Check quality when you cut cost. Cheaper and worse is easy. Cheaper at parity takes a gold set and a similarity floor you agree on before you start changing things.