More throughput from fewer GPUs: profiling and fixing GLM-5.3 prefill on NVIDIA B300

Long-context serving lives and dies by prefill, the pass that reads the whole prompt and produces the first token. For a 200k-token prompt, prefill is nearly all of the time to first token (TTFT). The standard way to avoid paying for it twice is to keep the attention KV cache of earlier prompts (the per-token key and value vectors every layer computes) and reuse it whenever a new prompt shares a prefix. LMCache does this for vLLM with two tiers: CPU memory local to each pod (L1), and a shared remote store (L2) that every pod can read and that survives restarts.

On a GLM-5.3 deployment serving an SLA workload of 9k to 200k-token prompts, turning on the L2 tier left median TTFT unchanged but hurt the tail badly. Prefill p99 rose from 3.96 s to 17.3 s at 16k tokens, and from 6.06 s to 25.0 s at 32k. This post covers how we found the causes, why each one behaved the way it did, and the changes that brought a hit-heavy workload to p50 0.40 s and p99 0.67 s with 98.5% of prompt tokens served from L2.

5

B300 nodes instead of 11, passing the same SLA at 100% success

3.6×

tokens per minute per node against the aggregated deployment (0.70M → 2.55M)

+34%

long-context prefill throughput per node (43.2k → 58.0k tok/s)

125×

lower p99 on cache-hit-heavy traffic (83.65 s → 0.67 s)

System architecture#

GLM-5.3 is a large mixture-of-experts (MoE) model with multi-head latent attention (MLA), the DeepSeek-style attention that compresses each token’s keys and values into one small latent vector shared by all heads. We serve it in FP8 (weights and KV cache) on p6-b300.48xlarge nodes, each with 8 NVIDIA B300 GPUs and 16 EFA network interfaces, using prefill/decode (P/D) disaggregation. Because prefill is compute-bound and decode is memory-bandwidth-bound, each runs on its own pods with its own parallelism.

Prefill: context parallelism with expert parallelism#

A prefill pod uses prefill context parallelism (PCP) across its 8 GPUs (--prefill-context-parallel-size=8): the prompt is split along the sequence, so each GPU (each rank) processes about one eighth of the tokens. Attention needs every token to see the tokens before it, so the ranks exchange their KV with an AllGather. After the exchange, every rank holds the full prompt’s KV.

We chose PCP over tensor parallelism (TP) because of MLA. TP splits attention heads across GPUs, but MLA’s latent KV is shared by all heads, so TP replicates the full KV and the attention input on every GPU. Per-GPU attention work and KV memory barely shrink. Splitting the sequence does shrink them, so one long prompt finishes much sooner.

The MoE layers use expert parallelism (EP8): each GPU hosts one eighth of the experts, and token activations move between GPUs with AllGather and ReduceScatter around the expert GEMMs.

This design has one property that matters for everything below: the 8 ranks move through every step in lockstep. Each step contains NCCL collectives, and a collective can’t finish until every rank arrives. Anything that delays one rank delays all eight.

KV hand-off and caching#

Prefill pods run a vLLM MultiConnector: NIXL first, LMCache second. NIXL transfers the finished KV directly from prefill GPU memory to decode GPU memory over EFA. LMCache splits the KV into 256-token chunks, about 12.4 MB each for GLM-5.3 in FP8, and stores them in L1 and L2. A lookup returns the longest prefix of the prompt whose chunks are all present. A gap anywhere means everything after it is recomputed.

Methodology#

We wanted measurements that match production without touching production. We rebuilt one production prefill pod’s request stream for a 16k-token SLA cell from router logs and the client database: 683 requests, with their original prompts and send times. We then replayed it against an isolated test engine on a free B300 node. The test engine used the identical container image, command line and environment, with two differences: the LMCache controller was off, so no production router could see the engine, and it had its own Valkey instance. The P/D hand-off was left out, so the tests are prefill-only. That keeps the measurement on the part of TTFT that production traces implicated: for requests with TTFT over 5 s, prefill was about 90% of TTFT.

Tools:

• Nsight Systems captured all 8 GPU workers (--trace=cuda,nvtx --capture-range=cudaProfilerApi), started and stopped through vLLM’s --profiler-config '{"profiler":"cuda"}' hook so the capture covered only the window of interest. LMCache’s NVTX ranges mark its store, lookup and retrieve calls on the timeline.
• py-spy took stack dumps of the engine processes, for stalls with no GPU activity.
• Valkey INFO commandstats was sampled every 5 s for server-side GET counts and per-call time, along with the main thread’s CPU time.

Production metrics ruled out queueing early. Mean queue time per pod was 0.02–0.15 s, and at 16k tokens, 183 of the 401 prefills that took over 3 s ran on a pod with nothing else in flight. The latency was inside the prefill itself.

Bottleneck 1: one rank doing every store#

In MLA mode, LMCache stores KV only from worker 0 (save_only_first_rank). The reasoning is sound under TP: every rank holds the same latent KV, so one copy is enough. Under PCP the KV is also replicated, after the AllGather, but now rank 0 copies the whole prompt’s KV to CPU inside the step, while the other seven ranks have nothing to do but wait for it at the next collective.

The Nsight timeline for two concurrent ~200k-token prompts shows it directly:

• Rank 0 wrote 19.1 GB of KV to CPU during the window, and its GPU was 82% busy against 98% on the other ranks.
• Ranks 1–7 accumulated about 1.55 s each of AllGather wait in a 9.5 s window.

That wait is not network time: in a collective, every rank’s time looks like communication until the slowest participant arrives.

The sharded store#

Since every rank holds the full KV after the AllGather, any rank can store any chunk. We assign chunk c to rank c mod W, with W = 8, and each rank stores, looks up and loads only its own chunks. The store work is spread evenly, and each rank’s L1 share is a 1/W slice of the pod’s CPU budget, so total memory is unchanged.

Two things have to hold for this to be correct. The ranks must agree on how much of a prompt is cached, and a load must never let one rank skip a collective that the others enter, because that hangs the step.

Load. Every rank issues the same collectives in the same order. First, one small exchange of each rank’s result: what it loaded, plus a fingerprint of the chunk list. Then one broadcast per chunk from its owner. If a rank failed to read a chunk, the exchange tells every rank, and they all treat the prefix as ending there. A miss or an error never lets one rank skip a broadcast the others enter.

Bottleneck 2: an activation kernel at ~2% of HBM bandwidth#

The same timeline showed FlashInfer’s fused SwiGLU + FP8 re-quantization kernel, which runs between the two expert GEMMs, taking 1.57 ms per call and 17% of prefill GPU time. It launches one CTA (thread block) per (token, top-k) slot of the all-gathered batch: a grid of 16 × 8192 × 8. Under EP8, about 7 in 8 of those slots route to experts on other GPUs, yet their CTAs still run the block reduction before discovering they have nothing to do. The kernel moved so little data for its runtime that it reached only about 2% of HBM bandwidth, which points to wasted work rather than a memory limit. FlashInfer PR #3969, an optimized DeepSeek FP8 activation kernel, takes it from 1572 µs to 230 µs per call. We built it for SM103 (B300) and validated it with long-context needle tests.

Bottleneck 3: L2 bandwidth against the cost of recompute#

On the production replay, where each prompt’s KV came from determined its latency:

An L2 hit was almost 3× slower than not caching at all. A simple cost model explains it.

A cost model for a cache hit#

Let n be the prompt length in tokens, s the KV bytes per token, B the bandwidth the cache tier delivers to one prompt, and R the engine’s prefill rate. A hit is only worth taking if loading is faster than recomputing:

T_load = n · s / B        T_recompute = n / R
hit pays off  ⇔  B > s · R

The prompt length cancels, so the break-even bandwidth depends only on the model and the hardware. For GLM-5.3 in FP8, s = 12.4 MB / 256 tokens ≈ 48 KB per token. From the cold replay, B300 prefills a 16k prompt in 0.31 s, so R ≈ 53k tokens/s:

B* = s · R ≈ 48 KB/token × 53k tokens/s ≈ 2.6 GB/s per prompt

A 16k prompt is 64 chunks, about 0.8 GB. One Valkey instance served 0.78–0.89 GB/s, so the model predicts a load time of about 0.9 s, which matches the 0.83–0.89 s we measured. At that bandwidth, L2 sits at about a third of break-even. A fast GPU makes recompute a demanding baseline.

Why one Valkey instance is the ceiling#

Valkey executes every command on a single main thread; its I/O threads only parallelize socket reads and writes. During the hit-heavy runs that thread spent 73–74% of wall-clock time executing GETs, and 89–91% in the busiest minute. We confirmed the ceiling in isolation with LMCache’s native RESP client and real chunks:

Why synchronous loading runs away#

With synchronous loading, which was the deployed setting, a load happens inside the engine step. Under PCP, that freezes all 8 GPUs. The engine then behaves like a single-server queue whose service time per hit is at least the load time. Its utilization is the arrival rate multiplied by the service time:

ρ = λ · T_service
hit-heavy replay: λ = 450 requests / 374 s = 1.20 req/s
one instance:    T_service ≥ 0.8 GB / 0.9 GB/s ≈ 0.9 s      ⇒  ρ ≈ 1.07 > 1
eight instances: T_service ≈ 0.8 GB / 10.76 GB/s ≈ 0.074 s  ⇒  ρ ≈ 0.09

With ρ > 1, the queue has no steady state: it grows for as long as the load lasts. Concurrent loads compete for the same server, which raises the service time further. We replayed 450 requests that fit in cache, with Valkey primed first, and measured exactly that:

• median latency of 3.5 s in the first third of the run, 9.0 s in the second and 52.5 s in the last;
• p99 of 83.7 s;
• up to 59 requests in flight;
• GPUs only 7% busy.

Asynchronous loading, the default LMCache path with the rank-0 store, capped latency at 3.2–3.8 s, but only by giving up. 49% of requests hit the 3 s async-lookup timeout and recomputed while their prefetch kept running in the background.

One Valkey instance per rank#

Raising B requires more server threads, so more instances. Spreading keys over several instances normally needs a cluster protocol or client-side consistent hashing. The sharded store already provides a static partition: rank r only ever touches chunks with c mod 8 = r. Instance r can therefore hold exactly rank r’s chunks, with no routing logic and no traffic between instances. Rank r of every prefill pod owns the same chunk indexes, so a prompt computed on one pod is still found by every other pod. The client change is a single substitution in the remote URL:

# Engine: "{rank}" in remote_url becomes the worker's rank (LMCache#5548)
LMCACHE_REMOTE_URL=resp://valkey-lmcache-{rank}.vllm.svc.cluster.local:6379
LMCACHE_EXTRA_CONFIG={"pcp_shard_store": true, "resp_num_threads": 32}
LMCACHE_ENABLE_ASYNC_LOADING=True

# Helm chart: one Deployment + Service per shard, spread across nodes
valkeySpec:
  shards: 8            # valkey-lmcache-0 .. valkey-lmcache-7
  maxMemory: "150gb"   # per shard, LRU eviction
  ioThreads: "4"

Each rank’s share of a 16k prompt is about 100 MB, fetched from its own instance in parallel with the other seven. At the measured 10.76 GB/s aggregate, the whole prompt arrives in about 74 ms, well inside the 0.31 s recompute budget.

Bottleneck 4: two client bugs that stalled the engine#

Bandwidth explained the steady cost. Two rarer events explained the worst outliers, and both left a GPU timeline that was simply empty.

A lost completion: 3 s per request#

One production pod began logging a 3 s LMCache lookup timeout on every routed request, starting at one point in time: 16 out of 16 requests in one minute, and 55 in total. The GPU timeline showed nothing, so we turned to py-spy. Every rank’s lookup thread was blocked in batch_exists_sync → fut.result() with no request outstanding. The synchronous methods of the native client submitted a request to the C++ client and only then registered the Python future that would receive the result. The completion was delivered on the event-loop thread:

This is a check-then-act race between two threads sharing the pending-future table. Per call it is rare, but the sharded store sends every lookup to all 8 ranks instead of only rank 0, multiplying the exposure by 8, and a single loss is permanent. From then on, every scheduler lookup waits out its 3 s timeout and reports a miss.

The fix holds one lock across submit-and-register and across the drain, so a completion can’t be drained before its future exists. A test with a fake native client forces the bad interleaving: 2 of 3 cases fail on the old code, and all pass with the fix. The A/B replays showed no measurable latency cost.

A GET of a missing key: a hang that wedges the connection#

RESP encodes a GET reply as a bulk string, $<length>\r\n<bytes>\r\n, and a missing key as the null bulk string $-1\r\n, just 5 bytes. The native connector knew how large a chunk should be, so it read a fixed-size header with recv_exactly: $, the 8-digit chunk size, then \r\n, 11 bytes in all. When the key was missing, only 5 bytes ever arrived. The read never completed, and that connection never served another request.

The window for this is real: LMCache checks for keys with EXISTS at lookup time and reads them with GET at load time. An L2 running LRU eviction can drop a chunk in between, and the fuller the cache, the more often that happens. A synchronous load then waits for blocking_timeout_secs (10 s) with all 8 GPUs stalled, and the client permanently loses a connection.

The fix parses the header up to CRLF and treats $-1 as a per-key miss, with the reply fully consumed so the connection stays usable. The C++ batch completion already carried per-key results, but the Python side ignored them and returned every buffer as loaded KV. It now frees the unfilled buffers and returns the loaded prefix. Against a real Valkey, with a key deleted between EXISTS and GET:

Asynchronous loading for the sharded store#

With 8 instances, the queueing model gives ρ ≈ 0.09, so synchronous loads keep up. We still wanted asynchronous loading, because it removes the failure mode completely: a slow load delays only its own request, while the engine keeps running others. LMCache rejected that combination in shard mode, because the async path was built on rank 0 doing all the lookups.

In async mode, the scheduler sends a lookup to every rank’s lookup server. Each rank prefetches in the background, and reports the cumulative token count its storage layer found as a contiguous run. The scheduler takes the minimum. To reuse that machinery unchanged, each rank prefetches only its own chunks and passes a custom cum_chunk_lengths: the start token of its k-th owned chunk, plus the end of the prompt. The storage layer then reports exactly a_r from the proof above, measured in tokens, and the minimum is the same prefix as in the synchronous path. A property test over 3,000 random chunk-presence patterns checks that equivalence.

One edge case had to change. When a rank found nothing, the storage layer reported 0 hits. With a sharded layout that is wrong. If rank 1’s first owned chunk (chunk 1) is missing, the correct answer is 256 tokens, because chunk 0 belongs to rank 0 and may well be present. Reporting 0 would throw away a valid prefix. The storage layer now reports the first entry of cum_chunk_lengths, which is 0 outside shard mode, so default behaviour is unchanged.

Turning async on also exposed a key-naming bug. In MLA mode, the remote backend rewrites every key to worker 0’s name, so all ranks share one copy. The async lookup path did that rewrite, but the async GET did not. Ranks 1–7 therefore found their chunks and then asked for keys that don’t exist, which is how we found the missing-key hang above.

Correctness#

Bit-exact comparison is not a usable test here: the engine is not deterministic from run to run, and two cold runs of the same prompt already produce different text. We tested semantics instead, with six needle-in-a-haystack prompts from 12.8k to 166.7k tokens, each with a random code at a random position. Each prompt was answered four times:

• cold;
• from L1 through async prefetch;
• from L2 through async prefetch after an engine restart, with 12,800 to 166,656 tokens loaded;
• cold again, as a control.

The code was found in 24 of 24 answers. Answers served from cache differ from cold ones no more than the two cold runs differ from each other. All runs logged zero lookup timeouts, zero async timeouts, zero double frees and zero errors.

Results#

Every run used the same 450-request hit-heavy stream (16k-token prompts) on one B300 prefill node, with L2 primed before each run. 98.5% of prompt tokens were served from L2:

H3 through H5b completed all 450 requests with zero lookup timeouts, zero GET timeouts and zero errors. In H5b the Valkey shards averaged 8.9 ms of server time per 12.4 MB GET.

As the model predicts, at ρ ≈ 0.09 async loading has no backlog to absorb, so H3 and H4 share a median and differ only in the tail. The remaining ~0.1 s over recompute at the median is spent in the engine, in the per-chunk broadcast and load path, rather than in Valkey.

Full deployment: 65% more throughput on 5 nodes instead of 11#

We then ran the whole configuration as a P/D system on our 200k-token SLA benchmark: 4 prefill and 1 decode B300 nodes, with the 8 Valkey shards spread across them. The benchmark sends a sustained mix of long prompts and checks TTFT against fixed limits: P90 under 70 s, P50 under 30 s and average under 30 s.

The baseline is the deployment that first passed this benchmark a week earlier. It was aggregated: 11 replicas, one per B300 node, each running prefill and decode together with TP8 on all 8 GPUs. We also compare two disaggregated deployments that used more nodes than this one.

Against the aggregated baseline, the 5-node deployment serves 65% more tokens per minute (12.75M against 7.74M) on 55% fewer nodes. Average TTFT is the same (20.1 s against 19.9 s), and P90 is better (30.5 s against 34.0 s). P50 is 4.9 s higher, and output tokens per second are 16% lower, 1,377 against 1,631, still far above the gate’s floor of 30.

Against the other disaggregated deployments: with two fewer nodes, the 4P + 1D deployment delivers 90% of the 7-node deployment’s throughput. Against the 6-node deployment, it has one fewer prefill node but higher throughput and much better latency: TTFT P50 falls from 28.4 s to 18.0 s, P90 from 44.5 s to 30.5 s, and its margin under the 30 s P50 limit grows from 1.6 s to 12 s. Against 7 nodes, a third less prefill capacity costs about 3 s at P50 and 5 s at P90, still well inside the gate.

The aggregated baseline ran a week earlier, on an older image and router, and its run contained fewer requests (528 against 879). Tokens per minute already accounts for run length, and all runs were judged against the same TTFT gates.

Per-node figures divide delivered tokens per minute by node count. That is a fair comparison when each deployment is saturated, which this run clearly was. If an earlier run had spare capacity, its per-node figure understates what it could have done.

Discussion#

• Size a cache tier against recompute, not against a miss. The break-even bandwidth B* = s · R grows with the GPU’s prefill rate. On B300, GLM-5.3 needs about 2.6 GB/s per prompt before an L2 hit pays off. As GPUs get faster, remote KV tiers have to keep up.
• Partition the cache along the parallelism. Once each rank owns a static slice of the chunks, scaling the store out needs no protocol, just a URL template, and bandwidth scales with the number of instances.
• With lockstep parallelism, a slow rank stalls the whole group. Under PCP, any work on one rank inside a step, whether a CPU copy, a synchronous load or a stuck client thread, costs all 8 GPUs. Look for idle time in collectives before tuning kernels.
• When the GPU timeline is empty, inspect the CPU threads. Both client bugs left nothing on the GPU timeline. Stack dumps and server-side command statistics found them.

Upstream contributions#

• LMCache #5545: shard the MLA KV store across prefill-context-parallel ranks
• LMCache #5547: native clients no longer lose completions drained before the future is registered
• LMCache #5548: {rank} in remote_url selects a per-worker remote instance
• LMCache #5550: async GET reads the worker-0 keys in MLA mode
• LMCache #5551: a RESP GET of a missing key is a miss, not a hang
• LMCache #5553: async loading for the sharded PCP store
• FlashInfer #3969: optimized DeepSeek FP8 activation (applied and validated on SM103)

Measurements taken October 2026 on AWS p6-b300.48xlarge nodes, using vLLM 0.31.0, LMCache 0.5.5, FlashInfer 0.7.0.post1 with PR #3969 applied, Valkey 8.1, NIXL over EFA, and GLM-5.3 in FP8 with an FP8 KV cache.