1.8× more prefill per B300 node: MegaMoE, CUDA graphs, and the cache that forgot how to save

53k

input tokens/s per prefill node at 128k, before

~94k

with the five changes combined (measured pieces)

+59%

cold prefill from DeepGEMM MegaMoE alone

56%

second-turn cache hits under Valkey stalls, from 0%

Where the time goes#

I run GLM-5.3 with prefill and decode on separate nodes, and push it with long-context, multi-turn chat. At 128k tokens of context almost every token a request costs is an input token, so once load rises, requests queue for prefill: adding concurrency stops adding throughput, and time to first token climbs instead. The decode side sits mostly idle.

That makes one number matter more than any other: how many prompt tokens per second can one 8-GPU prefill node get through? When I started, the answer was about 53,000. Everything below either raises that number or stops it from being wasted.

Five changes, one image#

I built and measured each of these on its own over two weeks. They haven’t yet run together, so the combined number is a projection from measured pieces.

The first two are about raw compute. The last three are about LMCache, the layer that should let the second turn of a conversation skip most of its prefill. They matter more than their throughput numbers suggest, because of the bug in the last row.

Why MegaMoE is the big one#

GLM-5.3 has 256 routed experts and sends every token to 8 of them. On a prefill node with 8 GPUs and expert parallelism, each GPU holds 32 experts, so most of a token’s work happens on another GPU. The default path handles that in three separate steps: all-gather the tokens, run the expert GEMMs, reduce-scatter the results. Profiles of that path showed NCCL taking 34–39% of prefill GPU time, much of it GPUs waiting for the slowest peer.

MegaMoE replaces the three steps with one persistent DeepGEMM kernel that moves tokens over NVLink while the tensor cores are busy. After the switch, step time at 128k fell 38%, and exposed communication dropped from about 35% to 5–8%. Accuracy held: GSM8K went from 96.36% to 96.59% (the same answer on 98.3% of questions, McNemar p = 0.63), and the needle-in-a-haystack test passed 20 of 20 up to 187k tokens. The vLLM integration is upstream as #63119.

With the MoE fast, the CPU became visible. At 16k the GPU was idle 19.5% of the time because nothing in prefill was captured as a CUDA graph. Piecewise graphs remove most of that launch overhead, which is why the two changes belong together.

The cache that forgot how to save#

My benchmark sends two-turn conversations. The second turn repeats the first turn’s prompt and adds a little, so in principle about half of all prompt tokens are already cached. At concurrency 1 the cache did hit about 55% for prompts up to 32k. At 64k and above, and for long prompts under load, hits fell to 0.5–3%.

One prefill node’s logs held the clue: 8,720 lines of store only 0 total chunks and 12,958 lookup timeouts. The CPU pool had 30 GB per GPU rank, plenty of room, yet it was full of chunks nothing could evict. Tracing every reference to those chunks led to LMCache’s job executor:

# lmcache/v1/storage_backend/job_executor/pq_executor.py, worker loop
result = await fn(*args, **kwargs)
# done was cancelled by a 2 s timeout -> InvalidStateError
done.set_result(result)
except Exception as e:
    # raises again, uncaught: the worker coroutine exits
    done.set_exception(e)

# The fix: only answer a caller that is still waiting
if not done.done():
    done.set_result(result)

A lookup wrapped in asyncio.wait_for times out after 2 s and cancels its future. The worker finishes the job anyway, tries to deliver the result to the cancelled future, crashes in its own error handler, and dies. Each timeout costs one worker. When the last one is gone, every Valkey write waits in the queue forever, and each waiting write keeps its chunk pinned in the CPU pool. The simulation below shows that sequence.

To confirm it, I reproduced the failure on two nodes. Turn one ran on one node and turn two on the other, so every second turn had to come from Valkey, while Valkey paused for a few seconds at a time:

The 56% is held down by how hard the test stalled Valkey: about 30% of the time. Without stalls the same setup hits 98%. Under real load Valkey is slow rather than paused, so hits should land much closer to the top. The fix is upstream as #6009, with tests that fail on the current release.

The two LMCache speedups#

Async save and the lookup wake-up are smaller, and both were found by looking at idle time. A profile of a 4k-prompt step showed the GPU busy 99.5% of the time with no KV connector and 88.9% with LMCache. The CPU waited for each forward pass to finish before copying KV out, one 256-token chunk at a time. Moving that copy to a side stream brought the GPU back to 99.1% busy.

The lookup was simpler. The async client sent its request to all 8 ranks and then slept 10 ms, every time, although the ranks answered in under half a millisecond. Waking up on the reply cut single-request latency at 4k from 81 ms to 66 ms, the same as running with no cache at all. That one is upstream as #6003.

What didn’t make the cut#

Some measured experiments didn’t earn a place:

• NVFP4 experts gave the same prefill speed as FP8 MegaMoE, so the format change isn’t worth its accuracy risk.
• Overlapping the context-parallel all-gather could save about 2%, but every version produced wrong outputs.
• Separate data-parallel prefill pools won 9–11% on prompts up to 16k and lost 34% at 128k.
• TensorRT-LLM: I found no public B300 result showing it ahead of SGLang or vLLM, and porting the stack would take weeks.

What’s next#

The next experiment is one build with all five changes, measured end to end with decode attached. Each patch applies as exact text and the build runs its unit tests, so a mismatch fails the build rather than producing a silently different binary. Two signals in the prefill logs should stay at zero: store only 0 total chunks, and growth in the executor queue.

Two risks are already known. The per-node numbers come from single-node runs, so the handoff to decode over NIXL will take some of the gain. And with more prefill nodes per decode node, one decode node carries more streams than anything I have tested so far. The first end-to-end run will show both.

Upstream contributions#

• vLLM #63119: DeepGEMM FP8 MegaMoE for GLM-5.x under prefill context parallelism
• LMCache #6009: a cancelled caller no longer kills the executor worker
• LMCache #6003: async lookup returns as soon as every worker answers

Measured October 2026 on 8×B300 nodes running GLM-5.3 in FP8 with prefill context parallelism (PCP8) and expert parallelism, on real-text prompts. Background on the KV-cache tier is in the earlier write-up and the paper When the KV Cache Is Slower Than the GPU.