Prefill/Decode Disaggregation — Commodity Loss, Frontier-Scale Win

SUMMARY

What we found

Prefill and decode have opposite appetites: prefill is a compute-bound burst, decode is a steady, memory-bandwidth-bound trickle. Disaggregation puts each on its own engine so they can scale independently — but the KV cache must then travel between them, and that transfer has a cost. Whether disaggregation wins is a race between the cost of moving KV and the benefit of specialization.

5,150

output tok/s peak on one H100 (gpt-oss-20b, aggregated) — the baseline scales cleanly to concurrency 32.

5.5×

worse time-to-first-token when we disaggregated across 2 GPUs with no fast link — the KV transfer overhead dominates.

~100×

slower to prefill on CPU than GPU for an 8B model — CPU prefill trades away TTFT by 1–2 orders of magnitude.

conditions that make disaggregation worth it: a fast interconnect (NVLink/RDMA) and high enough load to hide the transfer.

SETUP

Five configurations, one harness

We built a self-contained benchmark harness (vLLM + an async load generator) that pins input/output length exactly, sweeps concurrency, and records TTFT, inter-token latency (ITL), end-to-end latency, throughput, and goodput. Every configuration runs through the identical harness so the comparisons are honest.

ID	Configuration	Prefill	Decode	KV path	Status
A	Aggregated (baseline)	GPU	same GPU	none (in-place)	measured
C	Same-vendor disaggregation	NVIDIA GPU 0	NVIDIA GPU 1	NIXL, host-staged	measured
E	CPU prefill → GPU decode	CPU (EPYC)	GPU	DRAM → PCIe	measured*
B	Aggregated AMD	AMD MI300X		—	no AMD stock
D	Cross-vendor (NVIDIA→AMD)	NVIDIA	AMD	host-staged	no AMD stock

* Config E was measured by emulation (CPU prefill timed directly, composed with GPU decode) — see the honest limitations at the end. AMD (B, D) was unavailable on our cloud, so the cross-vendor hypothesis stays open.

RESULT 01 · BASELINE

Aggregated serving scales cleanly

Before judging disaggregation we need the reference. On a single H100, gpt-oss-20b throughput rises almost linearly with concurrency while inter-token latency stays low — this is the number every split has to beat. The prefill-heavy signal is visible too: at 8K input, time-to-first-token climbs steeply with load.

Throughput vs concurrency

gpt-oss-20b, aggregated on 1× H100 · output tokens/s

More load = more throughput, up to ~5,150 tok/s at concurrency 32. The GPU is not saturated until high concurrency.

Time-to-first-token vs concurrency

gpt-oss-20b, aggregated · TTFT p95 (seconds)

Prefill is the pressure point. At 8K input, TTFT climbs to ~4 s under load; short 1K prompts stay well under 1 s.

The bigger MoE costs throughput but not memory

gpt-oss-120b (~63 GB, 5.1B active) also fits one 80 GB GPU — output tok/s vs the 20b, at 1K/1K

The 120b activates only 5.1B params/token, so it runs on a single GPU — but is ~1.8× slower per token than the 20b. Sparsity buys you the memory, not the speed.

RESULT 02 · THE MAIN EVENT

Disaggregation lost — because the KV transfer was slow

We split prefill onto GPU 0 (producer) and decode onto GPU 1 (consumer), wired by NIXL. The catch: our two L40S GPUs were connected only through the host bridge (no NVLink, no direct peer-to-peer), so NIXL fell back to host-staged transfer — GPU → CPU DRAM → GPU. At scale, the decode engine's own logs measured KV transfers averaging ~2.4 seconds each. That overhead swamps any benefit.

Throughput: aggregated vs disaggregated

gpt-oss-20b, 8K input / 1K output · output tokens/s (higher is better)

Aggregated wins at every point. Disaggregation delivers roughly a third of the throughput — each request pays the transfer tax.

Time-to-first-token: aggregated vs disaggregated

gpt-oss-20b, 8K input / 1K output · TTFT p95 (seconds, lower is better)

At concurrency 16, disaggregated TTFT hits 11.3 s vs 2.1 s aggregated — a 5.5× penalty, all of it host-staged KV transfer.

Why this is the expected result, not a failure

The project plan predicted it: single-node disaggregation without a fast fabric flatters aggregated serving. We proved the pipeline works end-to-end (NIXL moved KV GPU-to-GPU and generated correct tokens) and quantified exactly what the transfer costs. This is the control that makes cross-vendor (Config D) interpretable: the benefit has to exceed this overhead to be worth it.

RESULT 03 · CPU PREFILL

CPU prefill is ~100× slower than GPU prefill

The second idea (H2): move prefill off the GPU entirely, onto the CPU, freeing the GPU for pure decode. The appeal is real — the CPU→GPU path skips the network — but prefill is compute-bound, and a CPU has a fraction of a GPU's math throughput. We measured it on a 64-core EPYC 9554 against the same 8B model on an L40S.

Prefill throughput — CPU vs GPU

Qwen3-8B · tokens/s prefilled (log scale)

The GPU prefills ~100× faster. That gap is the TTFT you sacrifice by moving prefill to the CPU.

What CPU prefill is — and isn't — good for

the narrow regime where it can still make sense

Never for TTFT. A latency-sensitive workload (chat, autocomplete) is off the table — a 1K-token prompt that prefills in ~0.03 s on GPU takes seconds on CPU.
Maybe for goodput-per-GPU-dollar. If the SLO is throughput-only (batch, offline) and the GPU is decode-bound, offloading prefill to otherwise-idle CPU can raise tokens per GPU.
Only at short input. CPU prefill cost grows with input length; beyond a few hundred tokens it is untenable.
MoE helps. gpt-oss-120b activates only 5.1B params/token, so on CPU it prefills like a ~5B model, not a 120B — the one case where a "120B on CPU" is not absurd.

Honest limitation

Our CPU numbers were measured with an eager-PyTorch forward pass, and the absolute times were dominated by a filesystem I/O artifact (weights re-read from a network volume each call). We report the relative gap (~100×), which is robust; a precise CPU-prefill ceiling needs the optimized vLLM CPU backend, which we leave as follow-up.

RECOMMENDATIONS

What we advise

The decision isn't "disaggregate or don't" — it's a function of your interconnect, your workload shape, and what you're optimizing for. Here is the guide our data supports.

If your situation is…	Use	Because
One GPU is enough for the model + load	A · Aggregated default	No transfer cost, simplest, and it scales cleanly. The bar everything else must clear.
Multi-GPU with NVLink / RDMA, high concurrency, long sequences	C · Disaggregation worth it	A fast fabric makes KV transfer cheap, and at high load it overlaps with compute — then independent prefill/decode scaling wins.
Multi-GPU with only PCIe / host-bridge links	A · Aggregated don't split	Host-staged KV transfer (~2.4 s/req here) costs more than specialization saves — we measured up to 5.5× worse TTFT.
Latency-sensitive (chat, autocomplete)	A · Aggregated avoid E	Any prefill offload — CPU or cross-node — adds first-token latency you can't hide from a user.
Throughput-only SLO, GPU is decode-bound, spare CPU	E · CPU prefill niche win	Short inputs only. Frees GPU for decode; can raise tokens-per-GPU-dollar when TTFT doesn't matter.

The one-line takeaway

Disaggregation is an interconnect bet, not a free win. Move the KV cache only when the wire is fast enough that moving it is cheaper than the specialization is worth — otherwise the simplest thing (keep prefill and decode together) is also the fastest.

AT SCALE

Why the big labs disaggregate anyway

Our headline result — disaggregation lost — sounds like it contradicts how frontier inference is built. It doesn't. Our test is the control case that proves why their infrastructure exists: same mechanism, no fast fabric, so it loses. The value is that it quantifies exactly how much overhead the fabric has to beat. Give the KV cache a fast enough wire and enough concurrency to hide behind, and the sign flips.

Factor	Our benchmark — it lost	Frontier-scale serving — it wins
Interconnect	PCIe / host bridge — no P2P, ~2.4 s per KV transfer	NVLink/NVSwitch in-node + InfiniBand/RoCE RDMA with GPUDirect between nodes
Concurrency	up to 32 — transfer is on the critical path	thousands — transfer overlaps compute and disappears under load
Fleets	2 GPUs, fixed roles	independent prefill and decode pools, autoscaled; P:D ratio tuned to traffic
Routing	round-robin	KV-cache-aware — routes to the node already holding the prompt's prefix
KV path	point-to-point, host-staged	pooled distributed KV store (DRAM/SSD tier) with prefix reuse

Disaggregation is an interconnect bet. The one thing that flips our result is a fast fabric — everything else is amplification.

What they optimize

The interconnect, first. KV moves GPU→GPU directly at hundreds of GB/s, so moving it is cheaper than the specialization is worth — the exact condition our host-staged link failed.
Independent scaling & ratio tuning. Long prompts → add prefill nodes; long generations → add decode nodes. You right-size each phase instead of buying GPUs that idle in one of them.
Prefix caching + KV-aware routing. Most production prompts share prefixes (system prompts, RAG context, chat history). A router sends each request to the node that already has that KV — often skipping prefill entirely.
Chunked prefill + continuous batching. Prefill is sliced to interleave with decode so neither phase stalls the pipeline, and the transfer is streamed layer-by-layer while decode starts.
Hardware specialization. Compute-dense GPUs for prefill, bandwidth-dense GPUs for decode — each phase on the silicon it actually needs.

What's public, and what isn't

Documented in the open: DeepSeek's inference system and Moonshot AI's Mooncake (a KV-cache-centric disaggregated architecture) both describe production PD-disaggregation; the technique is now standard in vLLM, SGLang, NVIDIA Dynamo, and llm-d. Closed: OpenAI's and Anthropic's serving stacks are not publicly disclosed — so we can't confirm their internals. Given the economics and the shared open-source lineage, it is a reasonable inference that frontier labs run something in this family, but that is an inference, not a claim about their systems.

METHOD

How to reproduce this

Everything is one command against the open-source harness. Results, run manifests (git SHA, vLLM version, GPU/CPU info), and every plot are in the repo.

git clone https://github.com/anshubansal2000/Heterogeneous-Prefill-Decode-Disaggregation
bash scripts/setup_runpod.sh gpt-oss-20b        # install vLLM + weights
python run.py --config A --model gpt-oss-20b    # aggregated baseline
python run.py --config C --model gpt-oss-20b    # NIXL disaggregation (2 GPUs)
python scripts/cpu_prefill_bench.py --model Qwen/Qwen3-8B   # CPU prefill

Workload matrix: input/output ∈ {1K, 8K}², concurrency 1–32, output length pinned exactly, prefix caching off to isolate raw compute. Total cloud cost for every run in this report: ~$9.60 on RunPod (H100 + 2× L40S + EPYC).

Open threads: the cross-vendor path (NVIDIA→AMD, Config D) needs AMD hardware we couldn't source; a precise CPU-prefill ceiling needs the vLLM CPU backend build. Both are natural next steps.