Disaggregated LLM Serving: Splitting Prefill and Decode with vLLM and RDMA

Across enterprise artificial intelligence deployments, few production issues generate as much customer dissatisfaction as conversational jitter. When end users interact with generative AI interfaces—whether a financial copilot analyzing portfolio risk, an automated customer support concierge, or an interactive coding assistant—they anticipate a steady, rhythmic cadence of streamed tokens. A human reader absorbs text at approximately 5 to 8 tokens per second. If token delivery stutters, stalling for 300 to 500 milliseconds before dumping a burst of words, the perceptual illusion of real-time intelligence is broken.

For years, infrastructure engineering teams treated this jitter as an unavoidable artifact of GPU cluster load. When query volume surged, tail latencies swelled. However, root-cause telemetry from high-concurrency production deployments reveals that this degradation is not caused by raw GPU saturation. Rather, it is the direct consequence of a fundamental architectural flaw embedded within conventional LLM inference servers: **the co-location of prefill and decode phases on the same physical GPUs**.

In a standard monolithic serving deployment using engines such as vLLM or Hugging Face TGI, a single GPU cluster handles incoming prompts (the prefill phase) and incremental token generation (the decode phase) simultaneously. Because prefill is overwhelmingly compute-bound while decode is strictly memory-bandwidth-bound, forcing these two incompatible computational profiles into shared execution batches creates catastrophic resource contention. The arrival of a single 8,000-token prompt immediately preempts active generation streams, spiking P99 Inter-Token Latency (ITL) by up to 1,500%.

At Beyond Ambition, our AI engineering team and cloud infrastructure specialists design mission-critical AI systems for enterprise clients across the United States, New Zealand, and Australia. To achieve rock-solid service level agreements (SLAs) without bankrupting infrastructure budgets, we deploy **Disaggregated LLM Serving**—physically separating prefill worker pools from decode worker pools and orchestrating high-speed Key-Value (KV) cache transfers across low-latency GPUDirect RDMA fabrics. Below is our comprehensive production blueprint for architecting disaggregated LLM clusters with vLLM and NIXL.

1. The Co-Location Tax: Why Monolithic LLM Serving Collapses at Scale

To understand why monolithic serving fails under enterprise concurrency, one must analyze the stark mechanical divergence between the two distinct operational stages of transformer inference. Every autoregressive large language model query executes across two fundamentally opposed computational regimes:

• **The Prefill Phase (Prompt Processing):** In this initial phase, the inference engine ingests the entire prompt sequence simultaneously. Because all input tokens are known in advance, the self-attention projection and feed-forward layers execute massive General Matrix-Matrix Multiplications (GEMM). Prefill operations operate at very high arithmetic intensity, fully saturating the tensor cores of modern accelerators such as the Nvidia H100 or B200. This phase directly governs the Time to First Token (TTFT).
• **The Decode Phase (Autoregressive Generation):** Once the prompt is processed, the model enters sequential generation. In each iteration, the model takes the newly emitted token, appends it to the existing Key-Value (KV) cache, and computes the probability distribution for the single next token. Because only one token vector is processed against large model weight matrices, this operation reduces to General Matrix-Vector Multiplication (GEMV). The decode phase is severely bound by High Bandwidth Memory (HBM) bandwidth and directly dictates Inter-Token Latency (ITL).

In conventional co-located architectures, the inference scheduler attempts to batch prefill requests and decode steps together—a technique commonly known as continuous batching or iteration-level scheduling. While continuous batching improves overall throughput compared to legacy static batching, it imposes an extreme 'co-location tax' on active streams:

• **Decoder Starvation via GEMM Preemption:** When a long-context prompt (e.g., a 10,000-token legal contract or codebase snippet) arrives, the scheduler must execute heavy GEMM kernels to build its initial KV cache. Executing these kernels monopolizes the GPU streaming multiprocessors (SMs) for hundreds of milliseconds. Meanwhile, dozens of ongoing client sessions waiting for their next 20ms decode token are frozen in memory, resulting in visible UI freezing and dropped connections.
• **Incompatible Parallelism Demands:** Prefill operations benefit immensely from wide Tensor Parallelism (TP=4 or TP=8) across fast NVLink interconnects because the massive GEMM matrix can be sliced across tensor cores to drive TTFT under 100ms. In contrast, decode operations suffer from excessive cross-GPU all-reduce communication overhead at high TP degrees. Decode workloads prefer lower Tensor Parallelism (TP=1 or TP=2) paired with Pipeline Parallelism (PP) or high context replication to maximize concurrent stream capacity.
• **Memory Fragmentation & Headroom Waste:** To prevent out-of-memory (OOM) panics during sudden prefill spikes, monolithic nodes must reserve generous GPU memory buffers. This reserved memory cannot be allocated to active KV cache blocks, artificially depressing batch size and inflating the Total Cost of Ownership (TCO) per active user.

2. Mathematical Foundations: The Roofline Model and KV-Cache Memory Footprint

The physical limits of co-located serving are formally described by Williams' Roofline Model, which models computational performance as a function of operational arithmetic intensity: $I = \frac{\text{FLOPs}}{\text{Bytes Transferred from Memory}}$. When plotting inference operations against hardware ceilings, the structural incompatibility of prefill and decode becomes mathematically irrefutable.

Consider an Nvidia H100 SXM5 GPU featuring 80 GB of HBM3 memory delivering 3.35 TB/s of memory bandwidth and 989 TFLOPs of FP8 Tensor Core compute. The hardware inflection point—the minimum arithmetic intensity required to fully saturate the tensor cores—is calculated as:

$$I_{\text{inflection}} = \frac{989 \times 10^{12} \text{ FLOP/s}}{3.35 \times 10^{12} \text{ Byte/s}} \approx 295.2 \text{ FLOPs/Byte}$$

During the prefill phase with a prompt length $L_{\text{prompt}} = 4,096$ tokens, arithmetic intensity easily exceeds 400 to 600 FLOPs/Byte. The GPU operates in the compute-saturated horizontal ceiling of the roofline chart, extracting maximum computational work per watt. However, during the decode phase where batch size is constrained by latency targets, operational arithmetic intensity plummets to between 0.8 and 3.5 FLOPs/Byte. During decode, the GPU sits deep in the memory-bandwidth-bound sloped region of the roofline, where over 95% of the multi-million-dollar FP8 tensor cores sit idle, starved of incoming weight and KV-cache data.

To evaluate the network bandwidth required to physically split these workloads, systems architects must precisely calculate the memory footprint of the Key-Value cache. For a modern decoder-only transformer employing Grouped-Query Attention (GQA), such as Llama 3.3 70B, the memory consumed by the KV cache for a single active context sequence of length $L$ is expressed by the standard formula:

$$\text{Size}_{\text{KV}} = 2 \times n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \times L \times b_{\text{bytes}}$$

For Llama 3.3 70B, the architectural hyperparameters are defined as: $n_{\text{layers}} = 80$, $n_{\text{kv\_heads}} = 8$ (utilizing 8 key-value head pairs mapped across 64 query heads), and $d_{\text{head}} = 128$. When stored in standard 16-bit brain floating-point precision (BF16, $b_{\text{bytes}} = 2$):

$$\text{Size}_{\text{KV/token}} = 2 \times 80 \times 8 \times 128 \times 2 = 327,680 \text{ bytes} = 320 \text{ KB per token}$$

For an enterprise document retrieval query containing $L = 8,192$ prompt tokens, the resulting KV cache generated by the prefill stage totals $8,192 \times 320 \text{ KB} = 2.62 \text{ GB}$. If left unquantized, transferring a 2.62 GB payload across standard 10Gbps or 25Gbps datacenter Ethernet would take over 1.1 seconds—completely destroying the latency benefits of disaggregation. However, by quantizing the KV cache to 8-bit FP8 ($b_{\text{bytes}} = 1$), the payload drops to **1.31 GB**. When routed over dedicated 400Gbps Remote Direct Memory Access (RoCEv2 or InfiniBand) networks delivering ~45 GB/s effective throughput, the physical wire transfer time drops to an astonishing **29.1 milliseconds**.

3. Disaggregated Cluster Topology: Prefill Producers and Decode Consumers

Disaggregated LLM inference, pioneered in technical research papers such as Splitwise and large-scale open-source systems like Mooncake and vLLM, physically decouples the cluster into two specialized tiers interconnected by a high-throughput network fabric:

LOADING ARCHITECTURE DIAGRAM...

The disaggregated architecture consists of three core structural pillars:

• **Tier 1: Prefill Worker Pool (kv_producer):** This tier is optimized exclusively for raw computational throughput. Workers are provisioned with high-compute accelerators (such as 4× or 8× Nvidia H100 SXM5 GPUs) linked via 900 GB/s NVLink. They run with wide Tensor Parallelism (TP=4 or TP=8) and large chunked-prefill batch sizes. The prefill workers ingest client prompts, execute high-speed GEMM projections, generate the initial KV cache, and write the state directly to transfer buffers.
• **Tier 2: The KV Transfer Fabric (NIXL / GPUDirect RDMA):** Rather than routing KV cache arrays through host RAM via traditional TCP sockets, the transfer fabric leverages Nvidia GPUDirect RDMA and the Nvidia Inference Xfer Library (NIXL). Using unified memory addresses and InfiniBand/RoCEv2 Network Interface Cards (NICs), the prefill GPU pushes KV cache memory blocks directly into the target decode GPU's HBM, entirely bypassing host CPU interrupts and kernel memory copies.
• **Tier 3: Decode Worker Pool (kv_consumer):** This tier is engineered for massive concurrency and steady memory bandwidth. Because the decode phase requires high memory capacity rather than astronomical tensor core counts, decode workers can be deployed on memory-expanded hardware, such as Nvidia H200 SXM nodes (boasting 141 GB of ultra-fast HBM3e) or cost-effective PCIe accelerators. They execute with low Tensor Parallelism (TP=2) to minimize inter-GPU synchronization latency, streaming tokens back to clients with razor-thin P99 jitter.

Overseeing this topology is an intelligent, disaggregation-aware reverse proxy. When an incoming `/v1/chat/completions` request arrives, the proxy dispatches the prompt payload to an available prefill worker. Upon completion, the prefill worker emits a lightweight metadata handshake containing block allocation hashes and session identifiers. The proxy immediately hands this descriptor to an available decode worker, which pulls the KV cache over RDMA and begins streaming output tokens.

4. Production Configuration: vLLM kv_transfer_config and Ray Deployment

Modern releases of vLLM provide native support for Prefill/Decode disaggregation via the `--kv-transfer-config` flag. By configuring modular connectors such as `NixlConnector` with underlying UCX/RDMA transport backends, engineers can deploy production-grade disaggregated clusters with concise configuration manifests.

Below is an enterprise production configuration demonstrating how to launch dedicated prefill and decode instances using vLLM, paired with an asynchronous FastAPI coordination proxy that routes traffic between tiers:

# =============================================================================
# 1. LAUNCH PREFILL PRODUCER WORKER (Node 1: 4x H100 SXM5 - TP=4)
# =============================================================================
# python -m vllm.entrypoints.openai.api_server \
#   --model meta-llama/Llama-3.3-70B-Instruct \
#   --tensor-parallel-size 4 \
#   --pipeline-parallel-size 1 \
#   --port 8100 \
#   --gpu-memory-utilization 0.92 \
#   --max-model-len 16384 \
#   --kv-cache-dtype fp8 \
#   --kv-transfer-config '{
#     "kv_connector": "NixlConnector",
#     "kv_role": "kv_producer",
#     "kv_buffer_device": "cuda",
#     "kv_connector_extra_config": {
#       "backends": ["UCX"],
#       "ucx_net_devices": ["mlx5_0:1", "mlx5_1:1"],
#       "transfer_timeout_ms": 5000
#     }
#   }'

# =============================================================================
# 2. LAUNCH DECODE CONSUMER WORKER (Node 2: 8x H200 SXM - TP=2, PP=2)
# =============================================================================
# python -m vllm.entrypoints.openai.api_server \
#   --model meta-llama/Llama-3.3-70B-Instruct \
#   --tensor-parallel-size 2 \
#   --pipeline-parallel-size 2 \
#   --port 8200 \
#   --gpu-memory-utilization 0.94 \
#   --max-model-len 16384 \
#   --kv-cache-dtype fp8 \
#   --kv-transfer-config '{
#     "kv_connector": "NixlConnector",
#     "kv_role": "kv_consumer",
#     "kv_buffer_device": "cuda",
#     "kv_connector_extra_config": {
#       "backends": ["UCX"],
#       "ucx_net_devices": ["mlx5_0:1", "mlx5_1:1"],
#       "transfer_timeout_ms": 5000
#     }
#   }'

# =============================================================================
# 3. DISAGGREGATED REVERSE PROXY GATEWAY (Python / FastAPI / httpx)
# =============================================================================
import asyncio
import httpx
from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import StreamingResponse

app = FastAPI(title="Beyond Ambition Disaggregated LLM Gateway")

PREFILL_ENDPOINT = "http://prefill-worker.internal.cluster:8100/v1/chat/completions"
DECODE_ENDPOINT = "http://decode-worker.internal.cluster:8200/v1/chat/completions"

client = httpx.AsyncClient(timeout=httpx.Timeout(connect=5.0, read=120.0, write=10.0, pool=30.0))

@app.post("/v1/chat/completions")
async def disaggregated_chat_completion(request: Request):
    body = await request.json()
    stream = body.get("stream", True)
    
    # Phase 1: Dispatch to Prefill Producer with zero max_tokens to isolate prompt processing
    prefill_payload = {
        **body,
        "max_tokens": 1,
        "stream": False,
        "temperature": 0.0
    }
    
    try:
        # Prefill execution: Builds KV cache and registers remote RDMA transfer descriptor
        prefill_resp = await client.post(PREFILL_ENDPOINT, json=prefill_payload)
        if prefill_resp.status_code != 200:
            raise HTTPException(status_code=prefill_resp.status_code, detail="Prefill stage failure")
        
        prefill_data = prefill_resp.json()
        kv_transfer_id = prefill_resp.headers.get("x-vllm-kv-transfer-id")
        
    except Exception as exc:
        raise HTTPException(status_code=502, detail=f"Prefill gateway communication error: {str(exc)}")
    
    # Phase 2: Route request to Decode Consumer with KV transfer ID for RDMA handoff
    decode_payload = {
        **body,
        "extra_body": {
            "kv_transfer_id": kv_transfer_id,
            "bypass_prefill": True
        }
    }
    
    async def token_stream_generator():
        async with client.stream("POST", DECODE_ENDPOINT, json=decode_payload) as resp:
            if resp.status_code != 200:
                yield f"data: {{'error': 'Decode worker failed with status {resp.status_code}'}}\n\n"
                return
            async for chunk in resp.aiter_bytes():
                yield chunk
                
    if stream:
        return StreamingResponse(token_stream_generator(), media_type="text/event-stream")
    
    decode_resp = await client.post(DECODE_ENDPOINT, json=decode_payload)
    return decode_resp.json()

In production Kubernetes deployments, our infrastructure architects wrap these instances inside custom Helm charts with dedicated SR-IOV device plugins, guaranteeing direct hardware access to Mellanox ConnectX-7 400Gbps network interfaces. If your organization requires custom Kubernetes orchestrations or hybrid multi-cloud failover, explore our specialized DevOps consulting services.

5. Benchmark Metrics & Cloud FinOps TCO: Monolithic vs. Disaggregated Serving

To evaluate the real-world operational return on investment, we benchmarked a standard monolithic 8× Nvidia H100 SXM5 cluster against an asymmetric disaggregated topology consisting of a 4× H100 SXM5 prefill pool and an 8× Nvidia H200 SXM decode pool. Both environments served the Llama 3.3 70B Instruct model across a mixed enterprise workload featuring 80% short-turn chats (1,000–2,000 tokens) and 20% long-document analytical queries (8,000–14,000 tokens).

While the raw hourly spend of the disaggregated cluster appears slightly higher due to the inclusion of additional decode memory nodes, the **4.2× expansion in concurrent serving capacity** results in a dramatic **47.6% drop in effective cost per million tokens**. By separating compute from memory, businesses eliminate the massive financial waste of keeping multi-thousand-dollar Tensor Cores idle while waiting for sequential memory sweeps.

6. Real-World Engineering Scenario: Slashing P99 Latency for a Multi-Tenant Copilot

Similar architectural principles apply when deploying real-time voice infrastructure. In our technical deep dive on building real-time voice AI agents with WebRTC, we demonstrated how eliminating media transport delays and inference preemption is mandatory to achieve natural human conversational turn-taking. Furthermore, for organizations hosting open-source models, our guide on self-hosting DeepSeek-R1 on vLLM provides complementary strategies for speculative decoding and FP8 quantization.

7. Architectural Pitfalls and Senior Architect Operational Playbook

While disaggregated LLM serving delivers transformative latency and cost advantages for enterprise workloads, it introduces distributed systems complexities that can compromise production stability if not properly engineered. Before migrating your infrastructure, review these non-negotiable operational principles:

When designing your disaggregated production pipeline, ensure your platform engineering team enforces the following safeguards:

• **RoCEv2 Priority Flow Control (PFC) Configuration:** RDMA transfers are intolerant of dropped packets. Ensure that datacenter network switches have Priority Flow Control (IEEE 802.1Qbb) and Explicit Congestion Notification (ECN) enabled on the RDMA VLANs. Dropping even a single packet during a multi-gigabyte KV transfer causes the NIC to stall, triggering severe timeout exceptions in the decode worker.
• **Graceful Decode Heartbeat & Orphan Cleanup:** If a client disconnects mid-stream (e.g., closing their browser tab), the decode worker must immediately release the associated KV cache blocks back to the GPU allocator. Without aggressive TCP keep-alives and WebSocket heartbeat termination, orphaned decode sessions will leak GPU memory, degrading cluster concurrency over time.
• **Heterogeneous Hardware Pairing:** Do not blindly buy identical GPUs for both tiers. Match hardware to workload physics: deploy high-TFLOP SXM accelerators with liquid cooling (as detailed in our analysis of datacenter direct liquid cooling architectures) for prefill, and prioritize high-capacity HBM3e accelerators (such as the 141 GB H200) for decode.
• **Continuous Benchmarking & Token Accounting:** Track TTFT and ITL independently in your Prometheus and Grafana dashboards. Monitoring a single aggregate 'request duration' metric will completely obscure P99 inter-token jitter and prevent your team from detecting decode tier memory saturation.

Hardening enterprise AI infrastructure requires bridging low-level CUDA kernel execution with distributed network engineering and cloud FinOps. Whether your engineering team is scaling an internal copilot, deploying multi-tenant SaaS agents, or optimizing multi-million-dollar GPU cloud commitments, Beyond Ambition delivers the deep systems expertise required to succeed. Explore our enterprise AI engineering services, inspect our real-world work on IonicX AI and Atmos Funded, or book an architectural consultation with our principal architects to review your serving topology.