KV Cache and Batching Optimization

When monitoring LLM serving, a peculiar phenomenon often occurs.

GPU utilization (GPU Utilization) can exceed 90%, making it seem like the hardware is about to burst,

yet the number of tokens processed per second hits rock bottom.

Resolving this contradiction is the core reason for the existence of modern LLM serving engines like vLLM and SGLang.

Why High GPU Utilization but Low Throughput? #

To put it simply, the GPU isn't busy performing significant computations (matrix multiplications using Tensor Cores); rather, it's either stuck in an I/O bottleneck (memory-bound) moving and waiting for memory data, or it cannot significantly increase batch sizes due to inefficient memory allocation.

1. Memory Bandwidth Bound: The decode phase is an extremely memory-bound task. During the time it takes to fetch weights and KV Cache from HBM to the compute cores, GPU units remain idle, just waiting. However, GPU monitoring tools like nvidia-smi interpret an active memory bus as the GPU working hard (Utilization 95%). In other words, what's being measured is memory bus utilization, not compute utilization.
2. Batch Size Constraint Due to Memory Fragmentation: In traditional serving methods, KV Cache memory for the maximum token length each request would use (e.g., 2048 tokens) was pre-allocated with a fixed-size static allocation. Even if a user actually inputs only 10 tokens and outputs 50 tokens, the remaining 1988 token spaces are wasted. This leads to VRAM OOM (Out of Memory) when trying to increase the batch size by adding more concurrent users, preventing batch expansion and ultimately lowering throughput.

How is Batch Size Determined? #

The optimal batch size is, as a general principle, to make it as large as the GPU VRAM allows.

This is because as the batch size increases, a single GPU weight can be read and reused for computations across multiple requests, making it closer to a compute-bound characteristic, which can lead to an explosive increase in throughput.

However, indiscriminately increasing the batch size can lead to OOM or increased queue waiting times, causing TTFT (Time to First Token) to exceed SLA standards. Therefore, technologies are needed to utilize hardware memory 100% without waste.

Understanding the Principles of 5 Key Optimization Techniques #

PagedAttention (vLLM Core) #

This technology introduces the virtual memory paging technique from operating systems into the KV Cache management system.

It splits the KV Cache into fixed-size small blocks (e.g., 16 tokens) and maps them to physically scattered VRAM spaces using a virtual block table.

Since there's no need to allocate memory in contiguous spaces, it reduces internal fragmentation from pre-allocation and external fragmentation from variable lengths to nearly 0%. This recovers 60-80% of wasted VRAM, allowing batch sizes to be increased by 2-4 times or more.

Continuous Batching (Iteration-level Scheduling) #

While traditional Static Batching cannot accept new requests until all requests within a batch are fully completed, Continuous Batching dynamically changes the batch on a per-token generation iteration basis.

At the end of each token generation, completed requests (EOS) are immediately removed from the batch, and the prefill phase of new requests waiting in the queue is inserted.

This completely eliminates inefficiencies where the GPU is idle due to users finishing early, or the entire system waits due to users writing long sentences.

Prefix Caching (Radix Attention) #

This technology stores and reuses the KV Cache for common prefixes of prompts in memory.

It manages the KV Cache for text shared by multiple users, such as system prompts, few-shot examples, and RAG Context, in a tree structure. When a request with an identical prefix arrives, the prefill operation is entirely skipped, and data is read directly from the cache.

This shortens TTFT to nearly 0ms in multi-turn conversations or RAG-based services, significantly reducing the volume of prefill computations.

Chunked Prefill #

This technique processes prefill requests by splitting them into several smaller chunks to eliminate the prefill bottleneck that occurs when large prompts arrive.

Processing a very long prompt all at once requires immense computation, causing all existing decoding batches to halt, leading to a spike in ITL (Inter-Token Latency). To prevent this, prefill is divided into chunks, such as 512 tokens, and processed gradually by bundling them with existing decode operations in the batch.

Effect: Even when a user with a long prompt enters, computations can be smoothly scheduled without compromising the ITL delay prevention for existing users' service experience.

Hands-on vLLM Engine Configuration and Tuning in Practice #

Let's explore production-level argument settings to maximize throughput by tuning the above options in the vLLM architecture.

# vLLM 최적화 서빙 인스턴스 실행 스크립트
python3 -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --port 8000 \
    --gpu-memory-utilization 0.95 \
    --block-size 16 \
    --max-num-seqs 256 \
    --enable-chunked-prefill True \
    --max-num-batched-tokens 2048 \
    --enable-prefix-caching

• --gpu-memory-utilization 0.95: This setting allocates 90% of the remaining VRAM, excluding model weights, entirely to the page attention block pool.
• --block-size 16: This is the number of tokens per block for PagedAttention. 16 or 32 is most efficient for CUDA alignment performance.
• --max-num-seqs 256: This is the maximum number of requests that can be simultaneously batched. To increase throughput, it should be set as high as the hardware can support.
• --enable-chunked-prefill True: Activates the coexistence of Prefill and Decode to prevent service interruptions when users with large prompts enter.

KV Cache Usage, Performance Comparison Report by Batch Settings #

The benchmark results below quantitatively analyze the differences in resource efficiency and throughput between the traditional serving architecture (Hugging Face Native + Static Batching) and modern optimized engines like vLLM (PagedAttention + Continuous Batching).

Performance Comparison Metrics: Llama-3-8B, NVIDIA A100 80GB Single Device Standard #

Data Interpretation and Insights #

• The Power of Fragmentation Elimination: A vs C
  • With traditional Native allocation, even with only 16 concurrent users, VRAM hit its virtual OOM limit. This was because memory was monopolized by phantom prompt regions that were not actually used.
  • In contrast, introducing PagedAttention (C) suppresses memory fragmentation to around 3.8%, allowing full utilization of physical memory. As the batch size could be increased to 128, throughput exploded by approximately 10 times, from 180 to 1850 tokens/sec.
• The Effect of Chunked Prefill: C vs D
  • In Group C, where only the batch size was maximized, we can observe that when a large prompt occasionally arrived, the entire batch would stutter, causing the P99 ITL to spike to 18.2ms.
  • By enabling Chunked Prefill as in Group D, the prompt computations for new requests are split and evenly interleaved with existing response computations, allowing for a maximum batch size of 256 while smoothly maintaining inter-token latency at around 14.1ms.