LLM Inference Engineering: Why Model Weights Are Only Part of Your GPU Memory Budget

A language model fits on your GPU.

Then you increase the context length, admit more concurrent requests, or let users generate longer responses—and the deployment runs out of memory.

The weight estimate was not necessarily wrong. It was incomplete.

Model weights describe one part of the memory budget. Serving a workload also means accounting for the state associated with active sequences, particularly the attention key-value cache.

That distinction is one of the central topics I explore in my book, AI Systems Engineering: From Prototype to Production. This article introduces the memory-sizing problem, explains where PagedAttention fits, and shows how to approach an estimate without confusing it with a production guarantee.

Disclosure: I’m the author of the book linked near the end.

What Is LLM Inference Engineering?

In this article, I use “LLM inference engineering” to describe the work of turning model execution into a serving system with explicit memory, latency, and throughput constraints.

The useful questions go beyond:

Can I load the model?

They include:

• How much sequence state can the deployment retain?

• What happens when requests have different context lengths?

• Which work can be reused?

• How does the scheduler admit new requests?

• Which latency targets matter for this application?

• What assumptions make a capacity estimate valid?

A model-loading test answers an important question. It does not answer all of these.

Why the KV Cache Matters

During autoregressive inference, a transformer can retain previously computed attention keys and values rather than recomputing them for every subsequent token. This retained state is commonly called the KV cache.

Its size depends on sequence length and model architecture. In a serving workload, multiple sequences can require retained KV state at the same time. The original vLLM explanation identifies this dynamic memory requirement as a central serving challenge. vllm

That is why two workloads using identical model weights can have very different memory requirements.

Compare:

Workload A
A few short requests
Short generated responses

Workload B
Many active requests
Long prompts
Long generated responses

The model weights may be unchanged. The amount of sequence state is not.

A First-Order KV-Cache Memory Formula

For a conventional full-attention decoder with uniform layer dimensions and a KV cache stored at a fixed precision, a useful starting estimate is:

[ M_{\mathrm{KV}} \approx 2L H_{\mathrm{KV}} D_{\mathrm{head}} b \sum_{i=1}^{B} T_i ]

Where:

• (L): number of transformer layers.

• (H_{\mathrm{KV}}): number of key-value heads per layer.

• (D_{\mathrm{head}}): dimension of each head.

• (b): bytes per cached element.

• (B): number of retained sequences in the estimate.

• (T_i): number of cached tokens in sequence (i).

• The factor of two accounts for both keys and values.

This is the same basic memory-accounting approach used in the numerical examples in my book sample. ppl-ai-file-upload.s3.amazonaws

For equal-length sequences, the sum becomes (BT):

[ M_{\mathrm{KV}} \approx 2L H_{\mathrm{KV}} D_{\mathrm{head}} bBT ]

The important point is the relationship: under these assumptions, KV storage grows linearly with the number of retained token positions—not exponentially.

A Worked Example

Take an illustrative configuration:

Layers:                     32
KV heads per layer:          8
Head dimension:            128
Bytes per cached element:    2
Cached tokens per sequence: 4,096
Retained sequences:         16

The cache requirement per token is:

[ 2 \times 32 \times 8 \times 128 \times 2

131{,}072 \text{ bytes} ]

That is 128 KiB per token.

For one sequence containing 4,096 cached tokens:

[ 131{,}072 \times 4{,}096

536{,}870{,}912 \text{ bytes}

512 \text{ MiB} ]

For 16 such sequences:

[ 16 \times 512 \text{ MiB}

8 \text{ GiB} ]

That is an estimate for the KV tensors under the stated assumptions.

It is not the total GPU memory requirement.

It does not include model weights, temporary execution memory, or the serving runtime’s other allocations. It also assumes independent cache storage without shared prefixes or compression.

If generation extends each sequence, update the retained-token count accordingly.

“Concurrent Users” Is Not a Complete Capacity Specification

In the calculation above, 16 means 16 retained sequences of a specified length.

It does not automatically mean the application supports only 16 registered users—or that it can meet a latency target for 16 simultaneous generations.

For a meaningful sizing exercise, define what concurrency means:

• Open application sessions?

• Requests waiting in a queue?

• Requests currently executing?

• Sequences whose cache remains resident?

• Requests completed within a latency target?

Then define the workload:

Input-length distribution
Output-length distribution
Arrival pattern
Latency objectives
Cache-retention behavior

I would treat a memory calculation as a screening step: it can expose an impossible or incomplete plan. A workload benchmark is still needed to establish useful serving capacity.

What PagedAttention Changes

PagedAttention addresses how KV memory is allocated and accessed.

Instead of requiring each request’s KV cache to occupy one large contiguous allocation, it divides the cache into blocks that can be placed in non-contiguous physical memory. The serving system maps logical token positions to the relevant blocks. vllm

This provides a more flexible way to manage variable-length sequences and reduce allocation waste.

But it is important to separate two questions:

1. How efficiently is memory allocated?

2. How much KV information does the workload require?

PagedAttention improves the first. It does not, by itself, change the dimensions or precision of every cached key and value.

That is why a deployment can benefit from paged allocation and still face a large KV-memory requirement.

Allocation and Reuse Are Different Levers

There is another distinction worth preserving: allocating KV state efficiently is not the same as reusing previously computed state.

vLLM’s automatic prefix-caching design allows matching token-prefix blocks to reference shared physical blocks. Its documentation describes a block-hashing approach for finding and reusing compatible cached prefixes. docs.vllm

For an engineering investigation, I would therefore keep these questions separate:

• Is memory being wasted by allocation?

• Are requests repeating reusable prefixes?

• Is the selected KV representation too expensive for the workload?

• Is the admitted workload larger than the memory budget permits?

These questions may lead to different interventions.

They should not be collapsed into “turn on a cache optimization.”

Build a Memory Budget Before Buying More Hardware

Here is the worksheet I would start with:

The calculated KV tensor size is only one row.

A useful report should make its assumptions visible. “This model needs 24 GB” is much less informative than a statement specifying the model representation, context lengths, concurrency definition, and measured runtime behavior.

Why I Wrote About These Internals

My book connects inference mechanics with the surrounding serving architecture.

The sample covers the prefill and decode lifecycle, attention-head layouts, model-weight and KV-memory calculations, and implementation trade-offs. Its contents also introduce PagedAttention, quantization, continuous batching, multi-GPU inference, and CUDA and Triton kernel development. ppl-ai-file-upload.s3.amazonaws

The aim is to help readers ask more precise questions about an inference system:

• What state is being stored?

• What work is repeated?

• What does the scheduler control?

• Which estimate is analytical?

• Which claim requires measurement?

The sample also includes an educational asynchronous serving simulator. It should be read as a way to inspect scheduling and metric collection—not as a GPU benchmark or evidence of production performance. ppl-ai-file-upload.s3.amazonaws

Explore the Book and Free Sample

If you want a longer treatment of inference memory, serving mechanisms, and implementation examples, you can explore my book:

AI Systems Engineering: From Prototype to Production — book details and free sample

The Leanpub page provides free sample chapters, so you can review the level and approach before purchasing. leanpub

Start with the sample’s memory-budget section. Check the assumptions, units, and scope against the workload you are trying to serve.

A useful sizing discussion should leave you with more than a GPU recommendation.

It should leave you with a model of the workload—and a clear account of what still needs to be measured.