• A larger context window can make an AI application more useful. A customer service copilot can retain a full case history. A coding assistant can work across more files. A retrieval-augmented application can bring more documents into a response.

• The same feature changes the infrastructure economics. Every token kept active creates state that the serving system must retain while generating the next token. Longer conversations can therefore use more memory per request, reduce concurrency and raise the cost per response, even when the model and GPU type do not change.

• Training is generally a bounded job; inference recurs whenever someone uses the application. As production adoption grows, budget planning must account for continuous serving as well as model development. Memory efficiency becomes a recurring unit-economics issue.

• This can be easy to miss in a pilot. A model may fit comfortably with short prompts and a handful of users, then become memory-bound as conversation history, retrieved passages and concurrent requests grow.

• Registered users and provisioned seats are poor proxies for this demand. A thousand users asking short questions create a different memory profile from a hundred users analysing long documents or running multi-step agents. What matters is live traffic: active sequences, tokens held in context, branches and state-retention time.

For decoder-only transformer LLMs, KV caching is a standard part of production inference. At each transformer layer, attention creates three types of numerical representation:

The cache retains keys and values for tokens already processed. When the model generates the next token, it reuses that state instead of recomputing the full history. This reduces repeated work as prompts and outputs become longer. [1]

The cache is not a plain-text copy of the conversation. It contains numerical representations associated with the active sequence. Those representations still occupy memory and remain available while the request is active.

Keeping the cache in GPU high-bandwidth memory gives the model fast access, but that memory is also needed for model weights, temporary working data and other requests.

This article establishes the memory baseline. Article 6, “Offloading the cache”, builds on it to examine when moving state to another tier earns back more capacity than it costs in delay.

For a model with full attention and equal-length, unshared sequences, a useful planning approximation is:

The factor of two accounts for both keys and values. Four inputs drive most of the practical variation:

• Model architecture.  More layers, KV heads or wider head dimensions increase the state stored per token. Grouped-query and multi-query attention can reduce this component by using fewer KV heads.

• Active tokens per sequence.  System prompts, retrieved documents, chat history, tool results and generated output add to sequence length while they remain in context.

• Active sequences.  Each request needs state for its own context, but compatible requests may share identical cached prefixes where the serving engine supports this. Parallel samples and agent branches add separate state as their histories diverge.

• Storage precision.  BF16 or FP16 values use two bytes each. Lower-precision cache formats can reduce memory demand, but quality and performance trade-offs must be tested on the target workload.

For unequal sequence lengths, replace “tokens per sequence × active sequences” with the sum of tokens across those sequences. If you already have total resident tokens, do not multiply by concurrency again. Shared physical blocks should be counted once.

Consider an illustrative 70-billion-parameter model with full attention, 80 layers, eight KV heads, a head dimension of 128 and a 128,000-token context. With a BF16 cache, one full-context sequence requires approximately:

•  Four such sequences require about 156 GiB of KV-cache memory across the model replica. For comparison, 70 billion parameters in BF16 occupy about 130 GiB before runtime overhead. In this scenario, the cache has already exceeded the raw model-weight footprint.

This is an illustration, not a universal sizing result. Actual memory use depends on architecture, cache sharing and precision, serving framework, parallelism and context utilisation. Sliding-window and hybrid models need layer-specific estimates.

The direction is what matters: weights are broadly fixed per replica; cache memory rises and falls with live traffic.

Start with the model replica’s usable memory after weights and runtime overhead. Then replay representative traffic, including long conversations and bursts, to find how many requests it can serve within the latency target. The advertised maximum context window describes a model capability; it does not establish affordable concurrency.

Measure context length and concurrency together. Two separate P95 figures do not describe their joint worst case. Record the distribution of resident tokens over time, admission delays and memory headroom, then test the load expected during a peak or replica failure.

If better cache management frees capacity, the first benefit may be more work on the existing fleet. It becomes a cash saving only when the organisation can reduce billed hours, release a commitment or avoid a planned purchase. Article 8 develops the ownership-versus-service comparison on a common basis.

A useful infrastructure partner should help turn the memory estimate into a deployment that can be tested and expanded. For this problem, RE:AI’s relevant capabilities are its GPU choices and scheduling services.

RE:AI lists bare-metal H100 and GB200 services and GPU Scheduling-as-a-Service. [2] This gives enterprises flexibility to match accelerator capacity and scheduling to actual workload demand rather than sizing infrastructure around model parameters alone. The serving engine still determines how cache is allocated, shared and reclaimed; the service catalogue alone does not establish a cache-performance result.

Use the enterprise’s model and traffic trace to validate concurrency, latency and operational headroom before committing to a larger deployment.

Long-context inference ultimately runs in physical infrastructure. Adding dense GPU capacity requires power, cooling, space and network paths that can sustain the workload.

Nxera’s 9 February 2026 opening announcement reports 58 MW of AI-ready capacity at DC Tuas, bringing its total Singapore data-centre capacity to 120 MW. The announcement describes direct-to-chip liquid cooling, integration with a cable landing station and a power usage effectiveness (PUE) of 1.25. [3]

For sensitive workloads, capacity planning must also account for where models, data and cached state are processed and retained, alongside who has access to them.

PUE compares total facility energy with the energy used by IT equipment. The published facility figure is not a per-workload efficiency result or a guarantee of available RE:AI capacity. Confirm the power allocation, cooling provision and expansion headroom for the proposed deployment.

The same announcement describes expansion in Batam and Johor. [3] That regional reach matters when applications, data and users span markets, but each deployment still needs an agreed location and measured network path. Physical capacity and connectivity make scaling possible; the workload test establishes how much useful inference that capacity delivers.

The better question is: with our observed context lengths and traffic peaks, how many active sequences can each replica serve at the required latency, resilience and cost?

A capacity plan grounded in live memory demand gives the enterprise a firmer basis for scaling than a model size or GPU count alone.

Bring that traffic profile to the RE:AI team at https://www.re-ai.ai/

Sources

[1] Hugging Face — How caching works.

[2] RE:AI — AI Cloud Services.

[3] Nxera — DC Tuas opening announcement, 9 February 2026.