You can work out how much VRAM an LLM needs in a few minutes if you know three numbers: the parameter count, the precision (i.e. the quantisation), and the context length multiplied by the number of concurrent users. Below is a simple formula, worked examples for models from 8B to over 2T parameters, and a table that maps straight to hardware.
“How much VRAM does an LLM need?” is the question we hear most often before sizing a GPU server. The answer “the model is 70B, so you need 140 GB” is sometimes right, but it regularly leads to buying hardware that is too big or, more often, too small. The reason is the KV cache, which many calculators simply ignore. Model examples are current as of November 2026; the open-weight market changes every month, but the method stays the same.
The formula: how much VRAM does an LLM need
The memory needed for inference (running the model) is approximately:
VRAM ≈ weights + KV cache + 10–20% overhead
where weights = number of parameters × bytes per parameter. Bytes per parameter depend on precision:
- FP16 / BF16 – 2 bytes per parameter (full precision, the format most models are published in),
- FP8 / INT8 – 1 byte,
- 4-bit (e.g. GGUF Q4, AWQ, GPTQ) – about 0.5 bytes. In practice slightly more, because quantisation scaling factors add a little.
The overhead covers the CUDA context, inference engine buffers, activations and memory fragmentation. Use 10% for a well-tuned engine and 20% if you want a comfortable margin.
An important point for MoE (Mixture of Experts) models: all parameters must fit in memory, even though only a subset (the “active” parameters) is computed for each token. A model labelled 1T-A52B needs as much memory as a trillion-parameter model. The active count affects speed, not VRAM.
Weights alone: examples for 8B, 27–32B, 70B, 120B and 400B+
8B model
For example Llama-PLLuM-8B. In BF16: 8 billion × 2 bytes = 16 GB. In FP8: 8 GB; at 4-bit: about 4 GB. The 4-bit version runs on a 12 GB card; the BF16 version needs a 24 GB GPU, and even then there is not much room left for context.
27–32B model
For example Qwen3.8-27B (27.8B parameters) or Gemma 4 31B. In BF16: 27.8 × 2 = 55.6 GB, and 62 GB for the 31B model. In FP8: 28–31 GB; at 4-bit: 14–16 GB. This class fits on a single 96 GB RTX PRO 6000 even in BF16.
70B model
For example Llama-PLLuM-70B. In BF16: 70 × 2 = 140 GB – more than a single 96 GB card, and in practice more than an H200 (141 GB) too, once you add the KV cache and overhead. In FP8: 70 GB; at 4-bit: 35 GB.
Around 120B (MoE)
For example gpt-oss-120b, with 117B parameters in total. In BF16: 117 × 2 = 234 GB; in FP8: 117 GB; at 4-bit: 58.5 GB. Only a small share of the parameters is computed per token, but memory must be reserved for all 117B. At 4-bit the model fits on one 96 GB card or on a 128 GB DGX Spark.
400B+ models
Llama 4 Maverick (about 400B): 800 GB in BF16, 400 GB in FP8, 200 GB at 4-bit. DeepSeek-V4.1-Flash (about 763B): 763 GB in FP8, about 380 GB at 4-bit. Kimi K3 (about 2.78T): 2,780 GB in FP8, 1,390 GB at 4-bit. Models like these need many GPUs, an HGX system or a cluster.
The KV cache: memory for context
While generating a response, the model stores intermediate results (the attention keys and values) for every token it has processed, so it does not have to recompute them. That is the KV cache. It grows linearly with context length and with the number of concurrent conversations. If you want a refresher on what a token actually is, see our article on tokens in large language models.
The standard formula for the KV cache per sequence (one conversation) is:
KV cache = 2 × layers × KV heads × head dimension × bytes per value × tokens
The leading 2 is there because keys (K) and values (V) are stored separately. You will find the architecture parameters in the model’s config.json (num_hidden_layers, num_key_value_heads and head_dim; if the last one is missing, the head dimension is hidden_size divided by num_attention_heads). For several concurrent conversations, multiply by their number.
Example: a 70B model on the Llama 3.1 architecture
Llama-PLLuM-70B is based on Llama 3.1 70B: 80 layers, 8 KV heads, head dimension 128. The KV cache in BF16 (2 bytes) per token is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, or about 0.33 MB. For one conversation that gives:
| Context length | KV cache – 1 conversation | KV cache – 10 conversations |
|---|---|---|
| 8,192 tokens | approx. 2.7 GB | approx. 26.8 GB |
| 32,768 tokens | approx. 10.7 GB | approx. 107 GB |
| 131,072 tokens | approx. 42.9 GB | approx. 429 GB |
For comparison, an 8B model on the same architecture (32 layers) needs about 0.13 MB per token, or about 4.3 GB for one conversation with a 32,768-token context.
The takeaway: with long contexts and many users, the KV cache can take more memory than the weights. A few things help. Some engines (vLLM, for example) can store the KV cache in FP8, which halves it. vLLM’s PagedAttention allocates memory as needed instead of reserving the full context for every request. And some architectures compress the KV cache heavily, which is why you should always check the specific model’s config.json.
End-to-end example: a 70B model for 10 people
Assume a 70B model in FP8, 10 concurrent conversations, each with up to 8,192 tokens of context:
- weights: 70 GB,
- KV cache in BF16: 10 × approx. 2.68 GB = approx. 26.8 GB,
- total: approx. 96.8 GB, and with 10–20% overhead: approx. 106–116 GB.
That does not fit on a single RTX PRO 6000 (96 GB). Even with an FP8 KV cache (approx. 13.4 GB) you land at about 92–100 GB – right at the limit. Sensible options are two RTX PRO 6000 cards (192 GB) or one B200 GPU (180 GB). A 128 GB DGX Spark will hold this setup, but with 273 GB/s of memory bandwidth, responses for 10 people will be slow – it is prototyping hardware, not production.
LLM VRAM table: model size, precision, memory and hardware
If you just want a quick answer to “how much VRAM does an LLM need?”, start here. The “weights” column excludes the KV cache and overhead. The hardware suggestions leave room for a moderate context. With many users and long documents, move one step up.
| Model (example) | Precision | Weights (approx.) | Example hardware |
|---|---|---|---|
| 8B (Llama-PLLuM-8B) | BF16 | 16 GB | 24 GB GPU (short context); 1× RTX PRO 6000 for many users |
| 8B | 4-bit | 4 GB | 12 GB GPU |
| 27–32B (Qwen3.8-27B, Gemma 4 31B) | BF16 | 56–64 GB | 1× RTX PRO 6000 (96 GB); DGX Spark (128 GB) |
| 27–32B | 4-bit | 14–16 GB | 24 GB GPU (short context) or 48 GB |
| 70B (Llama-PLLuM-70B) | BF16 | 140 GB | 2× RTX PRO 6000 (192 GB); 1× B200 (180 GB) |
| 70B | FP8 | 70 GB | 1× RTX PRO 6000 (little room for context); 2× RTX PRO 6000; DGX Spark for testing |
| 70B | 4-bit | 35 GB | 1× RTX PRO 6000 with plenty of headroom; 2× A40 (96 GB) |
| Around 120B MoE (gpt-oss-120b, 117B) | FP8 | 117 GB | 2× RTX PRO 6000; too tight for DGX Spark |
| Around 120B MoE | 4-bit | 58.5 GB | 1× RTX PRO 6000; DGX Spark |
| Around 400B (Llama 4 Maverick) | FP8 | 400 GB | 8× RTX PRO 6000 (768 GB); HGX B200 (1,440 GB) |
| Around 400B | 4-bit | 200 GB | 4× RTX PRO 6000 (384 GB); 2-unit DGX Spark cluster (256 GB, tight) |
| Around 750B (DeepSeek-V4.1-Flash, GLM-5.3) | FP8 | 750–765 GB | HGX B200 (1,440 GB); HGX B300 (approx. 2.3 TB) |
| Around 750B | 4-bit | approx. 380 GB | 8× RTX PRO 6000 (768 GB); 4-unit DGX Spark cluster (512 GB) |
| 1–1.65T (Mistral Large 4, DeepSeek-V4-Pro) | FP8 | 1,000–1,650 GB | HGX B200 (for approx. 1T); HGX B300 (up to approx. 1.65T) |
| 2.4–2.8T (Qwen3.8-2.4T-A95B, Kimi K3) | 4-bit | 1,200–1,390 GB | HGX B300; FP8 needs more than one node |
DGX Spark has 128 GB of unified memory shared by the CPU and GPU, so the operating system takes some of it. Its real advantage is capacity at low power draw, not speed. For a detailed comparison of the cards, see the best GPU for LLMs in 2026; for the unit itself, read NVIDIA DGX Spark as a Service.
Quantisation: what you save and what you risk
Going from BF16 to FP8 halves the memory for weights; going to 4-bit cuts it to a quarter. Large models usually tolerate FP8 well, and at 4-bit the quality loss is often small, although it depends on the model, the method and the task. Small models are more sensitive to quantisation. Before you commit, test the quantised version on your own data, especially for non-English tasks, where differences show up sooner than in general benchmarks.
We described a practical example of how quantisation lowers infrastructure cost in cutting AI infrastructure costs with quantisation. If you are still choosing a model, see our comparison of open-source LLMs to self-host.
Common mistakes when sizing LLM VRAM requirements
- Ignoring the KV cache. The model “fits”, but memory runs out with the first long document or the fifth user.
- Counting only active parameters in MoE. Memory is needed for all parameters.
- Confusing GB and GiB. 1 GiB is about 1.074 GB. System tools often report GiB, while datasheets use GB.
- Treating several GPUs as one pool. When a model is split across cards, each card carries its own overhead, and the split usually has to be even. Two 96 GB cards give you less than 192 GB of usable memory in practice.
- Applying inference numbers to training. Training and full fine-tuning also store gradients and optimiser states, so they need several times more memory. LoRA and QLoRA reduce this considerably.
FAQ: frequently asked questions
How much VRAM do I need for a 7B or 8B model?
In full BF16 precision, about 16 GB for the weights plus context, so a 24 GB card is the minimum. With 4-bit quantisation the weights take about 4 GB and the model runs on a 12 GB card.
How much VRAM does a 70B model need?
The weights take about 140 GB in BF16, 70 GB in FP8 and 35 GB at 4-bit. Add the KV cache – on the Llama 3.1 70B architecture about 2.7 GB per conversation with an 8,192-token context (in BF16) – and 10–20% overhead.
Do MoE models need less memory?
No. An MoE model computes only part of its parameters for each token, so it runs faster, but all of them must fit in memory. gpt-oss-120b needs memory for 117B parameters.
Can I run an LLM if I don’t have enough VRAM?
You can offload some layers to system RAM and compute them on the CPU (for example with llama.cpp), but generation then slows down dramatically. That is fine for testing, not for a team.
How does context length affect VRAM?
Linearly: twice the context means twice the KV cache for each conversation. With a 128k context and many users, the KV cache can take more memory than the model weights.
Send us the model, context length and number of users, and we will calculate the memory you need and propose a configuration – from ready-made GPU packages to GPU servers with RTX PRO 6000 or HGX B200/B300 on a custom quote. Get in touch.