GPU / AI · Updated 24 Sep 2026 · 11 min read
How much GPU VRAM do you need?
Buy VRAM from the model, quantization, context window, and concurrent users — not from a GPU marketing name. The tables below are planning ranges for inference. Confirm on your stack before you order.
What actually consumes VRAM
Three buckets matter. Weights scale with parameter count and precision. KV cache grows with context length, number of layers, and concurrent sequences. Runtime is CUDA, the serving framework (vLLM, TGI, llama.cpp, ComfyUI), fragmentation, and any adapters. A card that “fits the weights” can still OOM on a 32k prompt or a second user.
Quantization: the first lever
Lower precision shrinks weights. It also changes quality. Use this as a buyer’s map, not a quality guarantee.
| Format | Rough bytes / parameter | 8B weights only | 70B weights only | When it is used |
|---|---|---|---|---|
| FP16 / BF16 | ~2.0 | ~16 GB | ~140 GB | Highest fidelity inference; training; when you have 80 GB class cards |
| FP8 / 8-bit | ~1.0 | ~8 GB | ~70 GB | Strong production default on Hopper / Ada when the stack supports it |
| Q8 / INT8 | ~1.0 | ~8–10 GB | ~72–80 GB | Good quality on 24–48 GB cards for mid-size models |
| Q4 / 4-bit (GGUF, AWQ, GPTQ) | ~0.5–0.6 | ~4.5–6 GB | ~38–48 GB | Most common way to serve 70B on a single 80 GB GPU, or 7–13B on 16–24 GB |
Add KV cache and runtime on top. For 4k context and one user, add roughly 1–4 GB on small models and 8–20 GB on 70B-class models. At 32k context or several concurrent chats, cache can exceed the weights.
Practical VRAM table (inference, one user unless noted)
Figures are planning bands including a modest runtime, not lab measurements. “Typical GPU” is the smallest class I would shortlist, not a promise it will boot your exact container.
| Model / use | Q4 · 4k ctx | Q8/FP8 · 4k ctx | FP16 · 4k ctx | Long context (16k–32k) or 4 concurrent | Typical GPU class |
|---|---|---|---|---|---|
| Llama 3.1 / 3.2 8B, Mistral 7B — chatbot, RAG, tools | 8–10 GB | 12–16 GB | 18–24 GB | 16–24 GB | 16 GB if Q4 only; 24 GB RTX is the safer production floor |
| 13B–14B class (older Llama, some Qwen) | 10–14 GB | 18–22 GB | 28–36 GB | 24–40 GB | 24 GB Q4; 40–48 GB if you want FP16 |
| Qwen2.5 / Llama 32B–34B | 20–26 GB | 36–44 GB | 70–80 GB | 40–80 GB | Q4 on 24 GB is tight; prefer 48 GB or A100 80 GB |
| Mixtral 8×7B (MoE — active path smaller than 47B dense) | 24–32 GB | 40–50 GB | 90 GB+ | 48–80 GB | Single 48–80 GB or two 24 GB with tensor parallel |
| Llama 3.1 / 3.3 70B — stronger reasoning, coding | 40–50 GB | 75–90 GB | Needs multi-GPU | 80 GB+; 32k ctx often wants 80 GB even in Q4 | A100 80 GB or H100 80 GB; 2×48 GB if you must split |
| 70B+ / 405B / large MoE (Mixtral 8×22B, DeepSeek-class) | 80 GB+ or multi-GPU | Multi-GPU | Multi-GPU / H200 | Plan a cluster | Not a 24 GB shopping problem — get a shortlist |
| Whisper large-v3 transcription | ~8–12 GB including audio buffer | Batch jobs may need 16–24 GB | 16–24 GB RTX is usually enough | ||
| SDXL / Flux-class image gen | ~10–16 GB at 1024²; ControlNet / higher res 16–24 GB | Batch or video pipelines 24 GB+ | 24 GB consumer GPU is the common rental | ||
Match VRAM to the job, then pick the card
- Internal chatbot, 8B, a few users: 24 GB RTX (4090 / 5090 class) is usually enough. See GPU servers. A used dual-P100 box can also hold a quantized 35B — measured tok/s and power.
- Customer-facing RAG with 32k context: size cache, not just weights. 24 GB can fail where 48–80 GB is calm.
- 70B coding assistant for a team: start at 80 GB. Compare options on GPU servers for AI / LLMs.
- Image generation studio: 24 GB unless you batch or run video.
- Fine-tuning / LoRA: multiply the inference number. A model that infers in 24 GB may need 40–80 GB to train.
Relative GPU class scores on this site do not replace a VRAM check. Use server benchmarks after the memory floor is known.
Mistakes that waste money
- Renting H100 because the name is familiar, then serving 8B Q4 — a 24 GB card may be enough.
- Buying 16 GB because the weights spreadsheet said 14 GB, then adding 8k context and two users.
- Ignoring CPU RAM and NVMe: tokenizer, KV offload, and model files still need host memory and disk.
- Assuming two 24 GB GPUs equal one 48 GB GPU. Split models need a stack that supports tensor parallel and a fast interconnect.
How I would decide before checkout
- Write down model, quantization you will actually ship, max context, and peak concurrent sequences.
- Pick the band from the table and add ~15–20% headroom.
- If you land near a card’s limit, step up one VRAM class rather than “try it in production.”
- Compare live GPU offers or AI GPU configs, then confirm the order page spec.
FAQ
Usually no for production. A 70B model in Q4 already needs roughly 40 GB plus KV cache and runtime. A 16 GB card is a fit for 7B–8B models, or tightly quantized 13B-class models, not 70B serving.
No. Extra VRAM lets you load a larger model or longer context. Tokens per second still depend on memory bandwidth, tensor cores, batching, and the serving stack. An 80 GB card sitting idle on an 8B model can be slower than a well-fed 24 GB RTX.
Inference is mostly weights plus KV cache plus runtime. Training and LoRA fine-tuning add optimizer states and activations, so the same model can need two to four times more memory. Buy for the job you will run this quarter.
Plan to use no more than about 80–85% of advertised VRAM under a production-like prompt and concurrency test. The rest covers CUDA context, fragmentation, and traffic spikes.