A 27B model at Q4_K_M needs 16.5 GB for its weights and another 2.3 GB of KV cache and recurrent state at a 32K context. On a 24 GB RTX 4090 that leaves about 6 GB spare and generates about 48 tokens per second, within one token of the 47.7 measured on that exact card. This LLM VRAM calculator does that arithmetic for any model, quantization, context length and GPU, which turns the question of how much VRAM to run an LLM into a number instead of a rule of thumb.
Every input is traceable. Weights are the real GGUF file sizes read from the Hugging Face repositories people actually download, not parameters multiplied by a guessed byte count. The KV cache comes from the config.json shipped with each model, which matters because the hybrid attention in Qwen3.8, Gemma 4 and Nemotron cuts the cache four to forty times against an older dense model. Speed comes from a model fitted to 20 single-stream runs on five NVIDIA GPUs, the same runs behind the tested GPU buyer guide and the Ollama vs vLLM vs llama.cpp benchmark, then checked against the public llama.cpp scoreboards for AMD and Apple hardware.
Calibrated September 2026 against 20 measured GPU runs; card prices checked October 2026.
The address bar updates as the inputs change, so a configuration can be copied and shared as a link. Where a green “Measured” box appears, the model, quantization and card match a run from the test bench, and that measured figure outranks the estimate above it.
How accurate the VRAM calculator is
Memory is the easy half. The weights are a file size, and the KV cache is exact arithmetic once the layer layout is known. The checks below compare the calculator against memory actually allocated on real hardware:
| Case | Calculator | Measured |
|---|---|---|
| Qwen3.8 27B KV cache per token (Ollama log, 8,192 cells) | 64 KiB | 512 MiB / 8,192 = 64 KiB |
| Qwen3.8 27B, context raised from 8K to 64K | +3,584 MiB | +3,640 MiB |
| Qwen3.8 27B, 128K context, f16 cache, RTX 4090 | 26.3 GB, spills | spilled to CPU, 16.4 tok/s |
| Qwen3.8 27B UD-Q4_K_XL, 256K context, q4_0 cache, RTX 4090 | 24.0 GB, fits | fits, 22.2 GiB (23.8 GB) in use |
| Qwen2.5 32B, 32K context, RTX 3090 | 29.2 GB, spills, ~6 tok/s | spilled to CPU, 3.6 tok/s |
| Qwen2.5 32B, 4K context, RTX 3090 | 21.6 GB, fits | fits, 37 tok/s |
Every fit or spill verdict matched the hardware, and the 256K total landed within 1 percent of what llama.cpp allocated. The miss is the speed after a spill: the one dense spill measured ran at 3.6 tok/s against an estimate near 6. The calculator assumes only the overflow runs on the CPU, and the real run evidently kept more of the model there, so treat offload speeds for dense models as optimistic. Card capacity is compared in the unit it is sold in: a “24 GB” card has 24 GiB, which holds 25.8 GB of files in the decimal gigabytes Hugging Face reports.
Speed is the harder half. Single-stream token generation on a GPU is bound by memory bandwidth, because every new token reads the whole set of active weights once. Fitting the 17 dense runs gives one line: milliseconds per token = model GB ÷ (0.82 × bandwidth in GB/s) × 1000 + 0.85. In words, a GPU delivers about 82 percent of its rated bandwidth to token generation, plus a fixed 0.85 ms of per-token overhead. Across the 17 runs the root-mean-square error is 6.5 percent and the worst case is 15.7 percent.
| Model and card | Predicted | Measured | Source |
|---|---|---|---|
| Llama 3.1 8B Q4_K_M, RTX 4090 | 147 tok/s | 148 tok/s | Ollama run |
| Llama 3.1 8B Q4_K_M, RTX 3090 | 138 tok/s | 119 tok/s | Ollama run (worst fit) |
| Qwen2.5 32B Q4_K_M, RTX 5090 | 70 tok/s | 71 tok/s | Ollama run |
| Llama 3.3 70B Q4_K_M, L40S | 16.4 tok/s | 16 tok/s | Ollama run |
| Qwen3.8 27B UD-Q4_K_XL, RTX 4090 | 45 tok/s | 47 tok/s | llama.cpp run |
| Qwen3.5 9B Q4_K_M, A10G | 80 tok/s | 82 tok/s | llama-bench |
| Llama 2 7B Q4_0, RTX 5090 | 290 tok/s | about 298 tok/s | llama.cpp CUDA scoreboard |
| Llama 2 7B Q4_0, Strix Halo | 52.5 tok/s | about 52 tok/s | llama.cpp Vulkan scoreboard |
| Llama 2 7B Q4_0, Apple M5 Max | 118.5 tok/s | 119.9 tok/s | llama.cpp Apple scoreboard |
| Llama 2 7B Q4_0, DGX Spark | 55.8 tok/s | 57.2 tok/s | llama.cpp CUDA scoreboard |
The last four rows were never used for fitting. They come from the community scoreboards for CUDA, Vulkan and Apple silicon, and the same 0.82 factor lands within 3 percent on Strix Halo, the M5 Max and the GB10. Three hardware classes needed their own factor, and the calculator applies it automatically:
- HBM data-center cards: the A100 and H100 run a 7B model at only 43 and 40 percent of rated bandwidth, so the general line overshoots them by about 67 percent. Their scoreboard runs (191 and 268 tok/s) set their factors.
- Dual-die Apple Ultra chips: the M5 Ultra reaches 179 tok/s against 1,228 GB/s, an efficiency of 0.66 rather than 0.82.
- RX 7900 XTX on Vulkan: typical runs on the RADV driver sit near 158 tok/s, an efficiency of 0.73. The best proprietary-driver runs on the scoreboard reach about 183, so those users can expect about 15 to 20 percent more.
Mixture-of-experts models read more per token than their active parameter count suggests, because attention, shared experts and the router are stored at higher precision than the routed experts. Across Gemma 4 26B-A4B, Qwen3-Coder 30B-A3B and gpt-oss-20b on an RTX 4090, the measured reads came out 1.63 times the naive file size × active ÷ total. DeepSeek V4 Flash reads 2.8 times, calibrated to a published DGX Spark run at 19.1 tok/s. The calculator does not model multi-token prediction drafting, which Ollama turns on by default for Qwen3.8, Qwen3.6 and Gemma 4 and which roughly doubled their speed on the RTX 4090. Its CPU offload estimate is deliberately rough, the least precise number on the page.
The short answer: VRAM by model size
For readers who want the table rather than the tool, these are the real GGUF weight files for a current model at each size. Add the KV cache for the context you plan to run (next section) and about 0.7 GB of runtime overhead. Q4_K_M is the column that matters for local use, because it is the default quantization in Ollama and the usual balance of size and quality.
| Model size | Example (2026) | BF16 | Q8_0 | Q4_K_M | Fits on |
|---|---|---|---|---|---|
| 9B | Qwen3.5 9B | 17.9 GB | 9.5 GB | 5.7 GB | 8 GB card |
| 12B | Gemma 4 12B | 23.8 GB | 12.7 GB | 7.1 GB | 12 GB card |
| 27B | Qwen3.8 27B | 54.7 GB | 29.1 GB | 16.5 GB | 24 GB card |
| 32B | Qwen3 32B | 65.5 GB | 34.8 GB | 19.8 GB | 24 GB card (cap context) |
| 70B | Llama 3.3 70B | 141.1 GB | 75.0 GB | 42.5 GB | 48 GB card |
| 117B (MoE) | gpt-oss-120b | 62.6 to 65.4 GB (experts ship in MXFP4, so every GGUF lands near 63 GB) | 80 or 96 GB card | ||
| 284B (MoE) | DeepSeek V4 Flash | 90.9 GB at UD-IQ2_M, 155.1 GB at UD-Q4_K_XL | 128 GB unified memory at 2-bit | ||
Read it as a floor. The 32B row fits a 24 GB card only with the context capped, and the reason is in the next table.
KV cache per token for popular 2026 models
Two models of the same size can need wildly different VRAM at the same context, and parameter count gives no hint of it. The KV cache per token depends on how many layers use full attention and how many key-value heads they keep. Qwen3 32B stores 256 KiB for every token of context. Qwen3.8 27B stores 64 KiB, because only 16 of its 64 layers are full attention and the rest are Gated DeltaNet layers with a fixed-size state. Nemotron 3.5 Lightning stores 6 KiB. The figures below are computed from the config.json of each model at the default f16 cache type:
| Model | Weights (Q4_K_M) | KV per token | KV at 32K | KV at 128K | Total at 32K |
|---|---|---|---|---|---|
| Llama 3.1 8B | 4.9 GB | 128 KiB | 4.3 GB | 17.2 GB | 10.0 GB |
| Qwen3.5 9B | 5.7 GB | 32 KiB + 0.05 GB state | 1.1 GB | 4.3 GB | 7.6 GB |
| Gemma 4 12B | 7.1 GB | 16 KiB + 0.5 GB fixed | 1.0 GB | 2.7 GB | 9.0 GB |
| gpt-oss-20b | 11.6 GB | 24 KiB | 0.8 GB | 3.2 GB | 13.2 GB |
| Qwen3.8 27B | 16.5 GB | 64 KiB + 0.16 GB state | 2.3 GB | 8.7 GB | 19.6 GB |
| Qwen3-Coder 30B-A3B | 18.6 GB | 96 KiB | 3.2 GB | 12.9 GB | 22.6 GB |
| Nemotron 3.5 Lightning 30B-A3B | 18.9 GB (Q4_0) | 6 KiB + 0.05 GB state | 0.3 GB | 0.9 GB | 19.9 GB |
| Gemma 4 31B | 18.3 GB | 80 KiB + 1.3 GB fixed | 3.9 GB | 12.0 GB | 23.1 GB |
| Qwen3 32B | 19.8 GB | 256 KiB | 8.6 GB | 34.4 GB (YaRN) | 29.1 GB |
| Qwen3.6 35B-A3B | 22.1 GB | 20 KiB + 0.07 GB state | 0.7 GB | 2.8 GB | 23.7 GB |
| Llama 3.3 70B | 42.5 GB | 320 KiB | 10.7 GB | 42.9 GB | 54.1 GB |
| gpt-oss-120b | 62.8 GB | 36 KiB | 1.2 GB | 4.9 GB | 64.8 GB |
The “fixed” entries are sliding-window layers. Gemma 4 keeps a 1,024-token window on most layers, and llama.cpp sizes that cache at the window plus one 512-token batch, so those layers cost a flat amount regardless of context and only the few global layers grow with length. Gemma 4 global layers also share keys and values; the calculator counts both, so treat its Gemma figures as a close estimate rather than exact. The “state” entries belong to the hybrid models: each Gated DeltaNet or Mamba layer keeps a fixed f32 recurrent state per sequence, 150 MiB per sequence for Qwen3.8 27B. Ollama keeps that one sequence plus four recurrent-state snapshots for MTP drafting rollback, and its llama_memory_recurrent log line reads 748.12 MiB, exactly five times the calculated figure. The calculator counts one sequence per request, so on Ollama budget about 0.6 GB more for these models.
The practical reading: Qwen3 32B and Qwen3.8 27B look interchangeable on a spec sheet, but at 32K context one needs 29.1 GB and spills off a 24 GB card while the other needs 19.6 GB and fits with room to spare. A dense 70B at 128K context needs as much memory for its cache as for its weights.
The formula: weights plus KV cache plus overhead
Three things consume VRAM during inference. The weights are the big, fixed cost. The KV cache grows with context length and with the number of concurrent requests. Overhead is a smaller tax for the CUDA or Metal context and compute buffers, about 0.7 GB at short context and roughly 1.5 GB at 256K, paid once per GPU.
Weights are parameters multiplied by bytes per parameter. Dividing the real Llama 3.3 70B GGUF files by its 70.6 billion parameters gives the effective rate for each quantization:
| Format | Bytes/param | Llama 3.3 70B file |
|---|---|---|
| BF16 | 2.00 | 141.1 GB |
| Q8_0 | 1.06 | 75.0 GB |
| Q5_K_M | 0.71 | 50.0 GB |
| Q4_K_M | 0.60 | 42.5 GB |
| Q4_0 | 0.57 | 40.1 GB |
| IQ4_XS | 0.54 | 37.9 GB |
That rate only holds for GGUF files quantized from a BF16 checkpoint. Models that ship pre-quantized break it: gpt-oss-120b has 117 billion parameters and files between 62.6 and 65.4 GB at every quantization level, because its experts are already MXFP4. Read the file size, not the parameter count.
The KV cache stores the key and value tensors for every token already in the context. Per token it costs:
kv_bytes_per_token = 2 * attention_layers * kv_heads * head_dim * bytes_per_elementThe first three come from the config.json of the model as num_hidden_layers, num_key_value_heads and head_dim, with layer_types telling you which layers are full attention on hybrid models. Where head_dim is missing, as in the Qwen2.5 configs, divide hidden_size by num_attention_heads. The fourth is the cache type. Llama 3.3 70B has 80 layers with 8 key-value heads of dimension 128, so 2 × 80 × 8 × 128 × 2 bytes is 320 KiB per token. Grouped-query attention is what keeps that small: with a separate key and value for each of its 64 query heads, the same model would need 2.5 MiB per token. The cache type sets the last term. The f16 default is 2 bytes, q8_0 is 1.06 and q4_0 is 0.56, which is why switching the cache type roughly doubles or quadruples the longest context a card can hold. Both quantized cache types need flash attention to be active. Ollama turns it on automatically on supported GPUs and OLLAMA_FLASH_ATTENTION=1 forces it; llama.cpp needs -fa alongside -ctk and -ctv.
The practical rule that falls out of all three: pick a quantization that leaves at least one gigabyte of headroom below the VRAM of the card, never one that exactly equals it.
How fast will it run? Measured tokens per second per GPU
Fitting a model is half the question, and speed does not track VRAM capacity. The numbers below were measured on single rented GPUs running Ollama at Q4_K_M with a 4K context, generation rate in tokens per second, one request at a time:
| Model | RTX 3090 (24 GB) | RTX 4090 (24 GB) | RTX 5090 (32 GB) | L40S (48 GB) |
|---|---|---|---|---|
| Llama 3.1 8B | 119 tok/s | 148 tok/s | 250 tok/s | 113 tok/s |
| Qwen2.5 32B | 37 tok/s | 43 tok/s | 71 tok/s | 33 tok/s |
| Llama 3.3 70B | does not fit | does not fit | does not fit | 16 tok/s |
The number that matters is in the 32B row. The RTX 5090 with 32 GB runs the 32B model at 71 tokens per second, more than double the L40S with its larger 48 GB. More VRAM did not make it faster. Token generation is bound by memory bandwidth, not capacity, and the 1,792 GB/s of GDDR7 on the 5090 is more than twice the 864 GB/s of GDDR6 in the L40S. VRAM capacity decides what fits; memory bandwidth decides how fast it runs once it fits. The RTX 3090 vs RTX 5090 comparison shows the same effect across more models.
The same rule sizes newer models the calculator covers. On the RTX 4090, with multi-token prediction drafting switched off, Qwen3.8 27B measured 47.7 tok/s and Gemma 4 26B-A4B measured 155.9 tok/s. Similar file sizes, three times the speed, because the mixture-of-experts Gemma reads only a fraction of its weights per token. Those runs are written up in the Qwen3.8 27B local test and the Ollama models cheat sheet.
The offload cliff: why “it fits” is not enough
When a model plus its KV cache exceed VRAM, the runtime does not fail. It keeps the overflow layers in system RAM and computes them on the CPU. That sounds graceful and is not. GPU memory moves at 500 to 1800 gigabytes per second; DDR5 system memory moves at 50 to 80, and an 8-core cloud VM running DeepSeek V4 Flash entirely on the CPU turned that into about 25 GB/s of effective throughput. Every offloaded layer drags token generation down to that rate, and the result is a cliff, not a slope.
A concrete case from testing: the 32B model on a 24 GB RTX 3090. The weights are about 20 GB and fit. But recent Ollama versions size the context window to the VRAM they detect, and a 24 GB card is handed a 32K window by default. That 32K KV cache on top of the weights pushes the total past 24 GB, the overflow spills to the CPU, and throughput collapses:
ollama run qwen2.5:32b "Explain quantization in two sentences."With the context left at its auto-sized default, generation dropped to roughly 3.6 tokens per second, a tenth of the real rate of the card. The fix is to cap the context so the cache stays inside VRAM. Ollama reads the context length from the server process, so set it there and restart:
OLLAMA_CONTEXT_LENGTH=4096 ollama serveIf Ollama runs as a systemd service, add Environment="OLLAMA_CONTEXT_LENGTH=4096" to the unit and restart it, or cap a single model from inside the session with /set parameter num_ctx 4096. Setting the variable in front of ollama run does nothing, because ollama run is only the client and the server holds the context. With the cache capped to fit, the same model on the same card held at 37 tokens per second. Same hardware, same weights, ten times the speed, decided entirely by whether the cache fit.
Capping the context is one of two fixes. The other keeps the long context and shrinks the cache instead. On the RTX 4090, Qwen3.8 27B at a 128K context spilled with the default f16 cache and dropped to 16.4 tok/s. With OLLAMA_KV_CACHE_TYPE=q4_0 and flash attention active, the identical context stayed 100 percent on the GPU. The KV cache type selector in the calculator reproduces both outcomes.
Offloading also sets a floor on system RAM. The layers that spill out of VRAM live in main memory, so size system RAM to hold the offloaded portion plus headroom for the OS. GGUF memory-maps the file, so only the offloaded layers stay resident, and fast NVMe speeds the initial load but does nothing for generation once the model is mapped.
How context length eats VRAM
At 4K context the KV cache is a rounding error against the weights. At 128K it can exceed them: Llama 3.1 8B needs 4.9 GB of weights and 17.2 GB of cache at that length. The cache is allocated up front for the maximum context you configure, and it grows linearly with that length and with every concurrent request. Three levers keep long context affordable. The model architecture decides the first two, grouped-query attention and hybrid layers that skip the cache altogether. The third is yours: KV cache quantization to 8-bit or 4-bit, which trades a little quality for a much smaller cache. If a model that should fit keeps running out of memory, the context length is almost always the cause, not the weights.
Which GPU for each VRAM tier
Match the card to the largest model you intend to run at Q4_K_M, then leave a tier of headroom for context. The prices below were read from Amazon US on 20 October 2026 and sit far above list because of the ongoing memory shortage, so treat them as a band and check the live price. A 5090 at about $8,600 is more than four times its $1,999 launch price, and a new-stock 3090 at about $1,900 is still the cheapest way into the 24 GB tier.
| VRAM | Card | Approx. price (20 Oct 2026) | Runs at Q4_K_M |
|---|---|---|---|
| 16 GB | RTX 5060 Ti 16GB | about $770 | Gemma 4 12B, Qwen3.5 9B, gpt-oss-20b |
| 24 GB | RTX 3090 / RTX 4090 | about $1,900 / $4,500 | Qwen3.8 27B at 64K, a 32B at short context. A 70B does not fit |
| 32 GB | RTX 5090 | about $8,600 | 32B with long context, fastest single card |
| 48 GB | RTX 6000 Ada | about $8,500 | 70B at Q4_K_M with a 16K context |
| 96 GB | RTX PRO 6000 Blackwell | about $20,000 | gpt-oss-120b, or a 70B with a long context |
The two lines worth memorizing: 24 GB is the least you want for a 27B to 32B model, and 48 GB is the entry ticket to a 70B on a single card. The 32 GB RTX 5090 is the fastest single card here, but it is still one tier short of a 70B at Q4_K_M, which lands at 42.5 GB before context. At about $8,600 against about $8,500 for the 48 GB RTX 6000 Ada, the step up to 70B no longer costs more than the fastest 32 GB card, though the 6000 Ada generates tokens at a little over half the speed of the 5090.
Can two 24 GB GPUs run a 70B model?
Yes. Two 24 GB cards give 48 GB of combined VRAM, which holds Llama 3.3 70B at Q4_K_M with a 16K context and about 2 GB to spare, and it is often cheaper than one 48 GB card. Each card carries its own runtime overhead, which the calculator counts per GPU. Ollama and llama.cpp both split the layers across GPUs automatically; in llama.cpp that is the default --split-mode layer, and --tensor-split only changes the ratio. NVLink is not required; a pair of cards on ordinary PCIe slots works, adding a small inter-GPU latency. The catch is that VRAM scales but speed does not. Splitting a 70B across two RTX 3090s at about $1,900 each (new 3090 stock is old inventory and thins out fast) lets it fit and run, but at roughly the speed of a single card holding the model, not double, because generation is still bandwidth-bound per layer and the two GPUs work in sequence on each token. Two cheaper cards are the value path to a 70B; they are not a path to a faster 70B.
How much extra VRAM for multiple users?
Every number above assumes one request at a time. Serving multiple users changes the math, because the weights are shared but the KV cache is per request. Ten concurrent sessions do not load the model ten times, but they do allocate ten KV caches, and at long context that cache is the larger cost. The concurrent-requests selector in the calculator multiplies the cache accordingly. Budget VRAM for peak concurrency, not for a single chat.
The serving framework matters here more than the model. Ollama and llama.cpp pre-allocate a full-context KV cache for every parallel slot they open (OLLAMA_NUM_PARALLEL, or -np on llama-server), so each extra slot costs a whole cache up front whether it is used or not. vLLM uses PagedAttention, which allocates the cache in small pages on demand and wastes under 4 percent of it, against the large fraction a pre-allocating runtime leaves idle. That efficiency is why vLLM serves far more concurrent requests in the same VRAM and is the right tool once more than one person is hitting the endpoint. For a single local user, Ollama is simpler and the difference does not show. The production path, vLLM behind a real API, is covered in the vLLM production guide.
Unified memory: Apple, Strix Halo and DGX Spark
Apple silicon, the AMD Strix Halo and the NVIDIA GB10 share one memory pool between CPU and GPU, and the GPU uses it directly as VRAM. That changes the capacity math. A 128 GB machine holds DeepSeek V4 Flash at its 90.9 GB UD-IQ2_M tier, which no consumer discrete GPU can fit, and the mini PCs built to hold one are compared separately. Two caveats keep it honest. The GPU cannot map the whole pool. macOS allows about two thirds of RAM on smaller Macs and three quarters on larger ones, so a 64 GB Mac behaves like a 48 GB GPU until the limit is raised with sudo sysctl iogpu.wired_limit_mb. On Strix Halo the split is set by the BIOS UMA size or the Linux GTT limit, and the DGX Spark exposes about 121 GiB of its 128 GB. And bandwidth, not capacity, still sets the speed. An RTX 5090 with 32 GB and 1,792 GB/s generates Llama 2 7B at about 298 tok/s on the llama.cpp scoreboard, a 128 GB Strix Halo at 256 GB/s at about 52.
The current Apple lineup spreads that trade-off wide. The M5 Max moves 614 GB/s with up to 128 GB, the M5 Ultra 1,228 GB/s with up to 512 GB, while Strix Halo and DGX Spark both sit near 256 to 273 GB/s with 128 GB. On the same scoreboard model the M5 Max reaches 119.9 tok/s, the M5 Ultra 179.1, the DGX Spark 57.2, and an RTX 4090 about 170. The full comparison of unified memory vs VRAM for AI covers where each class wins.
Mixture-of-experts models change the math
A model like gpt-oss-120b activates only 5.1 billion of its 117 billion parameters per token, and DeepSeek V4 Flash activates 13 billion of 284 billion. Those active counts make the model generate faster, but they do nothing for VRAM. The full weight set still has to fit in memory. gpt-oss-120b needs 62.8 GB for weights alone, so it lands on a 96 GB card or a 128 GB unified-memory machine rather than a consumer GPU. Sizing a mixture-of-experts model by its active parameters instead of its total is the most common and most expensive mistake in this whole topic. The reverse also holds: a MoE that fits runs several times faster than a dense model of the same file size, which is what the Gemma 4 26B-A4B result above shows. The DeepSeek V4 Flash hardware test measures what happens when a large MoE runs from system RAM instead.
Can you run a 70B model on a 24 GB GPU?
No, not on a single 24 GB card in a usable way. A 70B at Q4_K_M needs 42.5 GB before context, and a 24 GB card is short by almost half. The runtime will start it by offloading about half the layers to the CPU, and throughput drops to one or two tokens per second, slower than reading speed. The real options for a 70B are a single 48 GB card such as the RTX 6000 Ada, two 24 GB cards splitting the model, or a unified-memory machine with 64 GB or more. A single 24 GB card is the right tool for a 27B to 32B model, not a 70B.
Q4_K_M or Q8: which quantization?
Q4_K_M for almost everyone. It uses 0.60 bytes per parameter against Q8_0’s 1.06, so it nearly halves the VRAM, and the quality cost is small. On Qwen3.5 9B, GSM8K accuracy measured 87.6 at Q8_0 and 84.8 at Q4_K_M. The better surprise sits one step smaller: IQ4_XS scored 86.4 on the same test from a file about 8 percent smaller than Q4_K_M, and it also generated faster. Q4_K_M stays the default because it is what Ollama ships, not because it wins every test. Q8 is worth the extra memory only when a task is sensitive to small numerical drift and the card has the room to spare. Below 4-bit, quality falls off faster than the memory savings justify: Q2_K scored 76.7. The full measurement is in the GGUF vs AWQ vs GPTQ comparison.
How much VRAM for fine-tuning?
More than inference, and it depends on the method. Full fine-tuning loads the weights, the gradients, and the optimizer states, which pushes a 7B well past 60 GB and often beyond 100 GB, and puts a 70B firmly in multi-GPU territory. Parameter-efficient methods change that completely. QLoRA quantizes the base model to 4-bit and trains only small adapter layers on top, which brings a 7B fine-tune under 16 GB and a 70B fine-tune to roughly 48 GB, the same single 48 GB card that runs a 70B at inference. The Unsloth QLoRA walkthrough runs one end to end on a single card.
VRAM to model size at a glance
The whole page compresses to one reference. Pick the row for the largest model you want to run, buy a card one tier above it for context headroom, and keep the context capped to what the work needs.
| You want to run | Minimum VRAM (Q4_K_M, 32K context) | Practical card |
|---|---|---|
| 8B to 12B | 10 to 12 GB | RTX 3060 12GB, RTX 5060 Ti 16GB |
| 20B to 35B, hybrid or MoE | 16 to 24 GB | RTX 3090, RTX 4090 |
| 32B dense | 32 GB | RTX 5090, or 24 GB with a q4_0 cache |
| 70B dense | more than 48 GB (48 GB covers 16K) | 2x RTX 5090, RTX PRO 6000, 96 GB unified |
| 120B+ (MoE) | 64 GB and up | RTX PRO 6000, 128 GB unified memory |
The weights tell you what fits, the bandwidth tells you how fast, and the context tells you whether either number holds under real load. Get all three right and a local model runs the way the benchmark says it should. The card that fits each tier, with the measured speed to back it, is laid out in the best GPU for LLMs guide.
