Run llama-server with no --port on a current build and it answers on 9931, not 8080. Pass -no-cnv to llama-cli and you get error: invalid argument, and --mlock is gone too. This llama.cpp cheat sheet matches what the binaries do today.
It covers llama-cli and llama-server flags, the OpenAI-compatible API, offload and context tuning, GGUF quants by VRAM, llama-bench and convert_hf_to_gguf. Built and run on llama.cpp build b12020 on an 8-vCPU Ubuntu 26.04 cloud VM with no GPU in October 2026, so every token rate below is a CPU rate.
Use the llama.cpp cheat sheet (quick reference)
Every row below was checked on build b12020 (GPU-only flags were accepted on the CPU build; their effect needs a GPU). $MODEL is the path to a GGUF file, which you’ll set once in the models section. Each binary prints its full flag list with --help, and almost every flag also reads an LLAMA_ARG_* environment variable, which the help text names next to it.
Get llama.cpp and models
| Command | What it does |
|---|---|
llama-cli --version | Print the build number and commit |
llama-server --list-devices | List GPUs the build can offload to |
llama-server -hf ggml-org/Qwen3-0.6B-GGUF | Download from Hugging Face and serve (Q4_K_M, else Q8_0) |
-hf user/repo:Q5_K_M | Pick a quant from the repo |
llama-cli --cache-list | List models already downloaded |
LLAMA_CACHE=/data/models | Move the download cache |
--offline | Use the cache only, no network |
Run prompts
| Command | What it does |
|---|---|
llama-cli -m $MODEL | Interactive chat with /exit, /regen, /clear, /read |
llama-cli -m $MODEL -p "..." -st | Answer one prompt, then exit |
--reasoning off | Skip the thinking block on reasoning models |
--no-display-prompt | Don’t echo the prompt back |
-sys "..." | Set a system prompt |
llama-completion -m $MODEL -p "..." -n 128 -no-cnv | Raw text completion, no chat template |
-n 256 | Cap generated tokens |
--temp 0.7 --top-p 0.9 --min-p 0.05 | Sampling (defaults 0.80, 0.95, 0.05) |
-s 42 | Fixed seed for repeatable output |
Serve the OpenAI-compatible API
| Command | What it does |
|---|---|
llama-server -m $MODEL | Serve on 127.0.0.1:9931 (the default) |
--host 0.0.0.0 --port 8080 | Listen on all interfaces, port 8080 |
-a qwen3-0.6b | Model name the API reports (--alias) |
--api-key secret123 | Require Authorization: Bearer on API calls |
-np 4 | Four parallel slots that split -c |
--metrics | Turn on the Prometheus /metrics endpoint |
--embeddings | Embeddings-only server for an embedding model |
--no-webui | Turn off the built-in chat page |
~/.config/llama.cpp/config.ini | Default flags for every llama.cpp binary |
Call the server
These rows assume you started the server with --port 8080; drop in 9931 if you kept the default.
| Command | What it does |
|---|---|
curl localhost:8080/health | Liveness check, open even with an API key |
curl localhost:8080/v1/models | Loaded model, n_ctx and size |
POST /v1/chat/completions | OpenAI chat API, "stream": true for SSE |
POST /v1/completions | OpenAI text completion |
"chat_template_kwargs":{"enable_thinking":false} | Turn thinking off per request |
POST /tokenize | Count tokens in a string |
curl localhost:8080/slots | What each slot is doing |
curl localhost:8080/metrics | Prometheus counters (needs --metrics) |
GPU offload, context and memory
| Command | What it does |
|---|---|
-ngl all | Put every layer on the GPU (default auto) |
-ngl 20 | Put 20 layers on the GPU, rest on CPU |
-c 8192 | Context size (0 loads the model’s trained max) |
-fa on | Flash attention on, off or auto (default auto) |
-ctk q8_0 -ctv q8_0 | Quantize the KV cache, needs -fa on |
-t 4 | CPU threads for generation |
-cmoe | Keep all MoE expert weights on the CPU |
-ncmoe 10 | Keep the experts of the first 10 layers on CPU |
-ts 3,1 | Split a model 3:1 across two GPUs |
-lm mlock | Lock the model in RAM (replaces --mlock) |
-fit off | Stop llama.cpp shrinking unset values to fit memory |
Benchmark, convert and quantize
| Command | What it does |
|---|---|
llama-bench -m $MODEL | pp512 and tg128 speed, 5 repetitions |
llama-bench -m $MODEL -t 2,4,8 | Sweep thread counts in one run |
llama-bench -m a.gguf -m b.gguf | Compare quants side by side |
llama-bench -m $MODEL -fa on,off -o csv | Flash attention A/B, CSV output |
python convert_hf_to_gguf.py DIR --outtype f16 | Hugging Face safetensors to GGUF |
--outtype q8_0 | Convert straight to Q8_0 |
llama-quantize in.gguf out.gguf Q4_K_M | Quantize an F16 GGUF |
llama-quantize --dry-run in.gguf Q4_K | Show the final size without writing |
llama-perplexity -m $MODEL -f text.txt | Measure quality loss of a quant |
Below the tables, each section runs these commands with the real output, so you can see what a healthy result looks like and what each failure prints.
Install the prebuilt llama.cpp binaries
You do not need a compiler. Every llama.cpp build on GitHub ships prebuilt tarballs for Ubuntu x64 (CPU, Vulkan, CUDA 12 and 13, ROCm, SYCL), and the CPU one runs on most x86_64 distros with a recent glibc. Those builds are tagged bNNNNN and marked as pre-releases, so releases/latest points at v0.8.0, a source-only tag with no binaries. Grab the newest build tag from the releases list instead:
TAG=$(curl -s "https://api.github.com/repos/ggml-org/llama.cpp/releases?per_page=1" | grep -m1 -oP '"tag_name": "\K[^"]+')
echo $TAGDownload the CPU tarball, unpack it and put its folder on your PATH. Swap ubuntu-x64 for ubuntu-vulkan-x64 or ubuntu-cuda-13.4-x64 if you’ve got a GPU. CUDA builds also need the matching cudart-llama-${TAG}-bin-ubuntu-cuda-13.4-x64.tar.gz runtime unpacked into the same folder, or an installed CUDA toolkit. The llama.cpp releases page lists the exact names:
mkdir -p ~/llama.cpp && cd ~/llama.cpp
curl -LO "https://github.com/ggml-org/llama.cpp/releases/download/${TAG}/llama-${TAG}-bin-ubuntu-x64.tar.gz"
tar xzf "llama-${TAG}-bin-ubuntu-x64.tar.gz"
echo "export PATH=\$HOME/llama.cpp/llama-${TAG}:\$PATH" >> ~/.bashrc
source ~/.bashrcCheck that the shell finds the new binaries before you go further, because an older llama.cpp earlier in your PATH is the usual reason a flag on this page “does not exist”. The version line also gives you the build number to quote in a bug report.
llama-cli --version
llama-server --list-devicesOn the CPU tarball the device list is empty, which is correct. A CUDA or Vulkan build lists each GPU here, and if yours shows (none) on a GPU box, you’ve got the CPU tarball or a missing driver library:
version: 0.8.0-dev (build 12020, commit 0ec4027a2)
built with GNU 11.4.0 for Linux x86_64
Available devices:
(none)The folder holds 22 binaries and 38 shared libraries side by side. The ones you’ll use daily are llama-cli, llama-completion, llama-server, llama-bench, llama-quantize and llama-perplexity. If you’d rather build from source with CUDA, the llama.cpp build guide for Linux walks through cmake and GPU offload.
Download GGUF models with -hf
The -hf flag downloads a GGUF from Hugging Face and loads it in one step, on llama-cli, llama-server and llama-bench alike. Add :Q5_K_M to pick a quant. Without a tag it tries Q4_K_M, then Q8_0, then the first GGUF in the repo. ggml-org/Qwen3-0.6B-GGUF ships BF16, F16, Q4_0 and Q8_0 but no Q4_K_M, so Q8_0 is what landed here.
llama-cli -hf ggml-org/Qwen3-0.6B-GGUF -p "hi" -n 1 -st
llama-cli --cache-listYour cache list names the repo and the quant it holds, which is the string you pass back to -hf later.
number of models in cache: 1
1. ggml-org/Qwen3-0.6B-GGUF:Q8_0Older guides send you to ~/.cache/llama.cpp, but current builds store downloads in the Hugging Face hub layout under ~/.cache/huggingface/hub, as models--ggml-org--Qwen3-0.6B-GGUF/snapshots/<hash>/Qwen3-0.6B-Q8_0.gguf. That means hf download and llama.cpp now share one cache, so a model you pulled for Python tools is already there. The build source checks these locations in order and uses the first one that’s set:
LLAMA_CACHE, used as is.HF_HUB_CACHE, then the olderHUGGINGFACE_HUB_CACHE.HF_HOMEwith/hubadded.XDG_CACHE_HOME/huggingface/hub, and finally~/.cache/huggingface/hub.
Point LLAMA_CACHE at a bigger disk when the home partition is small, since a single 70B Q4_K_M is about 40 GiB. Then set $MODEL once, so the -m examples below work as written.
MODEL=$(find ~/.cache/huggingface/hub -name 'Qwen3-0.6B-Q8_0.gguf' | head -1)
echo $MODELAny GGUF you downloaded yourself works the same way with -m /path/to/file.gguf. For gated repos, export HF_TOKEN or pass -hft.
Run prompts with llama-cli and llama-completion
The binary split is the change that breaks most old cheat sheets. llama-cli is now a chat client only: it applies the model’s chat template, prints a banner and takes slash commands. The old raw-text mode moved to llama-completion, so -no-cnv on llama-cli fails:
llama-cli -m $MODEL -p hi -n 16 -no-cnvIt stops before loading anything, so a script that still calls it this way fails in under a second rather than hanging.
error: invalid argument: -no-cnvFor a one-shot answer from llama-cli, use -st (single turn). Qwen3 is a reasoning model, so add --reasoning off unless you want the thinking block, and --no-display-prompt to keep the prompt out of the output:
llama-cli -m $MODEL -p "Explain what a GGUF file is in two sentences." -n 128 -t 8 -c 4096 -st --reasoning off --no-display-promptAfter the banner and the slash-command list, you get the answer and a one-line speed summary in place of the old perf block.
> Explain what a GGUF file is in two sentences.
A GGUF file is a container format used for storing and transferring data between different computing environments, often used in machine learning and data processing tasks.
[ Prompt: 158.4 t/s | Generation: 9.9 t/s ]
Exiting...The answer is wrong, which tells you more about a 0.6B model than about llama.cpp. GGUF is the single-file model format llama.cpp loads, holding the weights, tokenizer and metadata. Without --reasoning off, the same run printed a [Start thinking] block first and ran out of its 128 tokens mid-answer, which is the first thing to check when a reply looks cut off.
Scripts and pipelines that want plain text continuation use llama-completion, which still takes -no-cnv and prints the full perf block at the end:
llama-completion -m $MODEL -p "Explain what a GGUF file is in two sentences." -n 128 -t 8 -c 4096 -no-cnvThe two lines to read are prompt eval (how fast it reads your input) and eval (how fast it writes). Eval is the number users feel, because it sets how fast text appears.
common_perf_print: load time = 544.28 ms
common_perf_print: prompt eval time = 121.99 ms / 12 tokens ( 10.17 ms per token, 98.37 tokens per second)
common_perf_print: eval time = 11021.11 ms / 127 runs ( 86.78 ms per token, 11.52 tokens per second)
common_perf_print: total time = 11218.72 ms / 139 tokensBecause llama-completion skips the chat template, the model treats your prompt as text to continue. The same question produced a rambling “Answer:” block that repeated itself until it hit 128 tokens. Use it for completion-style prompts, and use llama-cli -st or the server when you want a chat answer.
Serve an OpenAI-compatible API with llama-server
llama-server is the binary most people run all day: an OpenAI-compatible API plus a web chat UI on one port. It binds to 127.0.0.1 by default, and on this build the default port is 9931. Run it with no --port and the log says so:
llama-server -hf ggml-org/Qwen3-0.6B-GGUF -t 4Watch the slot line too. With no -c, the server takes the model’s trained context (40,960 tokens here) and sizes the KV cache for it, so on a bigger model that default alone can eat several gigabytes.
0.08.349.670 W srv server_setup: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
0.13.173.312 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 40960, kv_unified = 'true'
0.13.186.419 I srv server_on_li: listening on http://127.0.0.1:9931Most clients and old docs expect 8080, so pin the port, the context and a short model alias. The alias is the name the API reports and what clients send as model:
llama-server -m $MODEL --host 127.0.0.1 --port 8080 -c 4096 -t 8 --alias qwen3-0.6bFrom a second terminal, check health and the model list. Load balancers and systemd checks poll /health, so it’s worth seeing what a good answer looks like.
curl -s localhost:8080/health
curl -s localhost:8080/v1/models | python3 -m json.tool/health returns {"status":"ok"}. The model list carries both an Ollama-style models array and the OpenAI data array, and the meta block is the useful part, because it shows the context you actually got against the one the model was trained for:
"id": "qwen3-0.6b",
"object": "model",
"owned_by": "llamacpp",
"meta": {
"vocab_type": 2,
"n_vocab": 151936,
"n_ctx": 4096,
"n_ctx_train": 40960,
"n_embd": 1024,
"n_params": 751632384,
"size": 798801920,
"ftype": "Q8_0"
}Here n_ctx is 4,096 because of -c 4096, a tenth of the 40,960 the model supports. That’s deliberate on a small box, and it’s the first field to check when long prompts get cut, since the server trims anything past it.
Send a chat completion
The endpoint is the standard OpenAI one, so any client that speaks that API works without changes. Start with a short request and a 64-token cap, because a small cap is where reasoning models trip people up.
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen3-0.6b","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":64}' | python3 -m json.toolLook at content. It’s empty, and finish_reason is length, because the model spent all 64 tokens thinking and the server split that into reasoning_content:
"finish_reason": "length",
"index": 0,
"message": {
"role": "assistant",
"content": "",
"reasoning_content": "Okay, the user wants me to say \"hello in five words.\" Let me think. First, I need to make sure it's exactly five words. Hello is three words, so I need to add two more. Maybe \"Hello, how are you?\" That's four words. Hmm, not quite."That’s the most common “llama-server returns empty content” report, and it is not a bug. You have three fixes: raise max_tokens, start the server with --reasoning off, or turn thinking off for one request with chat_template_kwargs:
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen3-0.6b","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":64,"chat_template_kwargs":{"enable_thinking":false}}' | python3 -m json.toolNow the answer comes back in content with finish_reason set to stop. The timings block shows 15 of the prompt tokens came from the cache, since the server reused the earlier prompt prefix:
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "Hello!"
}
[...]
"timings": {
"cache_n": 15,
"prompt_n": 3,
"prompt_ms": 160.62,
"prompt_per_token_ms": 53.54,
"prompt_per_second": 18.67762420620097,
"predicted_n": 3,
"predicted_ms": 54.675,
"predicted_per_token_ms": 27.3375,
"predicted_per_second": 36.57978966620942
}Add "stream": true and the server sends Server-Sent Events, one data: line per token, with reasoning_content and content deltas kept separate. Pass curl -sN so curl doesn’t buffer them.
Call it from the OpenAI Python SDK
Any OpenAI client works once you point base_url at the server. The API key can be any string while the server has no --api-key. Set up a venv with uv and open a new script file.
uv venv ~/llama-client && uv pip install --python ~/llama-client openai
vim ~/llama-client/client.pyPaste this script. Non-OpenAI fields such as chat_template_kwargs go in extra_body:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="sk-no-key-required")
resp = client.chat.completions.create(
model="qwen3-0.6b",
messages=[{"role": "user", "content": "Explain what a GGUF file is in one sentence."}],
max_tokens=96,
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)
print(resp.usage)
for chunk in client.chat.completions.create(
model="qwen3-0.6b",
messages=[{"role": "user", "content": "Count from 1 to 5."}],
max_tokens=48,
stream=True,
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
):
print(chunk.choices[0].delta.content or "", end="", flush=True)
print()Run it with the venv’s Python, with the server from the previous step still up on port 8080.
~/llama-client/bin/python ~/llama-client/client.pyYour first call prints the answer and a normal CompletionUsage object, and the second streams token by token, so both code paths work unchanged against llama.cpp (openai 3.42.0 here):
A GGUF file is a binary format used to store and transmit data in a compact and efficient manner, commonly used in machine learning and other data processing tasks.
CompletionUsage(completion_tokens=33, prompt_tokens=24, total_tokens=57, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cache_write_tokens=None, cached_tokens=0, image_tokens=None, text_tokens=None))
Counting from 1 to 5:
1, 2, 3, 4, 5.
So, the answer is: **1, 2, 3, 4, 5**.Agents and editors that speak the OpenAI API take the same base URL. The DeepSeek Harness local model guide points a coding agent at a local llama.cpp server the same way, so the setup above is most of the work.
Lock it down with –api-key and –metrics
The startup warning about “no API key is set and CORS allows all origins” matters the moment you pass --host 0.0.0.0. Anyone who can reach the port can then use your CPU or GPU, and a web page in your browser can call it too. Start the server with a key and the metrics endpoint on.
llama-server -m $MODEL --port 8080 -c 4096 --alias qwen3-0.6b --api-key secret123 --metrics -t 4The key doesn’t cover every path, and it is worth knowing which ones stay open before you expose the port. With the key set, the endpoints behaved like this:
- A missing or wrong key on
/v1/*returns HTTP 401 with{"error":{"message":"Invalid API Key","type":"authentication_error","code":401}}. Authorization: Bearer secret123returns HTTP 200./healthstays open with no key, so load balancer probes keep working./metricsneeds the key too, so give Prometheus a bearer token.
Without --metrics, /metrics answers HTTP 501 with “This server does not support metrics endpoint. Start it with `–metrics`”, and /v1/embeddings does the same until you start with --embeddings. That flag restricts the server to embeddings, so run it as a second instance with a dedicated embedding model, not on your chat server. That 501 is the answer to “why is my Prometheus target down”, so check the flag before the firewall. Pull the counters with the key.
curl -s -H 'Authorization: Bearer secret123' localhost:8080/metrics | grep -v '^#' | head -5Every metric carries the llamacpp: prefix. These are the counters to graph for throughput, since tokens divided by seconds gives you the rate. Twelve tokens in 0.47 seconds is about 25 t/s for the one request sent here.
llamacpp:prompt_tokens_total 15
llamacpp:prompt_tokens_cached_total 0
llamacpp:prompt_seconds_total 0.20209
llamacpp:tokens_predicted_total 12
llamacpp:tokens_predicted_seconds_total 0.472533Counters reset when the server restarts, so graph them with rate() in Prometheus rather than reading raw totals.
Set defaults in config.ini
Typing the same six flags on every start gets old. Current builds read /etc/llama.cpp/config.ini and then ~/.config/llama.cpp/config.ini before the environment and the command line, which override them. Keys are the long flag names without the leading dashes, so --ctx-size becomes ctx-size.
mkdir -p ~/.config/llama.cpp
vim ~/.config/llama.cpp/config.iniThis is the file that was tested. Options a binary does not support are skipped, so port only affects llama-server and the same file is safe for every tool.
port = 8080
ctx-size = 8192
threads = 4Start the server with two slots and no port flag, so the port has to come from the file.
llama-server -m $MODEL --alias qwen3-0.6b -np 2 --reasoning offThe first log line, using config file:, names the file it read. After it, the server picks up 4 threads and port 8080 and splits the 8,192-token context into two 4,096-token slots:
0.01.226.480 I cmn init: llama threadpool init, n_threads = 4
0.01.601.650 I srv load_model: initializing, n_slots = 2, n_ctx_slot = 4096, kv_unified = 'false'
0.01.622.547 I srv server_on_li: listening on http://127.0.0.1:8080That split is the thing to plan for. Once you set -np yourself the KV cache stops being unified and -c becomes the total, so -np 4 -c 8192 gives each user 2,048 tokens. With the default automatic slots the cache stays unified and every slot can use the full -c, as the earlier runs showed. Don’t put model or hf-repo in the file, because the upstream preset docs warn a configured hf-repo still loads when you pass -m.
Tune -ngl, -c and –flash-attn
Three flags decide whether a model fits and how fast it runs: -ngl (layers on the GPU), -c (context, which sizes the KV cache) and -fa (flash attention). Current builds default -ngl to auto and --fit to on, so llama.cpp shrinks any value you didn’t set until the model fits in free device memory. Values you set yourself are left alone, which is why an explicit -c 131072 can still fail with out-of-memory.
The KV cache is the part people underestimate. It grows linearly with -c, and the default log level hides it, so run with -lv 4 and read the llama_kv_cache line. Try it at 32K context on the same Qwen3 0.6B model.
llama-completion -m $MODEL -p hi -n 8 -no-cnv -lv 4 -t 4 -c 32768 2>&1 | grep 'llama_kv_cache: size'At 32K tokens the cache alone is 3.5 GiB, more than four times the 762 MiB of weights.
0.03.593.022 I llama_kv_cache: size = 3584.00 MiB ( 32768 cells, 28 layers, 1/1 seqs), K (f16): 1792.00 MiB, V (f16): 1792.00 MiBQuantizing the cache to q8_0 roughly halves it. The V cache can only be quantized with flash attention on, so the pair always goes together.
llama-completion -m $MODEL -p hi -n 8 -no-cnv -lv 4 -t 4 -c 32768 -fa on -ctk q8_0 -ctv q8_0 2>&1 | grep 'llama_kv_cache: size'Below are the three runs side by side, including the 2K baseline, with peak RSS from /usr/bin/time -v. Quantizing the cache saved 1.6 GiB of RAM at 32K, and the same halving applies to VRAM when the cache sits on a GPU.
| Setting (Qwen3 0.6B Q8_0) | KV cache | Peak RSS |
|---|---|---|
-c 2048 | 224 MiB | 1.03 GiB |
-c 32768 | 3,584 MiB | 4.31 GiB |
-c 32768 -fa on -ctk q8_0 -ctv q8_0 | 1,904 MiB | 2.67 GiB |
The per-token cost is 2 × layers × KV heads × head size × bytes. For Qwen3 0.6B that’s 2 × 28 × 8 × 128 × 2 bytes, or 112 KiB a token, which is exactly the 224 MiB measured at 2,048 tokens. A Llama 3 8B (32 layers, 8 KV heads) costs 128 KiB a token, so an 8K context adds 1 GiB on top of the weights. Run that sum before you pick a quant.
On a GPU, the order you change things in matters, because each step costs more speed than the one before it.
- Set
-cto what you’ll really use, not the model maximum. - Try
-ngl all; if it doesn’t fit, add-fa on -ctk q8_0 -ctv q8_0before you drop layers. - On MoE models, try
-cmoeor-ncmoe Nnext, since only a few experts run for each token and the shared layers stay on the GPU. - Only then lower
-nglone step at a time and rerun llama-bench.
On a CPU-only build, -ngl is accepted and does nothing, and flash attention made no measurable difference here: pp512 at 104.05 t/s off against 106.05 on, and tg64 at 30.40 against 30.33, both inside the run’s own ± spread. So leave -fa on auto on CPU unless you want a quantized V cache, because turning it on buys you nothing else there. The GPU numbers for the same flags are in the Ollama vs vLLM vs llama.cpp benchmark.
Pick a GGUF quant by VRAM
Q4_K_M is the right default for most cards. It’s what -hf picks, and llama-quantize --help lists it at 4.58 GiB for Llama 3 8B with a +0.1754 perplexity cost. Q5_K_M (+0.0569) and Q6_K (+0.0217) are worth the space when you have it, while Q8_0 (+0.0026) is close to lossless. Below Q4, Q3_K_M (+0.6569) is the last stop before quality drops off, and Q2_K (+3.5199) is for “it runs at all” only.
The table scales those Llama 3 8B file sizes by parameter count, so you can read off the weights for common model sizes. It leaves out the KV cache and about 0.5 to 1 GiB of runtime overhead, which you add from the last section.
| Quant | GiB per 1B params | 8B | 14B | 32B | 70B |
|---|---|---|---|---|---|
| Q3_K_M | 0.47 | 3.7 | 6.5 | 14.9 | 32.6 |
| Q4_K_M | 0.57 | 4.6 | 8.0 | 18.2 | 39.9 |
| Q5_K_M | 0.66 | 5.3 | 9.3 | 21.2 | 46.5 |
| Q6_K | 0.76 | 6.1 | 10.7 | 24.5 | 53.6 |
| Q8_0 | 0.99 | 8.0 | 13.9 | 31.7 | 69.4 |
| F16 (2 bytes a weight) | 1.86 | 14.9 | 26.1 | 59.6 | 130.4 |
An 8K context costs 1 GiB on an 8B model and about 2.5 GiB on a 70B (80 layers at the same 8 KV heads), so leave 1.5 to 3.5 GiB free for cache and runtime. The picks per card fall out of that budget.
| VRAM | Best fit at 8K context | Stretch |
|---|---|---|
| 6 GB | 8B Q4_K_M at 4K context | 4B Q8_0 |
| 8 GB | 8B Q5_K_M | 8B Q6_K with -ctk q8_0 -ctv q8_0 |
| 12 GB | 14B Q4_K_M or 8B Q8_0 | 14B Q5_K_M at 4K context |
| 16 GB | 14B Q6_K | 24B Q4_K_M |
| 24 GB | 32B Q4_K_M | 32B Q5_K_M with q8_0 KV |
| 32 GB | 32B Q6_K | 70B Q3_K_M with partial -ngl |
| 48 GB | 70B Q4_K_M | 70B Q4_K_M with q8_0 KV and 16K context |
If you’re between two rows, lean toward the bigger model at the smaller quant, since Q4_K_M costs a 14B far less than the 6 billion extra parameters give it. For the per-card VRAM picture, including used datacenter cards, the LLM VRAM calculator runs the same arithmetic for any model, and the GGUF vs AWQ vs GPTQ comparison shows how GGUF quants hold up against the GPU-only formats.
Benchmark with llama-bench
llama-bench gives you the two numbers that matter: pp512 (prompt processing, tokens per second reading 512 input tokens) and tg128 (generating 128 tokens). Every test flag takes a comma list, so one command sweeps a setting. Start with thread counts, because the default of every core is often wrong.
llama-bench -m $MODEL -p 512 -n 128 -t 4,8 -r 2Read the ± column before the rate. At 8 threads the generation rate fell to 16.65 with a spread of ± 11.84, against a steady 39.81 at 4 threads.
Using every vCPU made generation slower, not faster. The VM was shared, and with other jobs on the box, 8 threads meant some of them waited on busy cores every step. A second sweep of -t 2,4,8 while the load average climbed from 12 to 17 made the point harder: tg128 ran 16.66 at 2 threads, 23.28 at 4 and 0.68 at 8. Generation is limited by memory bandwidth anyway, so past a few threads you only add contention. On your own machine, sweep -t from half your physical cores upward and keep the lowest count that holds the top rate. A large ± spread means your numbers aren’t usable yet, so raise -r or stop whatever else is running.
Once the thread count is settled, these four forms cover most other questions. They test flash attention, GPU offload, long-context speed and CSV export, in that order.
llama-bench -m $MODEL -fa on,off -t 4
llama-bench -m $MODEL -ngl 0,99 -p 512 -n 128
llama-bench -m $MODEL -pg 2048,128 -d 0,8192
llama-bench -m $MODEL -o csv > bench.csv-pg runs a combined prompt-plus-generation test, and -d runs at a filled context depth, which shows how far speed drops once the cache is full. -o csv, json, jsonl and sql add the CPU model, build commit and every setting per row, so you can line runs from different machines up later. In the flash attention column, auto shows up as -1.
Convert a Hugging Face model with convert_hf_to_gguf
When a model only ships as safetensors, you convert it yourself. The whole loop is five steps, and the only slow one on a big model is the download:
- Clone the source at your build tag and install the converter’s Python requirements.
- Download the safetensors repo.
- Convert it to an F16 GGUF.
- Quantize the F16 file with
llama-quantize. - Compare perplexity against F16 before you ship the quant.
The converter is a Python script in the source tree, not in the binary tarball, so clone the same tag you installed. A matching tag keeps the converter and the binaries agreeing on the GGUF format.
git clone --depth 1 --branch ${TAG} https://github.com/ggml-org/llama.cpp ~/llama.cpp/src
cd ~/llama.cpp
uv venv convvenv
uv pip install --python convvenv -r src/requirements/requirements-convert_hf_to_gguf.txtThe requirements file points at PyTorch’s CPU index, so you get torch==2.11.0+cpu instead of a multi-gigabyte CUDA wheel. All 28 packages installed in about 5 seconds with uv. Next, download a small model to practise on. SmolLM2 135M is 257 MB of safetensors, so the whole loop runs in a minute.
convvenv/bin/hf download HuggingFaceTB/SmolLM2-135M-Instruct --local-dir hf/SmolLM2-135M-InstructConvert it to F16 GGUF first. F16 is the clean starting point for every other quant, and requantizing an already quantized file loses quality, which is why llama-quantize refuses unless you pass --allow-requantize.
mkdir -p gguf
convvenv/bin/python src/convert_hf_to_gguf.py hf/SmolLM2-135M-Instruct --outfile gguf/smollm2-135m-f16.gguf --outtype f16It took 17 seconds and ends with a success line. If the script stops on an unknown model architecture instead, the model is newer than your clone, so pull a newer tag and convert again.
INFO:gguf.gguf_writer:gguf/smollm2-135m-f16.gguf: n_tensors = 272, total_size = 269.1M
INFO:hf-to-gguf:Model successfully exported to gguf/smollm2-135m-f16.ggufThe converter writes f32, f16, bf16 and q8_0 directly with --outtype, and --outtype q8_0 gave a file identical in size to quantizing afterwards (139 MB). For K-quants such as Q4_K_M, run llama-quantize on the F16 file.
llama-quantize gguf/smollm2-135m-f16.gguf gguf/smollm2-135m-Q4_K_M.gguf Q4_K_MQuantizing took under 2 seconds on a model this small. Skip the per-tensor lines and read the summary at the end, because this one has a catch.
llama_model_quantize_impl: model size = 256.63 MiB (16.00 BPW)
llama_model_quantize_impl: quant size = 98.87 MiB (6.17 BPW)
llama_model_quantize_impl: WARNING: 180 of 272 tensor(s) required fallback quantization
llama_quantize: quantize time = 1749.17 msA Q4_K_M at 6.17 bits per weight isn’t really a 4-bit file. K-quants work in blocks of 256 values, and SmolLM2’s 576-wide rows don’t divide by 256, so 180 tensors fell back to other types (the log shows them going to q5_0). Large models with 4096- or 5120-wide layers don’t hit this, but small and odd-sized models do, and the fallback warning is how you spot it. The other quants came out as the table shows, with perplexity run on the F16 and Q4_K_M files.
| SmolLM2 135M | File size | Perplexity (10 chunks, -c 128) |
|---|---|---|
| F16 | 259 MB | 15.06 ± 1.69 |
| Q8_0 | 139 MB | not run |
| Q6_K | 132 MB | not run |
| Q5_K_M | 107 MB | not run |
| Q4_K_M | 101 MB | 16.04 ± 1.78 |
Q4_K_M cost about 6.5% perplexity on this tiny model, a lot more than the +0.18 llama-quantize lists for Llama 3 8B, since small models have less redundancy to absorb rounding. Check a quant of your own with llama-perplexity -m model.gguf -f text.txt against the F16 file on the same text. Once the GGUF looks right, it loads in llama-server -m like any download, and the Ollama commands cheat sheet shows how to import the same file into Ollama with a Modelfile.
Fix common llama.cpp errors
Each of these came up while building this sheet, with the exact message the build printed.
error: invalid argument: -no-cnv
llama-cli is chat-only now. Run the same command with llama-completion, or drop -no-cnv and add -st to get one chat answer and exit. The same message appears for --no-mmap and --mlock, which became -lm none and -lm mlock.
couldn’t bind HTTP server socket, hostname: 127.0.0.1, port: 8080
Another process holds the port, often a llama-server you started earlier. The server logs exiting due to HTTP server error and exits with code 1. Find the owner with ss -ltnp | grep 8080, stop it, or pick another --port.
failed to open GGUF file (No such file or directory)
The -m path is wrong, usually because the file is inside the Hugging Face cache rather than where you expected. Run llama-cli --cache-list, then find ~/.cache/huggingface/hub -name '*.gguf' for the real path, or use -hf and let llama.cpp resolve it.
quantized V cache requires flash_attn to be enabled
You passed -ctv q8_0 with -fa off, and the context never gets created. Set -fa on whenever you quantize the V cache, so the pairing is explicit in your service file.
failed to quantize: requantizing from type q8_0 is disabled
You pointed llama-quantize at a file that’s already quantized, here the Q8_0 Qwen3 download. Quantize from the F16 or BF16 GGUF instead. --allow-requantize forces it, but the help text warns this can severely reduce quality, so only use it when no 16-bit source exists.
failed to mlock … Cannot allocate memory
-lm mlock ran, printed this warning and carried on without locking, because the memlock limit (ulimit -l printed 8192 KiB on the test VM) is far below a 762 MiB model. Raise LimitMEMLOCK=infinity in the systemd unit that runs llama-server, or drop -lm mlock if the box has no swap anyway.
the argument has been removed. use –spec-draft-n-max
--draft and --draft-min were renamed when speculative decoding gained more types. The full message names both replacements: --spec-draft-n-max (with --spec-draft-n-min) for a draft model from -md, or --spec-ngram-mod-n-max for n-gram drafting with no second model. --spec-default turns on the built-in defaults.
Replace old llama.cpp flags
If you’re updating scripts written against an older build, this is the map. Every “now” entry was checked on b12020:
| Old usage | Now |
|---|---|
llama-cli -no-cnv -p "..." | llama-completion -no-cnv -p "...", or llama-cli -st |
--mlock | -lm mlock |
--no-mmap | -lm none |
--draft N | --spec-draft-n-max N |
| llama-server on port 8080 | Port 9931 unless you pass --port |
-ngl 99 to offload everything | -ngl all, or leave the auto default |
-fa as a bare switch | -fa on|off|auto, default auto |
--jinja for tool calling | On by default, --no-jinja turns it off |
Models in ~/.cache/llama.cpp | ~/.cache/huggingface/hub |
Pin the build tag in anything you deploy, because flags still move between builds. When a flag stops working, llama-server --help | grep -A2 -- '--flag' on the new build tells you the replacement faster than any changelog. More references like this one live on the cheat sheets hub.
