voyage-4-nano GGUF

Official GGUF versions of voyageai/voyage-4-nano for llama.cpp and Ollama.

The embeddings match the original model, so they share the embedding space of the whole Voyage 4 series. You can index documents with voyage-4-nano locally and query with voyage-4, voyage-4-lite or voyage-4-large through the Voyage API, or the other way around.

Files

Measured against the original model (transformers, fp32) on English, Hungarian, Chinese, code and long inputs up to 6000 tokens. Q8_0 is a good default; use F16 when you need the closest match.

Requirements

• llama.cpp build b11729 or newer
• Ollama 0.40.7 or newer

Older versions fail to load the model (wrong number of tensors).

Usage

The model is trained with these prompts. Prepend them to your text:

• queries: Represent the query for retrieving supporting documents:
• documents: Represent the document for retrieval:

Normalize input text to Unicode NFC (for example unicodedata.normalize("NFC", text) in Python or text.normalize("NFC") in JavaScript). The original tokenizer does this automatically, llama.cpp does not, so decomposed (NFD) text such as macOS file names would give different embeddings.

The output is a 2048-dimensional, L2-normalized embedding. Thanks to Matryoshka training you can truncate it to 1024, 512 or 256 dimensions (and re-normalize) with little loss in quality.

llama.cpp

llama-server -hf MongoDB/voyage-4-nano-GGUF:Q8_0 --embedding -c 32768 -b 32768 -ub 32768

curl http://localhost:8080/v1/embeddings -H "Content-Type: application/json" -d '{
  "input": ["Represent the query for retrieving supporting documents: Which planet is known as the Red Planet?"]
}'

Ollama

ollama pull hf.co/MongoDB/voyage-4-nano-GGUF:Q8_0

curl http://localhost:11434/api/embed -d '{
  "model": "hf.co/MongoDB/voyage-4-nano-GGUF:Q8_0",
  "input": ["Represent the document for retrieval: Mars, known for its reddish appearance, is often referred to as the Red Planet."],
  "dimensions": 1024
}'

Ollama truncates and re-normalizes the embedding when dimensions is set.

Long inputs

voyage-4-nano uses bidirectional attention, so each input has to fit into a single batch, and the batch size is the maximum input length.

• Ollama: this repository sets num_ctx and num_batch to 32768 (see params), so ollama pull supports the full 32k context out of the box. This reserves about 6 GB of memory. If your inputs are shorter, you can lower it per request, for example "options": {"num_ctx": 8192, "num_batch": 8192} in /api/embed (about 1.3 GB), or in a Modelfile with PARAMETER num_ctx 8192 and PARAMETER num_batch 8192. Changing the values reloads the model, so set them once per application.
• llama.cpp: pass the size explicitly, for example -c 32768 -b 32768 -ub 32768. With default settings, llama-server rejects inputs longer than 512 tokens (input is too large to process).

Note on earlier community conversions

GGUF files of voyage-4-nano made before llama.cpp supported it run with causal attention and without the final 1024 -> 2048 projection. They load without errors, but their embeddings are not in the voyage-4 embedding space (cosine similarity to the original embeddings is close to zero). Please use the files in this repository instead.

License

Apache 2.0, same as the original model. See LICENSE.txt and NOTICE.txt.