OpenAI’s embeddings endpoint quietly became one of the most-used pieces of the entire API stack in 2026. Every retrieval-augmented generation (RAG) pipeline, every “chat with your docs” tool, and every recommendation engine built on top of GPT-6 Astra, Sol, or Luna leans on the same primitive underneath: a vector produced by text-embedding-3-small or text-embedding-3-large. Search volume for “openai embedding models” and “text-embedding-3-small” has stayed high month over month, and for good reason: developers keep hitting the same wall. They can call a chat model fine, but turning a folder of PDFs or a support ticket archive into something a model can actually search is a different skill entirely.
This tutorial walks through the full setup: getting a scoped API key, installing the current OpenAI Python SDK, generating your first embedding, choosing between the two current models, shrinking vector size without losing much accuracy, storing vectors in a real database, and wiring the whole thing into a GPT-6 Astra-powered answer generator through the Responses API. By the end you will have a complete, working semantic search project you can drop into a real application, not just a toy snippet.
The context matters here. Through most of 2024 and 2025, teams defaulted to the older text-embedding-ada-002 model out of habit, mostly because it was the first embedding model widely documented in tutorials. OpenAI’s own model guidance has since pushed the current text-embedding-3 generation as the standard, and the Assistants API, which many early RAG tutorials were built around, was fully retired on August 26, 2026. Anything you build today should target the Responses API and the current embedding models directly, not the retired interfaces still floating around in older blog posts and Stack Overflow answers.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
How OpenAI Embeddings Actually Work
An embedding is a list of floating-point numbers (1,536 of them for text-embedding-3-small at default settings) that represents the meaning of a piece of text in a high-dimensional space. OpenAI describes it plainly: “embeddings are a numerical representation of text that can be used to measure the relatedness between two pieces of text,” according to the text-embedding-3-large model documentation. Two sentences that mean similar things end up close together in that space, even if they don’t share a single word in common. “How do I reset my password?” and “I forgot my login credentials” land near each other. “How do I reset my password?” and “What time does the store open?” do not.
That property is what makes embeddings useful for far more than search. The same underlying vectors power document clustering (grouping thousands of support tickets into themes without manual tagging), classification (routing an incoming message to the right team based on similarity to labeled examples), deduplication (catching near-identical support tickets or job postings), anomaly detection (flagging a transaction description that sits unusually far from normal patterns), and recommendation systems (surfacing products or articles whose embeddings sit close to ones a user already engaged with). Semantic search is simply the most common entry point, since it’s the piece most developers need first when they want a chatbot to answer questions about their own content instead of only what the underlying model was trained on.
Prerequisites: What You Need Before You Start
Before writing any code, get these pieces in place. None of them are optional if you want the examples in this guide to run without modification.
- An OpenAI platform account with billing enabled at platform.openai.com. The embeddings endpoint is pay-as-you-go, and there is no free permanent tier.
- Python 3.10 or newer (3.11 or 3.12 recommended for faster async I/O when batching requests).
- The current OpenAI Python SDK, installed with
pip install --upgrade openai. Version pinning matters here because the SDK’s Responses API surface has changed shape multiple times in 2026. - PostgreSQL 15 or newer with the
pgvectorextension, or a Qdrant instance (self-hosted via Docker or the managed cloud tier), for the storage steps. - Basic familiarity with the command line and one text editor or IDE. This guide uses plain Python scripts, no framework required.
- A rough idea of what you’re indexing (support tickets, product docs, a codebase, meeting transcripts) so you can follow the chunking step with your own data instead of the sample text.
You do not need a GPU, a vector database cluster, or an enterprise OpenAI plan. The embeddings endpoint runs entirely server-side, and your machine only needs to make HTTP requests and store the resulting arrays of floating-point numbers. That’s worth emphasizing because it’s the most common misconception people bring to this tutorial from local-LLM tutorials that assume you’re running inference on your own hardware. With OpenAI’s hosted embeddings, the heavy lifting happens on OpenAI’s infrastructure, and your job is entirely about plumbing: getting text in, getting vectors out, and storing them somewhere queryable.
Steps 1-3: Get Your API Key, Install the SDK, and Make Your First Call
Step 1: Create a Scoped API Key
Log in to the OpenAI dashboard and open the API keys page. Instead of reusing an organization-wide secret, create a new project and scope a key to it. This keeps your embeddings workload’s spend and rate limits isolated from anything else you run on the same account, which matters once you start batching thousands of chunks. Name the key something identifiable, like embeddings-prod or embeddings-dev, and store it somewhere other than your source code.
Step 2: Install and Upgrade the OpenAI Python SDK
Open a terminal, create a virtual environment, and install the SDK.
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade openai psycopg2-binary numpy qdrant-clientSet the API key as an environment variable rather than hardcoding it. On macOS or Linux:
export OPENAI_API_KEY="sk-proj-your-key-here"On Windows PowerShell, use $Env:OPENAI_API_KEY = "sk-proj-your-key-here". The Python SDK reads this variable automatically, so you never pass the key directly in your scripts.
Step 3: Make Your First Embeddings API Call
According to OpenAI’s own documentation, “to get an embedding, send your text string to the embeddings API endpoint along with the embedding model name (e.g., text-embedding-3-small),” as described in the official embeddings guide. Here is the minimum working example:
from openai import OpenAI
client = OpenAI()
response = client.embeddings.create(
model="text-embedding-3-small",
input="OpenAI embeddings turn text into a vector of numbers."
)
vector = response.data[0].embedding
print(len(vector)) # 1536
print(vector[:5]) # first five floatsIf this prints a length of 1536 and a handful of floating-point numbers, your setup is correct. That 1,536-element array is the default output size for text-embedding-3-small, and it is what every downstream step in this tutorial is built around.
Steps 4-5: Pick a Model and Tune Vector Size
Step 4: Choose Between text-embedding-3-small and text-embedding-3-large
As of September 2026, OpenAI’s currently documented embedding lineup is still the third-generation pair introduced in early 2024: text-embedding-3-small and text-embedding-3-large. There has been no fourth-generation successor published on OpenAI’s official model pages, so any article or forum post referencing a “text-embedding-4” model is describing a third-party wrapper or a speculative listing, not an OpenAI release. Both current models accept up to 8,191 tokens per input and are priced per million input tokens.
| Model | Price per 1M tokens | Default dimensions | Max dimensions | Token limit |
|---|---|---|---|---|
| text-embedding-3-small | $0.02 | 1,536 | 1,536 | 8,191 |
| text-embedding-3-large | $0.13 | 3,072 | 3,072 | 8,191 |
| Legacy text-embedding-ada-002 (older generation) | ~5x more than small | 1,536 | 1,536 | 8,191 |
The practical rule most teams settle on: start with text-embedding-3-small for everything. It’s roughly 6.5x cheaper per token than text-embedding-3-large, and third-party MTEB evaluations put it around 62.3 versus roughly 64.6 for the large model, a real but modest quality gap for most retrieval workloads. Only move to the large model if your own evaluation set (not a generic benchmark) shows a measurable recall or ranking improvement that justifies the extra cost and storage.
Multilingual content is one area where it’s worth testing both models directly rather than assuming. Both text-embedding-3-small and text-embedding-3-large were trained with improved multilingual performance over the older ada-002 generation, but the gap between the two current models can widen or narrow depending on the language pair and domain. If your corpus mixes English support tickets with, say, Spanish or Japanese ones, run your evaluation set (covered later in this guide) across each language separately rather than assuming a single aggregate score tells the whole story.
Step 5: Shorten Vectors With the Dimensions Parameter
Both current models support a dimensions parameter that truncates the output vector while preserving most of its semantic structure. This matters because vector size directly drives your database storage bill and query latency: a 3,072-dimension vector takes twice the disk space of a 1,536-dimension one.
response = client.embeddings.create(
model="text-embedding-3-large",
input="Shortened embeddings still capture most of the meaning.",
dimensions=1024
)
vector = response.data[0].embedding
print(len(vector)) # 1024, not 3072One rule that trips up almost everyone the first time: once you pick a dimension size for a collection, every vector in that collection, documents and queries alike, must use the same model and the same dimension count. Changing the setting later means re-embedding your entire corpus and rebuilding the index, not a quick config flip.
Steps 6-7: Prepare Your Documents at Scale
Step 6: Chunk Long Documents Before Embedding
The 8,191-token limit sounds generous until you try to embed a full technical manual or a long support thread in one call. OpenAI’s own cookbook guidance lays out the standard pattern plainly: “the simplest way to use embeddings for search is as follows: split your text corpus into chunks smaller than the token limit, embed each chunk of text, store those embeddings in your own database or in a vector search provider, embed the search query, find the closest embeddings in your database, return the top results,” as documented in the OpenAI cookbook.
A simple, effective chunking function for prose or documentation:
import tiktoken
def chunk_text(text, max_tokens=500, overlap=50):
enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode(text)
chunks = []
start = 0
while start < len(tokens):
end = start + max_tokens
chunk_tokens = tokens[start:end]
chunks.append(enc.decode(chunk_tokens))
start = end - overlap
return chunksKeeping chunks around 300-500 tokens with a small overlap (30-50 tokens) preserves context across chunk boundaries without embedding the same sentence twice at full price. For code or structured data, chunk along logical boundaries (functions, sections, rows) instead of a fixed token count.
Step 7: Batch Requests to Control Cost and Latency
The embeddings endpoint accepts a list of strings in a single input field, so there is rarely a reason to fire one HTTP request per chunk.
def embed_batch(texts, model="text-embedding-3-small", batch_size=100):
vectors = []
for i in range(0, len(texts), batch_size):
batch = texts[i:i + batch_size]
resp = client.embeddings.create(model=model, input=batch)
vectors.extend([d.embedding for d in resp.data])
return vectorsFor very large one-time backfills, such as indexing a whole knowledge base for the first time, look at OpenAI's Batch API, which processes jobs asynchronously within a 24-hour window at a discount off standard pricing. It's the right tool when you're embedding hundreds of thousands of chunks and don't need the result in real time.
Steps 8-9: Store Vectors in a Database
Step 8: Store Embeddings in Postgres With pgvector
If your application already runs on PostgreSQL, pgvector is usually the path of least resistance, since you keep vectors, metadata, and your existing relational data in one system with one set of credentials.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE documents (
id BIGSERIAL PRIMARY KEY,
content TEXT NOT NULL,
embedding VECTOR(1536)
);
CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops);Inserting from Python is straightforward once the table exists:
import psycopg2
conn = psycopg2.connect("dbname=ragdb user=postgres")
cur = conn.cursor()
for text, vector in zip(chunks, vectors):
cur.execute(
"INSERT INTO documents (content, embedding) VALUES (%s, %s)",
(text, vector)
)
conn.commit()Step 9: Store Embeddings in Qdrant as a Dedicated Alternative
If you'd rather not add vector workloads to your primary transactional database, Qdrant is a solid open-source option with strong metadata filtering, and it runs the same way locally or in its managed cloud.
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct
qdrant = QdrantClient(url="http://localhost:6333")
qdrant.recreate_collection(
collection_name="docs",
vectors_config=VectorParams(size=1536, distance=Distance.COSINE)
)
points = [
PointStruct(id=i, vector=vec, payload={"text": text})
for i, (text, vec) in enumerate(zip(chunks, vectors))
]
qdrant.upsert(collection_name="docs", points=points)Both approaches work. Pick pgvector if simplicity and a single database win out, and pick Qdrant if you expect the collection to grow past a few million vectors or need advanced filtering on payload fields.
Steps 10-11: Search and Generate Answers
Step 10: Compute Cosine Similarity and Rank Results
OpenAI's documentation is direct about how retrieval actually works under the hood: "to retrieve the most relevant documents we use the cosine similarity between the embedding vectors of the query and each document, and return the highest scored documents," per the embeddings guide. If you're not using a vector database's built-in search yet, here's the manual version so you understand what's happening:
import numpy as np
def cosine_similarity(a, b):
a, b = np.array(a), np.array(b)
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def search(query, corpus_texts, corpus_vectors, top_k=3):
query_vector = client.embeddings.create(
model="text-embedding-3-small", input=query
).data[0].embedding
scores = [cosine_similarity(query_vector, v) for v in corpus_vectors]
ranked = sorted(zip(scores, corpus_texts), reverse=True)
return ranked[:top_k]Once you move to pgvector or Qdrant, the database engine performs this comparison for you using an approximate nearest-neighbor index (HNSW in both cases), which scales to millions of vectors far better than the manual loop above.
Step 11: Connect Embeddings to GPT-6 Astra for RAG-Style Answers
Retrieval only gets you the right passages. To turn those passages into a written answer, feed them into a chat model through the Responses API, the interface OpenAI now recommends for all new integrations, including GPT-6 Astra, GPT-6 Sol, and GPT-6 Luna.
top_matches = search("How do I reset my password?", chunks, vectors)
context = "\n\n".join(text for score, text in top_matches)
answer = client.responses.create(
model="gpt-6-astra",
input=f"Answer the question using only this context:\n\n{context}\n\nQuestion: How do I reset my password?"
)
print(answer.output_text)For high-volume, low-latency use cases like autocomplete or ticket triage, swap in gpt-6-luna, OpenAI's efficient model built for repeatable work at scale, instead of the flagship Astra model. Reserve gpt-6-sol for cases that need heavier multi-step reasoning over the retrieved context.
Steps 12-13: Managed Retrieval and Production Hardening
Step 12: Try the Managed Alternative: OpenAI Vector Stores and File Search
Not every project needs to manage its own vector database. OpenAI's vector stores, paired with the file_search tool inside the Responses API, hand OpenAI the chunking, embedding, and indexing work in exchange for less control over the internals.
vector_store = client.vector_stores.create(name="support-docs")
client.vector_stores.files.upload_and_poll(
vector_store_id=vector_store.id,
file=open("faq.pdf", "rb")
)
response = client.responses.create(
model="gpt-6-astra",
input="What is the refund policy?",
tools=[{"type": "file_search", "vector_store_ids": [vector_store.id]}]
)Use this route for internal tools, prototypes, and small-to-medium document sets where engineering time is scarcer than the extra managed-service cost. Reach for a self-hosted embeddings-plus-database pipeline when you need custom chunking logic, hybrid keyword-plus-vector search, or fine-grained control over exactly what gets retrieved.
Step 13: Add Caching, Retries, and Rate Limit Handling
Rate limits are account- and tier-specific, and they change as usage grows, so hardcoding a static requests-per-minute number is fragile. Build in resilience instead:
import time
from openai import RateLimitError
def embed_with_retry(texts, model="text-embedding-3-small", max_retries=5):
for attempt in range(max_retries):
try:
return client.embeddings.create(model=model, input=texts)
except RateLimitError:
wait = 2 ** attempt
time.sleep(wait)
raise RuntimeError("Exceeded retry limit for embeddings request")Cache embeddings for content that doesn't change. A hash of the input text as the cache key works well, so you never pay to re-embed the same paragraph twice. For a support knowledge base that updates weekly, this alone can cut embedding spend by well over half.
Complete Working Project: End-to-End Semantic Search Script
Putting every step together into one runnable file gives you a working semantic search tool you can point at any folder of text files.
import os, glob, time
import numpy as np
from openai import OpenAI, RateLimitError
client = OpenAI()
MODEL = "text-embedding-3-small"
def chunk_text(text, max_words=300):
words = text.split()
return [" ".join(words[i:i+max_words]) for i in range(0, len(words), max_words)]
def embed_with_retry(texts, max_retries=5):
for attempt in range(max_retries):
try:
return client.embeddings.create(model=MODEL, input=texts)
except RateLimitError:
time.sleep(2 ** attempt)
raise RuntimeError("Rate limit retries exhausted")
def build_index(folder):
chunks, vectors = [], []
for path in glob.glob(f"{folder}/*.txt"):
with open(path) as f:
for chunk in chunk_text(f.read()):
chunks.append(chunk)
resp = embed_with_retry(chunks)
vectors = [d.embedding for d in resp.data]
return chunks, vectors
def cosine(a, b):
a, b = np.array(a), np.array(b)
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def search(query, chunks, vectors, top_k=3):
q_vec = embed_with_retry([query]).data[0].embedding
scored = sorted(
((cosine(q_vec, v), t) for v, t in zip(vectors, chunks)),
reverse=True
)
return scored[:top_k]
def answer(query, chunks, vectors):
matches = search(query, chunks, vectors)
context = "\n\n".join(t for _, t in matches)
resp = client.responses.create(
model="gpt-6-astra",
input=f"Using only this context, answer the question.\n\nContext:\n{context}\n\nQuestion: {query}"
)
return resp.output_text
if __name__ == "__main__":
chunks, vectors = build_index("./docs")
print(answer("How do I reset my password?", chunks, vectors))Drop a folder of .txt files into ./docs, run the script, and you get a command-line semantic search and question-answering tool in roughly 50 lines. Swap the in-memory cosine loop for pgvector or Qdrant once your corpus grows past a few thousand chunks.
text-embedding-3-small vs text-embedding-3-large vs Gemini Embedding 2 vs Voyage 3
OpenAI isn't the only option, and picking a provider is worth a five-minute comparison before you commit a whole pipeline to one vendor.
| Model | Provider | Price per 1M tokens | Max dimensions | Notable trait |
|---|---|---|---|---|
| text-embedding-3-small | OpenAI | $0.02 | 1,536 | Best default for cost-sensitive production |
| text-embedding-3-large | OpenAI | $0.13 | 3,072 | Higher reported MTEB score, more storage |
| Gemini Embedding 2 | ~$0.20 (released March 2026) | Not fixed, multimodal | Native text, image, video, and audio embeddings | |
| voyage-3 / voyage-3-large | Voyage AI | $0.06-$0.18 | Varies by model | Tuned for retrieval-heavy RAG workloads |
For a text-only application already on the OpenAI API, sticking with text-embedding-3-small keeps your stack simple and your bill predictable. Gemini Embedding 2's multimodal reach makes sense if you're indexing images or audio alongside text and don't want to run a separate pipeline for each media type. Voyage AI is worth a benchmark run if retrieval quality on your specific domain, such as legal text, code, or medical documents, matters more than raw price. Cohere's embed-v3 line is another name that comes up in comparisons, though pricing and dimension specifics are less consistently documented across sources than the four models above, so treat any specific numbers for it as worth double-checking against Cohere's current pricing page before you build around them.
Switching providers mid-project is more disruptive than switching between text-embedding-3-small and text-embedding-3-large, since every vendor's embedding space is different. A Gemini Embedding 2 vector and an OpenAI text-embedding-3-small vector are not comparable to each other, even if both happen to be normalized. If you want to keep the option open to switch providers later without a full rebuild, store your original source text alongside the vectors (not just the vectors themselves) so re-embedding with a different provider only requires reprocessing text you already have on hand, not requesting it all over again.
Choosing a Vector Database for Your Embeddings
| Database | Best for | Deployment | Metadata filtering |
|---|---|---|---|
| pgvector | Apps already on PostgreSQL | Self-hosted extension | Full SQL WHERE clauses |
| Pinecone | Managed production at scale, minimal ops | Fully managed cloud | Metadata filters |
| Qdrant | Open-source flexibility with strong filtering | Self-hosted or managed cloud | Payload-based filters |
| Milvus | Very large, distributed collections | Self-hosted cluster or managed | Scalar field filters |
Don't over-engineer this decision on day one. A single Postgres instance with pgvector comfortably handles collections up to a few million vectors for most applications, and you can migrate to a dedicated vector database later if query latency or scale genuinely demands it. Premature adoption of a distributed system like Milvus for a 10,000-document knowledge base adds operational overhead with no measurable benefit.
The migration path between these options is generally smoother than teams expect, since the underlying operation, nearest-neighbor search over a fixed-dimension vector, is conceptually the same everywhere. What changes is the query syntax, the indexing configuration, and how metadata filters get expressed. If you build your retrieval layer behind a small internal interface (a search(query, filters) function, for example) rather than calling pgvector's SQL directly from a dozen places in your codebase, swapping the backend later becomes a contained change instead of a rewrite.
Evaluating Retrieval Quality Before You Ship
It's tempting to run one or two test queries, see reasonable-looking results, and call the pipeline done. That approach reliably breaks in production once real users start typing queries that don't resemble the examples you tested with. Before shipping, build a small labeled evaluation set: 20-50 realistic questions paired with the document or chunk that should be retrieved for each one. This doesn't need to be elaborate. A spreadsheet with a query column and an expected-source column is enough to start.
eval_set = [
{"query": "How do I reset my password?", "expected_source": "faq.txt#password"},
{"query": "What's your refund window?", "expected_source": "policy.txt#refunds"},
]
def evaluate(eval_set, chunks, vectors, sources, top_k=3):
hits = 0
for case in eval_set:
results = search(case["query"], chunks, vectors, top_k=top_k)
retrieved_sources = [sources[chunks.index(text)] for _, text in results]
if case["expected_source"] in retrieved_sources:
hits += 1
return hits / len(eval_set)This gives you a single number, recall at top_k, that you can track as you change chunk size, switch between text-embedding-3-small and text-embedding-3-large, or adjust the dimensions parameter. Without this baseline, "we upgraded the embedding model" is a guess about improvement instead of a measured one, and teams routinely pay for the larger, more expensive model without ever confirming it actually helped their specific queries. Re-run the same eval set every time you touch the retrieval pipeline, and treat any drop in the score as a regression worth investigating before it reaches users.
Common Pitfalls When Building With OpenAI Embeddings
Most embeddings pipelines fail quietly rather than loudly. A search that returns technically valid but irrelevant results doesn't throw an exception, but it does erode trust in the feature over time. These are the mistakes that show up most often in code review and in production incident reports.
- Mixing models or dimension sizes within one collection. Cosine similarity between a 1,536-dimension vector and a 3,072-dimension vector will either error out or produce meaningless scores. Pick one model and one dimension setting per collection and never change it without a full re-index.
- Embedding raw HTML or markdown without stripping tags. Boilerplate like nav menus and footer links pollutes the embedding and drags irrelevant results into your top matches. Extract clean text before chunking.
- Chunking purely by character count instead of tokens. A 2,000-character chunk can be well under or well over the token budget depending on the language and content, leading to silent truncation. Use a tokenizer like tiktoken to chunk by actual token count.
- Skipping normalization when computing similarity manually. If you compute dot products instead of true cosine similarity and forget to normalize vectors, longer documents can score artificially higher regardless of relevance.
- Re-embedding unchanged content on every pipeline run. Without a content hash or last-modified check, teams routinely re-embed their entire corpus nightly, burning budget on documents that haven't changed at all.
- Assuming a higher-dimension model always wins. Teams frequently default to text-embedding-3-large "to be safe," pay 6.5x more, and never run the comparison that would show the small model performs identically for their specific retrieval task.
Troubleshooting Guide
These are the specific errors and symptoms that come up most often once a pipeline moves from a local script to something handling real traffic. Work through this list before assuming the problem is with the model itself. In the overwhelming majority of cases, it's a configuration mismatch somewhere in the surrounding plumbing.
- "RateLimitError: 429 Too Many Requests": you're exceeding your tier's tokens-per-minute or requests-per-minute limit. Implement exponential backoff (shown in Step 13) and batch multiple texts per request instead of sending them one at a time.
- Embeddings return but cosine similarity scores all look nearly identical: check whether you're comparing vectors from two different models or two different dimension settings. The comparison is invalid across mismatched vector spaces.
- "InvalidRequestError: This model's maximum context length is 8191 tokens": your chunk exceeded the token limit. Reduce your chunk size in the chunking function and verify you're counting tokens, not words or characters.
- pgvector index queries are slow despite an HNSW index: run
ANALYZEon the table after bulk inserts, and confirm your query actually uses thevector_cosine_opsoperator class matching how the index was built. - Qdrant collection returns zero results for an obviously relevant query: verify the collection's distance metric matches how you're querying. A collection created with Euclidean distance will not behave like cosine similarity.
- Costs are higher than expected on the OpenAI usage dashboard: check whether a bug is re-embedding the same documents on every run, and confirm you're not accidentally calling text-embedding-3-large everywhere due to a leftover default in a config file.
- Search results seem semantically "close but wrong": this often means chunks are too large and blend multiple unrelated topics into one embedding. Reduce chunk size and increase overlap slightly.
- GPT-6 Astra answers hallucinate details not present in the retrieved context: tighten your prompt to explicitly instruct the model to say "I don't know" when the context doesn't contain the answer, and confirm your top_k value is actually retrieving relevant passages.
- Batch API job stays in "in_progress" far longer than expected: Batch jobs process asynchronously within a 24-hour window by design. For anything latency-sensitive, use the synchronous embeddings endpoint instead.
Advanced Tips for Production Semantic Search
Once the basic pipeline works, a few refinements separate a demo from a production system. First, combine vector search with traditional keyword search (hybrid search): pgvector and Qdrant both support this, and it catches exact-match queries like part numbers or error codes that pure semantic search sometimes misses. Second, add a lightweight reranking step: retrieve a larger top_k (say 20) with embeddings, then use a chat model to rerank the top candidates against the exact query before generating an answer. This costs a little extra latency but noticeably improves answer quality on ambiguous queries.
Third, monitor embedding drift over time. If you periodically re-embed your corpus with an updated chunking strategy or switch models, keep the old and new vector collections separate during a transition period so you can A/B test retrieval quality before fully cutting over. Fourth, set a hard monthly budget alert in the OpenAI dashboard, since embedding costs scale with document volume and update frequency, and a runaway indexing job on a large document set can produce a surprising bill overnight if left unchecked. Finally, log which chunks get retrieved most often, since that data tells you which parts of your knowledge base are actually useful and which ones can be trimmed or archived, keeping your index leaner and your searches faster.
One more production concern worth building in early rather than retrofitting later: redact or mask sensitive data (customer names, account numbers, API keys) before it ever reaches the embeddings endpoint, especially if the content comes from user-generated support tickets or emails. Embeddings are sent over the network to OpenAI's servers like any other API call, and while OpenAI's data-usage policies for API traffic are documented separately from consumer ChatGPT products, treating the embeddings pipeline the same way you'd treat any third-party API that touches customer data is the safer default. A simple regex-based scrubber for emails, phone numbers, and obvious identifiers running before the chunking step closes most of that gap without much engineering effort.
Frequently Asked Questions
What is the difference between the OpenAI embeddings API and the Responses API?
The embeddings endpoint (/v1/embeddings) converts text into numeric vectors for search and comparison. The Responses API generates natural-language answers using models like GPT-6 Astra. In a RAG pipeline, embeddings handle retrieval and the Responses API handles generation, and the two are complementary rather than competing tools.
Is text-embedding-3-small good enough for production, or do I need the large model?
For most applications, yes. Third-party MTEB comparisons put it only a few points below text-embedding-3-large while costing roughly 6.5x less per token. Start with the small model and only upgrade if your own evaluation on real queries shows a measurable quality gap.
Can I reduce embedding costs for a large existing document set?
Yes. Use the Batch API for one-time bulk indexing jobs, cache embeddings by content hash so unchanged documents are never re-embedded, and default to text-embedding-3-small unless you have specific evidence the large model performs better on your data.
Do I need a vector database, or can I just store embeddings in a regular table?
For a few thousand documents, storing vectors as JSON or an array column and computing similarity in Python works fine. Once you cross tens of thousands of vectors, an indexed vector database like pgvector, Pinecone, or Qdrant becomes necessary for acceptable query speed, since brute-force comparison scales linearly with corpus size.
What happens if I change the dimensions parameter after I've already indexed data?
You must re-embed and re-index your entire collection. Vectors of different sizes or from different models are not comparable, so a mixed collection will silently produce broken or nonsensical search results rather than throwing an obvious error in every case.
Should I use OpenAI's managed vector stores or build my own with pgvector or Qdrant?
Managed vector stores paired with the file_search tool are faster to set up and require no infrastructure, making them a good fit for internal tools and prototypes. Self-hosted pgvector or Qdrant gives you full control over chunking, hybrid search, and metadata filtering, which matters more as an application matures and requirements get specific.
Are OpenAI's rate limits the same for every account?
No. Rate limits are tied to your usage tier, which is determined by billing history and account age, and they can change as your usage grows. Rather than hardcoding assumed limits, build retry logic with exponential backoff so your pipeline adapts automatically to whatever tier your account is on.
Can I use OpenAI embeddings with a non-OpenAI chat model for the generation step?
Yes. Nothing ties embeddings to a specific generation model. You can embed with text-embedding-3-small and pass retrieved context to any chat model that accepts text input. Many teams do mix providers this way to optimize cost or latency independently at each stage of the pipeline.
How do I decide how many chunks to retrieve for each query?
There's no universal number, since it depends on how densely your documents pack information and how much context your generation model can use effectively. A common starting point is retrieving 3-5 chunks for narrow factual questions and 8-10 for broader questions that likely span multiple documents. Use the evaluation set described earlier to measure recall at different top_k values and pick the smallest number that still reliably surfaces the right source, since every extra chunk adds tokens, cost, and a small amount of noise to the final generation step.
Related Coverage
- Set Up Claude Fable 5.1 API: 12 Steps, 90 Min [2026]
- Veo 3.1 API Setup: Migrate From Sora 2 in 12 Steps [2026]
- Grok 4.7 API Setup: 12 Steps, $2/$6 per 1M Tokens [2026]
- Hy Image 3.5 API Setup: 12 Steps, $0.024/Image [2026]
- Computer-Use AI Agent Setup: 13 Steps, 90 Min [2026]
