Retrieval-Augmented Generation (RAG) has become one of the most common approaches for building AI applications that need to work with external or private knowledge. Instead of asking an LLM to answer a question entirely from its training data, a RAG system retrieves relevant information from a document collection and provides that information to the model as context.
However, the quality of a RAG application depends heavily on retrieval. If the retriever fails to identify the right documents, even a highly capable LLM may generate an incomplete or incorrect answer. One reason retrieval can fail is the semantic difference between a user’s question and the documents that contain the answer. Users typically ask questions in conversational language, while knowledge bases often contain technical explanations, formal terminology, procedures, and domain-specific concepts.
Hypothetical Document Embeddings (HyDE) provides one approach to this problem. Instead of embedding the user’s question directly, HyDE first asks an LLM to generate a hypothetical document that could answer the question. That generated document is then converted into an embedding and used to retrieve relevant documents from the actual knowledge base.
The hypothetical document is not treated as factual evidence. It is primarily a retrieval mechanism. The final answer should still be grounded in the real documents retrieved from the knowledge base.
1. What Is Hypothetical Document Embeddings (HyDE)?
Hypothetical Document Embeddings (HyDE) is a technique for improving dense document retrieval by using a language model to generate a hypothetical document from a user’s query before performing the search.
The technique was introduced in the research paper “Precise Zero-Shot Dense Retrieval without Relevance Labels” by Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. The paper proposes using an instruction-following language model to generate a hypothetical document that represents a potential answer to the query. This generated document is then encoded into an embedding using a contrastively trained encoder, and the resulting embedding is used to retrieve similar documents from a corpus.
An important aspect of HyDE is that the generated document does not need to be factually correct. Its purpose is to provide a richer semantic representation of the query that can be used for retrieval. The actual documents retrieved from the knowledge base are then used as the source of information for the downstream application.
In this way, HyDE adds an intermediate generation step between the user’s query and the document retrieval process, allowing the system to search using a hypothetical document rather than the original query alone.
2. Why HyDE Is Needed
Traditional dense retrieval relies on the embedding of the user’s query to find semantically similar documents. While this approach can work well, retrieval quality can decrease when the query is short, vague, or expressed differently from the language used in the document collection.
This creates a gap between the user’s query and the content of the documents being searched. A user may express an information need as a question, while the relevant documents may contain detailed explanations, technical terminology, or domain-specific language. As a result, the query embedding may not always provide enough semantic information to identify the most relevant documents.
HyDE addresses this problem by using an instruction-following language model to generate a hypothetical document that could answer the user’s query. The generated document provides additional context and terminology that can create a richer representation of the user’s information need before the retrieval step.
HyDE is especially relevant for zero-shot retrieval, where there are no relevance labels available to train or fine-tune a specialized retriever. The original HyDE research showed that generating a hypothetical document and encoding it with a contrastively trained encoder could improve retrieval performance across several information-retrieval tasks.
The main reason for using HyDE is to improve the representation used during retrieval. Instead of relying only on the wording of the user’s query, HyDE generates a document-like representation that can provide a stronger semantic signal for finding relevant documents.
3. How HyDE Works
HyDE adds a document-generation step before the normal embedding and retrieval process.
The process begins when the user submits a question. The question is sent to an LLM, which generates a hypothetical document that could answer it.
The hypothetical document is then passed to an embedding model. The resulting embedding is used to search the vector database containing the embeddings of the application’s real documents.
The vector database returns the most relevant documents. These documents, together with the original user question, are then provided to an LLM to generate the final response.
The important point is that HyDE uses the hypothetical document for retrieval, not as the source of truth for the final answer.
4. HyDE vs. Traditional RAG
| Aspect | Traditional RAG | HyDE |
|---|---|---|
| Initial input | User query | User query |
| Retrieval process | Query is embedded directly | LLM first generates a hypothetical document |
| What is embedded? | User query | Hypothetical document |
| Vector search | Searches using the query embedding | Searches using the hypothetical document embedding |
| Extra LLM call | Not required before retrieval | Required to generate the hypothetical document |
| Main benefit | Simple and efficient retrieval | Can improve retrieval when query and document language differ |
| Main drawback | May struggle with short or differently worded queries | Adds latency and LLM cost |
| Final answer | Generated from retrieved documents | Generated from retrieved documents |
5. How HyDE Handles Hallucination
A natural concern with HyDE is that the LLM-generated document may contain incorrect information. This is possible because the hypothetical document is generated rather than retrieved from a trusted knowledge source. However, the generated document is not automatically treated as factual evidence. Its primary purpose is to create an embedding that represents the user’s information need.
After generation, the hypothetical document passes through the embedding and retrieval stages. The resulting search retrieves actual documents from the application’s knowledge base. Those documents can then be used as the context for the final answer. This separation is important. An inaccurate statement in the hypothetical document does not necessarily appear in the final response because the hypothetical document itself is not the source being presented to the user.
Hallucination can still affect retrieval if the generated content introduces irrelevant concepts. HyDE therefore does not eliminate the risks associated with hallucination. Instead, it places the generated content in an intermediate role where the retrieval system uses it to locate information from the real document collection.
6. Production Guardrails
HyDE adds an LLM to the retrieval path, which introduces additional engineering concerns around reliability, latency, privacy, and observability. A production implementation should include safeguards that allow the system to fail gracefully and make retrieval behavior easier to monitor and troubleshoot.
Set Timeouts and Use Retrieval Fallbacks
Hypothetical document generation should have an explicit timeout. If the LLM is slow, unavailable, or returns an error, the application should avoid blocking the user’s request. A practical fallback is to return to traditional retrieval by embedding the original query and searching the vector database directly. This allows the application to continue serving requests even when HyDE generation is unavailable.
The timeout should be configured at the LLM client level so that slow or failed requests are terminated predictably and HyDE does not become a single point of failure.
Keep Hypothetical Documents Concise
The hypothetical document should contain enough information to represent the user’s query without introducing unnecessary detail. Longer generated responses can introduce unrelated concepts, add noise to the embedding, and reduce retrieval precision.
Set a maximum output length for the LLM generation step. For technical documentation, a relatively short response is often sufficient because the purpose is to improve retrieval rather than produce a complete answer. Keeping the generated content focused also helps reduce token consumption, latency, and the additional cost associated with HyDE.
Sanitize Sensitive Information Before LLM Processing
User queries may contain personal or confidential information that should not be sent directly to an external model provider. Sensitive data should therefore be identified and removed or replaced with appropriate placeholders before hypothetical document generation.
This protection should be enforced within the HyDE interface itself rather than relying on individual callers to sanitize their input correctly. This creates a consistent privacy boundary around the generation step.
Applications handling regulated or confidential data may also require additional controls based on their specific requirements and the policies of the selected model provider.
Add Observability Across the Retrieval Pipeline
HyDE introduces several stages that can affect retrieval quality, including hypothetical document generation, embedding, and vector search. Without visibility into these stages, it can be difficult to determine why retrieval results are poor or why requests are taking longer than expected.
Production systems should capture structured telemetry such as the original query, prompt version, generated hypothetical document, generation and embedding latency, retrieved document identifiers, and similarity scores.
This information can support latency monitoring, retrieval evaluation, troubleshooting, and detection of changes in system behavior over time. Logging should also follow appropriate privacy controls, particularly when queries or generated content may contain sensitive information.
These safeguards help ensure that HyDE remains a useful retrieval enhancement without introducing unnecessary reliability, privacy, cost, or monitoring challenges.
7. A Minimal HyDE Implementation Using Java
The easiest way to understand HyDE is to first look at a conventional RAG retrieval process and then modify the retrieval step to introduce a hypothetical document.
The example below uses Quarkus and LangChain4j to demonstrate the implementation. HyDE itself is not tied to either technology. The same pattern can be implemented with other Java frameworks, LLM providers, embedding models, and vector databases.
Normal RAG Retrieval
In a conventional RAG pipeline, the user’s query is embedded directly and the resulting vector is used to search the document collection. The retrieved documents are then passed to the LLM as context.
The following example uses LangChain4j’s EmbeddingModel and InMemoryEmbeddingStore. The in-memory store keeps the example simple; a production application can replace it with a supported vector store such as PGVector, Redis, Qdrant, or another database.
public class NormalRetriever {
@Inject
EmbeddingModel embeddingModel;
private final EmbeddingStore<TextSegment> embeddingStore = new InMemoryEmbeddingStore<>();
public void indexDocuments() {
List<String> documents = List.of("Database connection pools allow applications to reuse existing "
+ "database connections instead of creating a new connection for "
+ "every request.",
"Java virtual threads allow applications to handle large numbers "
+ "of concurrent tasks with lightweight threads managed by the JVM.",
"HTTP connection keep-alive allows multiple requests to reuse "
+ "the same network connection instead of establishing a new one.",
"Caching frequently accessed data can reduce repeated database "
+ "queries and improve application response times.",
"Message queues allow applications to process work asynchronously "
+ "and reduce the amount of work performed during a synchronous request.");
for (String document : documents) {
TextSegment segment = TextSegment.from(document);
Embedding embedding = embeddingModel.embed(segment).content();
embeddingStore.add(embedding, segment);
}
}
public List<EmbeddingMatch<TextSegment>> retrieve(String query) {
Embedding queryEmbedding = embeddingModel.embed(query).content();
return embeddingStore.search(dev.langchain4j.store.embedding.EmbeddingSearchRequest.builder().queryEmbedding(queryEmbedding).maxResults(2).build()).matches();
}
}
The important part of this implementation is the retrieval path. The application takes the user’s question, creates an embedding directly from that question, and uses the embedding to search the vector store.
For example, the application could receive the query: Why does my application become slow when many users connect at the same time?
The query itself is converted into an embedding, and that embedding is used to search the vector store. This represents the standard retrieval pattern that HyDE modifies.
Adding HyDE to the Retrieval Step
HyDE keeps the same document collection, embedding model, and vector store, but changes what is sent to the embedding model. Instead of embedding the user’s query directly, the application first asks an LLM to generate a short hypothetical passage that could answer the question. The generated passage is then embedded and used for the vector search.
With Quarkus LangChain4j, the hypothetical-document generation can be defined as an AI service. Quarkus generates the implementation for an interface annotated with @RegisterAiService, while @SystemMessage and @UserMessage define the instructions and dynamic input sent to the model.
@RegisterAiService
@SystemMessage("""
Write a short passage from technical documentation that could answer the user's question.
Use precise and factual technical language.
Do not include the question itself. Return only the passage.
""")
public interface HypotheticalDocumentGenerator {
@UserMessage("{query}")
String generate(String query);
}
The retrieval service can then use the generated passage instead of the original query when creating the search embedding:
public class HydeRetriever {
@Inject
EmbeddingModel embeddingModel;
@Inject
HypotheticalDocumentGenerator generator;
private final EmbeddingStore<TextSegment> embeddingStore = new InMemoryEmbeddingStore<>();
public List<EmbeddingMatch<TextSegment>> retrieve(String query) {
// Generate a hypothetical document
String hypothetical = generator.generate(query);
// Embed the hypothetical document
Embedding searchEmbedding = embeddingModel.embed(hypothetical).content();
// Search the real document collection
return embeddingStore.search(
EmbeddingSearchRequest.builder()
.queryEmbedding(searchEmbedding)
.maxResults(2)
.build())
.matches();
}
}
The key difference between the two implementations is what gets embedded. In normal RAG, the embedding model receives the user’s query. With HyDE, the LLM first transforms the query into a hypothetical document, and the embedding model receives that generated document instead.
The documents stored in the vector database do not need to change. HyDE changes the representation used to search those documents.
After the HyDE retriever returns the relevant documents, the application can provide those documents and the original query to an LLM to generate the final response. The hypothetical document remains an intermediate retrieval artifact. It helps locate relevant content but is not treated as the source of truth for the final answer.
In a production application, the InMemoryEmbeddingStore would typically be replaced with a persistent vector database. The documents would be processed during ingestion, converted into embeddings, and stored before users begin querying the system. The same architecture can also be implemented without Quarkus or LangChain4j. These technologies are used here simply to demonstrate how the HyDE concept can fit into a Java application.
8. Conclusion
HyDE improves the retrieval stage of a RAG system by generating a hypothetical document before performing vector search. Instead of embedding the user’s question directly, the system embeds the generated document and uses it to find relevant information in the knowledge base. This approach can be useful when users and documents use different terminology or when user queries are too short to provide enough semantic context for effective retrieval.
