Retrieval-Augmented Generation, commonly known as RAG, is one of the most useful patterns for building AI applications that need to answer questions using application-specific or private data. Instead of relying completely on the knowledge available inside a Large Language Model (LLM), a RAG application first retrieves relevant information from an external knowledge base and then provides that information to the model as context. Spring AI makes this pattern relatively easy to implement in a Spring Boot application. It provides abstractions for embedding models, vector stores, documents, text splitting, and chat models.
1. How RAG Works
Suppose an organization has internal documentation containing information about its products. A user might ask, “What is the refund policy for premium customers?” A normal LLM may not know the organization’s latest refund policy, so sending the question directly to the model can result in an incorrect, outdated, or generic response. With Retrieval-Augmented Generation (RAG), the application first searches its own documents for information relevant to the user’s question. The user’s question is converted into an embedding, which is then used to perform a vector similarity search against the documents stored in the vector database. Based on this search, the application retrieves the most relevant document chunks and combines them with the user’s original question to create additional context for the LLM. The LLM uses this retrieved context to generate a more accurate and grounded response, while citations can identify the documents from which the supporting information was retrieved. The important point is that the LLM does not directly search the vector database; instead, the Spring Boot application performs the retrieval, prepares the relevant context, sends it to the LLM, and returns the generated answer along with its citations.
1.1 Understanding the RAG Workflow
RAG stands for Retrieval-Augmented Generation. It combines information retrieval with text generation. A basic RAG implementation can be divided into two main phases: the indexing phase and the retrieval and generation phase. In the indexing phase, documents are prepared so that they can be searched efficiently. Large documents are first divided into smaller pieces called chunks instead of being stored as one large piece of text. For example, a document containing the statements “Premium customers can request refunds within 30 days. Standard customers can request refunds within 14 days. Refunds are returned to the original payment method.” could be divided into separate chunks containing each of these pieces of information. Smaller chunks generally improve retrieval because the application can locate the specific portion of a document that is relevant to the user’s question. After chunking, an embedding model converts every chunk into a numeric vector that represents its semantic meaning. For example, the text “Premium customers can request refunds within 30 days.” might be represented internally as a vector such as [0.018, -0.021, 0.043, …, 0.012]. Text with similar meanings tends to have vectors that are closer to each other in the embedding space. These vectors are then stored in a vector store together with the original chunk text and useful metadata such as the source file name and document title. For example, a chunk might contain metadata such as source “refund-policy.txt” and title “Refund Policy”. This metadata is particularly useful when the application needs to provide citations with its answers. In the retrieval and generation phase, when a user asks a question such as “What is the refund period for premium customers?”, the embedding model converts the question into a query embedding. The application performs a vector similarity search using this embedding to find document chunks whose meaning is most closely related to the question. The retrieved chunks are then included as context in the prompt sent to the chat model. For example, the prompt might instruct the model to answer using only the provided context and include the information “Premium customers can request refunds within 30 days.” along with the user’s question. The model uses this retrieved information to generate an answer that is grounded in the organization’s own data rather than relying entirely on the knowledge learned during the model’s training.
2. Building a RAG Application with Spring Boot and Spring AI
2.1 Add the Required Maven Dependencies
To build the RAG application, we need dependencies for creating REST APIs and integrating Spring AI with OpenAI. Add the following dependencies and Spring AI BOM configuration to the project’s pom.xml file.
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-bom</artifactId>
<version>${spring-ai.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
The spring-boot-starter-web dependency provides the components required to create REST endpoints in the Spring Boot application, while spring-ai-starter-model-openai adds Spring AI’s OpenAI integration for working with chat and embedding models. The spring-ai-bom is imported through dependencyManagement to manage compatible Spring AI dependency versions from a single place. The ${spring-ai.version} property should be defined in the properties section of the pom.xml, allowing the Spring AI version to be managed centrally without specifying a separate version for each Spring AI dependency.
2.2 Configure OpenAI Chat and Embedding Models
Next, configure the OpenAI API key and the models that will be used for chat completion and generating embeddings. Add the following configuration to the application.yml file.
spring:
application:
name: spring-ai-rag-demo
ai:
openai:
api-key: '${OPENAI_API_KEY}'
chat:
options:
model: gpt-4o-mini
temperature: 0
embedding:
options:
model: text-embedding-3-small
The api-key property reads the OpenAI API key from the OPENAI_API_KEY environment variable, which avoids hardcoding sensitive credentials in the application configuration. The chat.options.model property specifies the model used to generate answers from the retrieved context, while setting temperature to 0 makes the responses more deterministic. The embedding.options.model property specifies text-embedding-3-small as the embedding model used to convert document chunks and user queries into numerical vectors. These vectors are later used by the vector store to find document chunks that are semantically similar to the user’s query.
2.3 Create the Spring Boot Application
Now, create the main Spring Boot application class that serves as the entry point for the RAG application. The @SpringBootApplication annotation enables Spring Boot auto-configuration and component scanning.
package com.example.rag;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class RagApplication {
public static void main(String[] args) {
SpringApplication.run(RagApplication.class, args);
}
}
The RagApplication class starts the Spring Boot application through the SpringApplication.run() method. The @SpringBootApplication annotation combines configuration, auto-configuration, and component scanning, allowing Spring Boot to automatically discover components such as controllers, services, and configuration classes in the com.example.rag package and its subpackages. Once the application starts, Spring Boot also initializes the configured Spring AI components required for the RAG workflow.
2.4 Configure SimpleVectorStore and ChatClient
Next, configure the vector store used to store and search document embeddings and the ChatClient used to communicate with the chat model. To keep the example simple and focused on the RAG workflow, we will use Spring AI’s SimpleVectorStore.
package com.example.rag.config;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.vectorstore.SimpleVectorStore;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class AiConfig {
@Bean
public VectorStore vectorStore(EmbeddingModel embeddingModel) {
return SimpleVectorStore.builder(embeddingModel).build();
}
@Bean
public ChatClient chatClient(ChatModel chatModel) {
return ChatClient.builder(chatModel).build();
}
}
The AiConfig class defines two Spring beans required by the RAG application. The vectorStore() method creates a SimpleVectorStore using the configured EmbeddingModel. When documents are added to this vector store, the embedding model converts their text into numerical vectors, which can later be searched using similarity search. The chatClient() method creates a ChatClient using the configured ChatModel, providing a convenient API for sending the user’s question and retrieved document context to the LLM and receiving the generated answer. SimpleVectorStore is suitable for this demonstration because it requires no external vector database, although a persistent vector store such as PGVector would normally be preferred for a production application.
2.5 Create the Sample Knowledge Base
Next, add a couple of sample text documents that will act as the knowledge base for our RAG application. These documents contain the information that will later be split into chunks, converted into embeddings, stored in the vector store, and retrieved when a user asks a relevant question. Create src/main/resources/documents/refund-policy.txt with the following content:
Refund Policy Premium customers can request a full refund within 30 days of purchase. Standard customers can request a full refund within 14 days of purchase. Refunds are returned to the original payment method. After the applicable refund period has expired, purchases are normally non-refundable. Customers may contact support if there are exceptional circumstances.
The refund-policy.txt document contains information about refund periods, payment methods, and exceptions. When a user asks a refund-related question, the RAG application can retrieve relevant chunks from this document and provide them to the LLM as context. Now create src/main/resources/documents/shipping-policy.txt with the following content:
Shipping Policy Standard shipping normally takes 5 to 7 business days. Premium customers receive free express shipping. Express shipping normally takes 1 to 2 business days. Tracking information is sent to the customer's registered email address after the order has been dispatched.
The shipping-policy.txt document provides information about standard and express shipping, premium customer benefits, and order tracking. Keeping refund and shipping information in separate source files also makes citations more useful because the application can identify which document supplied the information used to answer a particular question. In the next step, these documents will be loaded, divided into smaller chunks, and stored in the vector store for semantic retrieval.
2.6 Load, Chunk, and Store Documents
Next, load the sample text files from the application’s resources and split their content into smaller chunks using Spring AI’s TokenTextSplitter. Each document is also assigned metadata so that the original source can be identified later when returning citations.
package com.example.rag.service;
import jakarta.annotation.PostConstruct;
import org.springframework.ai.document.Document;
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.core.io.ClassPathResource;
import org.springframework.stereotype.Service;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
@Service
public class DocumentLoader {
private final VectorStore vectorStore;
public DocumentLoader(VectorStore vectorStore) {
this.vectorStore = vectorStore;
}
@PostConstruct
public void loadDocuments() throws IOException {
List < Document > documents = new ArrayList < >();
documents.add(loadDocument("documents/refund-policy.txt", "refund-policy.txt", "Refund Policy"));
documents.add(loadDocument("documents/shipping-policy.txt", "shipping-policy.txt", "Shipping Policy"));
TokenTextSplitter splitter = new TokenTextSplitter(200, 100, 5, 1000, true);
List < Document > chunks = splitter.apply(documents);
vectorStore.add(chunks);
System.out.println("Stored " + chunks.size() + " chunks in vector store.");
}
private Document loadDocument(
String path, String source, String title) throws IOException {
ClassPathResource resource = new ClassPathResource(path);
String content = resource.getContentAsString(
StandardCharsets.UTF_8);
return new Document(
content, Map.of("source", source, "title", title));
}
}
The DocumentLoader service is responsible for preparing the application’s documents for retrieval. The @PostConstruct annotation causes the loadDocuments() method to run after the Spring bean has been initialized, so the knowledge base is prepared when the application starts. The loadDocument() method uses ClassPathResource to read each text file and creates a Spring AI Document containing both the document text and metadata such as source and title. The TokenTextSplitter then divides the documents into smaller chunks so that retrieval can target relevant sections instead of entire documents. Calling splitter.apply(documents) produces the list of chunks, while vectorStore.add(chunks) sends those chunks to the configured vector store, where the embedding model converts their text into numerical embeddings for similarity search. The metadata associated with each document is retained with its chunks and can later be used to create citations in the RAG response. Finally, the application prints the number of chunks stored in the vector store, which provides a simple confirmation that the document-loading process has completed.
2.7 Create Response and Citation Models
The REST API should return not only the answer generated by the LLM but also information about the documents used to produce that answer. For this purpose, create a simple Citation record to represent each retrieved source.
package com.example.rag.model;
public record Citation(String source, String title, String content) {}
The Citation record contains three fields: source identifies the original file, title provides a readable name for the document, and content contains the retrieved document chunk. These values allow the API consumer to see which source was used and inspect the supporting content behind the generated answer. Next, create RagResponse.java to represent the complete response returned by the RAG API.
package com.example.rag.model;
import java.util.List;
public record RagResponse(String answer, List < Citation > citations) {}
The RagResponse record combines the LLM-generated answer with a list of Citation objects representing the document chunks retrieved from the vector store. This gives the client both the final response and the supporting sources in a structured format. Java records are a good fit for these DTOs because they provide a concise way to define immutable data objects without manually writing constructors, getters, equals(), hashCode(), or toString() methods.
2.8 Implement RAG Retrieval and Response Generation
Now we can implement the main RAG service that connects retrieval with response generation. The service searches the vector store for document chunks related to the user’s question, builds the retrieved context, sends that context to the LLM, and finally returns the generated answer together with citations.
package com.example.rag.service;
import com.example.rag.model.Citation;
import com.example.rag.model.RagResponse;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
@Service
public class RagService {
private final VectorStore vectorStore;
private final ChatClient chatClient;
public RagService(VectorStore vectorStore, ChatClient chatClient) {
this.vectorStore = vectorStore;
this.chatClient = chatClient;
}
public RagResponse ask(String question) {
// 1. Retrieve the most relevant document chunks.
List < Document > documents = vectorStore.similaritySearch(SearchRequest.builder().query(question).topK(3).build());
if (documents == null || documents.isEmpty()) {
return new RagResponse("I could not find relevant information.", List.of());
}
// 2. Build the context that will be supplied to the LLM.
String context = IntStream.range(0, documents.size())
.mapToObj(i -> {
Document document = documents.get(i);
return """
[Source %d]
Title: %s
File: %s
Content: %s
""".formatted(
i + 1,
document.getMetadata().getOrDefault("title", "Unknown"),
document.getMetadata().getOrDefault("source", "Unknown"),
document.getText());
})
.collect(Collectors.joining("\n\n"));
// 3. Ask the model to answer only from retrieved context.
String answer = chatClient.prompt()
.system("""
You are a question-answering assistant.
Answer the user's question using only the
supplied context.
If the context does not contain enough
information, say that you do not have enough
information.
Cite supporting sources using [Source 1],
[Source 2], etc.
Do not invent sources.
""")
.user("""
Context: %s
Question: %s
""".formatted(context, question))
.call()
.content();
// 4. Convert retrieved documents into citations.
List < Citation > citations = documents.stream()
.map(document -> new Citation(String.valueOf(document.getMetadata().getOrDefault("source", "Unknown")), String.valueOf(document.getMetadata().getOrDefault("title", "Unknown")), document.getText()))
.toList();
return new RagResponse(answer, citations);
}
}
The RagService contains the main retrieval and generation logic of the application. First, vectorStore.similaritySearch() searches the vector store using the user’s question, and topK(3) limits the result to the three most relevant document chunks. If no documents are found, the method returns a simple response indicating that relevant information could not be found. Next, the retrieved documents are combined into a context string, where each chunk is assigned a label such as [Source 1] and includes its title, source file, and content. This context is then supplied to the LLM through ChatClient together with the user’s original question. The system prompt instructs the model to answer only from the retrieved information, indicate when the available context is insufficient, and reference supporting information using source labels such as [Source 1]. After the answer is generated, the retrieved Document objects are converted into Citation objects using their source metadata, title, and chunk content. Finally, the method returns a RagResponse containing both the generated answer and the structured citations, allowing the client to display the response together with the documents that supported it.
2.9 Expose the RAG API with a REST Controller
Finally, create a REST controller to expose the RAG functionality through an HTTP endpoint. The controller accepts the user’s question as a request parameter and passes it to the RagService for retrieval and answer generation.
package com.example.rag.controller;
import com.example.rag.model.RagResponse;
import com.example.rag.service.RagService;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/api/rag")
public class RagController {
private final RagService ragService;
public RagController(RagService ragService) {
this.ragService = ragService;
}
@GetMapping("/ask")
public RagResponse ask(@RequestParam String question) {
return ragService.ask(question);
}
}
The RagController exposes the RAG service as a REST API. The @RestController annotation tells Spring that this class handles HTTP requests and that returned objects should be serialized directly into the response body, typically as JSON. The @RequestMapping("/api/rag") annotation defines the base URL for the controller, while @GetMapping("/ask") maps GET requests to the ask() method. The @RequestParam String question parameter reads the user’s question from the URL, for example /api/rag/ask?question=What is the refund policy for premium customers?. The controller then delegates the request to ragService.ask(question), where the vector search, context preparation, LLM call, and citation creation take place. The resulting RagResponse is automatically converted into JSON and returned to the client.
2.10 Run and Test the RAG Application
Now that all the required components are in place, we can run the Spring Boot application and test the RAG endpoint. Before starting the application, make sure the OPENAI_API_KEY environment variable is configured with a valid OpenAI API key.
export OPENAI_API_KEY=your-openai-api-key mvn spring-boot:run
When the application starts, the DocumentLoader loads the refund and shipping policy files, splits them into chunks, generates embeddings, and stores those chunks in the SimpleVectorStore. Depending on the document content and chunking configuration, the console will display output similar to the following.
Stored 2 chunks in vector store.
Once the application is running, send a request to the RAG endpoint with a question about the information available in the sample documents. For example, the following request asks about the refund policy for premium customers.
curl "http://localhost:8080/api/rag/ask?question=What%20is%20the%20refund%20policy%20for%20premium%20customers%3F"
The application performs a similarity search, retrieves the relevant refund-policy chunk, sends it to the LLM along with the question, and returns the generated answer with citations. A response may look similar to the following.
{
"answer": "Premium customers can request a full refund within 30 days of purchase. Refunds are returned to the original payment method. [Source 1]",
"citations": [
{
"source": "refund-policy.txt",
"title": "Refund Policy",
"content": "Refund Policy\n\nPremium customers can request a full refund within 30 days of purchase. Standard customers can request a full refund within 14 days of purchase. Refunds are returned to the original payment method."
}
]
}
The answer field contains the response generated by the LLM using the retrieved document context, while the citations array contains the actual document chunks retrieved from the vector store. In this example, [Source 1] in the generated answer corresponds to refund-policy.txt, allowing the client to identify the source supporting the answer. The exact wording of the generated response may vary between requests, but the answer should remain grounded in the information retrieved from the sample documents.
3. Conclusion
In this tutorial, we built a simple Retrieval-Augmented Generation (RAG) application using Spring Boot and Spring AI. We started by loading sample documents, splitting them into smaller chunks using TokenTextSplitter, and storing their embeddings in a SimpleVectorStore. When a user submits a question, the application performs a similarity search to retrieve the most relevant document chunks and supplies them as context to the LLM through Spring AI’s ChatClient. The generated answer is then returned together with citations containing the source document information and supporting content. This approach allows the LLM to generate responses based on application-specific data instead of relying only on its training knowledge. While SimpleVectorStore keeps this example easy to understand, the same basic RAG workflow can later be extended with a persistent vector database and larger document collections for real-world applications.
