LangChain4j RAG in Java: Embedding, Chunking, and Retrieval Without the Python Envy

Eleftheria DrosopoulouAugust 7th, 2026Last Updated: August 1st, 2026

0 477 4 minutes read

A practical pipeline for document ingestion, embedding, and retrieval, all on the JVM, with a Spring AI comparison for good measure.

For a long time, if you wanted to build a Retrieval-Augmented Generation pipeline, the unspoken rule was simple: write it in Python. LangChain, LlamaIndex, and a dozen notebooks later, you had something working. Meanwhile, Java developers watched from the sidelines, translating tutorials in their heads and wondering why an enterprise-grade language needed a sidecar script just to talk to a vector store.

That gap has closed. LangChain4j now gives Java the same building blocks Python developers have had for years, and it does so without leaving the JVM, without spinning up a Python process, and without the dependency headaches that come with mixing runtimes. Let’s walk through what a real pipeline looks like, end to end.

Why This Matters for Java Shops

Most production systems that would benefit from RAG already run on Java. Insurance platforms, banking backends, logistics engines, they are Spring Boot or Jakarta EE applications with years of operational maturity behind them. Bolting on a Python microservice just to do embeddings introduces a new deployment target, a new runtime to patch, and a new network hop between your core system and the thing answering user questions. Keeping the pipeline in Java removes all three problems at once.

The Four Stages of a LangChain4j RAG Pipeline

Regardless of which framework you pick, a RAG pipeline breaks down into the same four moves: load the documents, split them into manageable pieces, turn those pieces into vectors, and retrieve the right ones when a question comes in.

1. Document Ingestion

LangChain4j ships loaders for the file system, and separate modules exist for PDFs, URLs, and cloud storage such as Amazon S3. For most teams starting out, FileSystemDocumentLoader covers plain text and parsed PDFs, and it recurses through directories if you point it at a folder instead of a single file.

2. Chunking Strategy

This is the step everyone underestimates. Feed the model whole documents and you blow through context limits and dilute relevance. Chunk too aggressively and you sever the sentence that actually answers the question. LangChain4j’s recursive splitter tries to break on paragraph and sentence boundaries first, only falling back to a hard character cut when it has no choice, and it supports an overlap so that context is not lost at the edges of a chunk.

3. Embedding Without Leaving the JVM

This is the part that used to require a Python bridge. LangChain4j bundles ONNX-based embedding models that run in-process, on the JVM, with no external API call and no GPU required for the smaller ones. AllMiniLmL6V2EmbeddingModel is the workhorse default, and it needs nothing more than a Maven dependency to work offline.

4. Retrieval and Prompt Injection

Once your segments are embedded and sitting in a vector store, retrieval is a similarity search: take the user’s question, embed it with the same model, and pull back the closest matching chunks. LangChain4j’s AiServices builder wires a ContentRetriever straight into your chat interface, so the retrieved context gets injected into the prompt automatically, without you hand-assembling strings.

Here is what that looks like end to end, using an in-memory store so you can run it locally with no external database:

// 1. Load a document from disk
Document document = FileSystemDocumentLoader.loadDocument(
        Path.of("docs/product-manual.txt"),
        new TextDocumentParser());

// 2. Split it into overlapping chunks (500 chars, 50-char overlap)
DocumentSplitter splitter = DocumentSplitters.recursive(500, 50);

// 3. Pick a local, in-process embedding model (no API key needed)
EmbeddingModel embeddingModel = new AllMiniLmL6V2EmbeddingModel();

// 4. Ingest into an in-memory vector store
EmbeddingStore<TextSegment> store = new InMemoryEmbeddingStore<>();
EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder()
        .documentSplitter(splitter)
        .embeddingModel(embeddingModel)
        .embeddingStore(store)
        .build();
ingestor.ingest(document);

// 5. Wire retrieval into a chat assistant
ContentRetriever retriever = EmbeddingStoreContentRetriever.builder()
        .embeddingStore(store)
        .embeddingModel(embeddingModel)
        .maxResults(3)
        .build();

Assistant assistant = AiServices.builder(Assistant.class)
        .chatModel(chatModel)
        .contentRetriever(retriever)
        .build();

String answer = assistant.chat("What does the warranty cover?");

Five steps, one dependency tree, zero Python. The Maven coordinates you need are langchain4j, langchain4j-embeddings-all-minilm-l6-v2, and whichever chat model integration you use, all documented in the official RAG tutorial.

Local, in-process embedding models bundled with LangChain4j’s easy-rag module. Parameter counts from the models’ published cards.

How Spring AI Does the Same Job

If your codebase already leans heavily on Spring Boot, you might reach for Spring AI instead, and the outcome is nearly identical, just wired together with different vocabulary. Where LangChain4j has an EmbeddingStoreIngestor, Spring AI leans on its VectorStore abstraction directly. Where LangChain4j hands retrieval to a ContentRetriever attached to AiServices, Spring AI does it through a QuestionAnswerAdvisor attached to a ChatClient. The advisor intercepts your prompt, runs a similarity search behind the scenes, and augments the request before it ever reaches the model, so from the calling code’s point of view, RAG becomes a single line added to a builder chain.

Neither approach is more “correct” than the other. LangChain4j gives you a bit more explicit control over the ingestion pipeline out of the box, while Spring AI’s advisor pattern feels more natural if your team already thinks in terms of Spring interceptors and beans.

LangChain4j’s provider and vector store breadth, as listed in its official GitHub repository.

LangChain4j vs Spring AI, Side by Side

Concern	LangChain4j	Spring AI
Ingestion pipeline	`EmbeddingStoreIngestor` with pluggable splitter and transformer	Manual chunking (e.g. `TokenTextSplitter`) plus `VectorStore.add()`
Retrieval wiring	`ContentRetriever` injected into `AiServices`	`QuestionAnswerAdvisor` injected into `ChatClient`
Local embeddings	Bundled ONNX models, fully offline	Typically delegates to a provider API or Ollama
Framework coupling	Framework-agnostic, works in plain Java, Quarkus, Spring	Built for the Spring ecosystem
Best fit	Multi-module, non-Spring, or polyglot Java shops	Teams already standardized on Spring Boot

Before You Ship This Pipeline

Swap the in-memory store for a persistent one (Qdrant, PGVector, Chroma) before anything touches production traffic.
Tune chunk size and overlap against your actual documents, not the tutorial defaults.
Cap maxResults on retrieval deliberately; more chunks means a bigger, costlier prompt, not automatically a better answer.
Log which chunks were retrieved for each answer, it is the single best debugging tool you will have.
Re-embed everything if you ever change embedding models; old and new vectors are not comparable.

What We Learned

Java no longer needs to borrow Python’s tooling to build a working RAG pipeline. LangChain4j covers ingestion, chunking, in-process embedding, and retrieval with a handful of composable classes, and it runs entirely inside the JVM, including the embedding step, which used to be the hardest part to keep local. Spring AI reaches the same destination through advisors instead of ingestors, which will feel more natural if your stack is already Spring-native. Either way, the pattern is the same: load, split, embed, retrieve, inject, and the framework you pick should follow your existing architecture rather than the other way around.