Large Language Models can take several seconds, or sometimes much longer, to produce a complete response. In a traditional blocking application, the caller waits until the entire response has been generated before receiving anything. Streaming changes this interaction model. Instead of waiting for the complete LLM response, the application can consume information incrementally as the model produces it. The Embabel Agentic AI Framework brings this idea into the JVM and Spring ecosystem. More importantly, Embabel streaming is not restricted to raw text tokens. The framework can stream structured Java objects and, when supported by the underlying model/provider, reasoning or “thinking” events. Streaming can also participate in agentic tool execution.
1. Overview
Embabel is a JVM-based framework for building agentic AI applications using Java or Kotlin, with natural integration into the Spring ecosystem. It enables developers to combine Large Language Model (LLM) interactions with strongly typed Java or Kotlin domain objects and existing application code, making it easier to incorporate AI capabilities into conventional enterprise applications. Embabel also supports tools that allow LLM-powered agents to interact with application services and external systems, while its agent model provides actions and goals for defining what an agent can do and what it is expected to accomplish. In addition, dynamic planning allows the framework to determine an appropriate sequence of actions at runtime based on the available capabilities, application state, and desired goal. Together, these features provide a structured approach for building AI agents that can reason, invoke application functionality, work with domain data, and execute multi-step workflows within JVM and Spring-based applications. For streaming scenarios, some of the important Embabel abstractions are:
| Component | Purpose |
|---|---|
StreamingPromptRunnerBuilder | Builds and executes streaming prompts. |
StreamingEvent<T> | Represents an event containing either reasoning/thinking or a structured object. |
LlmMessageStreamer | Provider-neutral abstraction for one streaming LLM inference. |
StreamingToolLoop | Coordinates streaming inference with tool execution. |
Flux<T> | Project Reactor publisher used to consume values reactively as they arrive. |
1.1 What Is Structured Streaming?
Normal LLM streaming typically delivers a response incrementally as small pieces of text or tokens, which works well for conversational interfaces where the primary goal is to display generated text as quickly as possible. Enterprise applications, however, often need more than arbitrary text because their business logic operates on strongly typed domain objects such as products, customers, bookings, or transactions. Without structured streaming, an application may need to collect all streamed text, reconstruct the complete response, parse the generated JSON, and then convert it into Java objects before processing can begin. Structured streaming simplifies this process by allowing completed structured data to be converted into typed domain objects and emitted as application events as soon as they are available. For example, when an LLM generates information for multiple products containing fields such as name, category, and price, each completed product can be represented directly as a Java Product object and processed independently rather than waiting for the entire response to finish. This approach is particularly useful in Embabel because its programming model is centered around strongly typed domain objects, enabling streamed LLM output to integrate naturally with Java application logic instead of requiring applications to work primarily with unstructured strings.
1.1.1 Understanding StreamingEvent
For structured streaming with reasoning, Embabel represents the response as a reactive stream of events, where each event can contain either reasoning information produced during the LLM interaction or a completed structured domain object. This allows an application to distinguish between intermediate reasoning-related content and actual business data as the response is generated. When a reasoning event is received, the application can identify it as thinking content and, where the selected model and provider expose such information, process or observe that content separately. When an object event is received, it represents a completed domain object that has been successfully constructed from the model’s structured output and can immediately be passed to normal application logic. For example, while generating product recommendations, the stream may first contain reasoning-related events as the model evaluates the requirements and later emit completed Product objects containing fields such as name, category, and price. This event-based approach enables Embabel applications to consume reasoning and structured results through the same reactive stream while still keeping the two types of information clearly separated.
1.1.2 Structured Streaming: Framework Limitations and Considerations
- Complete objects must be available before emission: Structured streaming is more complex than ordinary text streaming because the framework must receive enough structured data to construct a valid Java object. Unlike text streaming, where individual tokens can be displayed immediately, Embabel must preserve the integrity of the target object. For example, a
Productcontaining a name, category, and price cannot safely be emitted when only the name has been generated. Structured streaming should therefore be understood as incremental delivery of completed objects rather than field-by-field streaming of partially constructed Java objects. - Scalar values require special handling: Structured object streaming is primarily designed around object-shaped schemas. Simple values such as strings do not naturally contain fields describing their structure, so directly requesting a
Stringcan create ambiguity. Embabel addresses this through wrapper types such asStringResult, which represent scalar values within an object structure and make them compatible with structured output processing. - Reasoning depends on the model and provider: Reasoning support is not guaranteed for every LLM. Some models expose explicit thinking or reasoning information, some providers return it separately, and others may not expose it at all. Applications should therefore treat reasoning events as optional and avoid making core business functionality dependent on them. Structured objects should remain the primary application output, with reasoning treated as supplementary information.
- Tool-enabled streaming may require multiple LLM turns: A tool-enabled request can involve several inference cycles. The model may first determine that additional information is required, request an application tool, receive the tool result, and then continue processing the request. Embabel coordinates these interactions and updates the conversation before subsequent LLM inference. Streaming does not remove the agentic tool loop; it allows information from the different stages to be delivered incrementally.
- Streaming does not guarantee accuracy: Streaming changes how and when generated information is delivered, but it does not make LLM output deterministic or guarantee a correct business result. Applications should still use well-defined domain models, clear prompts, validation, appropriate model selection, timeouts, and error handling, particularly when generated results are used by downstream business processes.
2. Spring Boot Implementation of LLM Streaming
2.1 Maven Configuration
Start from the current Embabel Java/Spring Boot template or project creator so that the Embabel and Spring AI dependency versions are aligned. A minimal project will also need Spring WebFlux because the HTTP endpoints return Reactor Flux streams.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<!-- Use the Embabel starter/provider dependencies from the current Embabel Java template for the version you are running. -->
2.2 Domain Object
The streaming examples use a simple Product domain object to represent the structured data generated by the LLM. It defines the fields that Embabel expects when converting the model’s structured output into a Java object.
package com.example.streaming.model;
public record Product(String name, String category, double price) { }
The Product record defines three fields: name, category, and price. When structured streaming is requested using Product.class, Embabel uses this type as the target representation for the LLM output and emits completed Product instances through the reactive stream. Using a strongly typed domain object eliminates the need for application code to manually collect raw text, parse JSON, and map individual fields, allowing the streamed result to be consumed directly by normal Java business logic.
2.3 Streaming Service Implementation
The StreamingService contains the core Embabel streaming logic and demonstrates four common scenarios: streaming a single structured object, streaming multiple objects, streaming structured output with reasoning, and combining streaming with tools and reasoning. Each example uses a PromptRunner and StreamingPromptRunnerBuilder to execute the LLM request reactively.
package com.example.streaming;
import com.embabel.agent.api.common.Ai;
import com.embabel.agent.api.common.PromptRunner;
import com.embabel.agent.api.common.streaming.StreamingEvent;
import com.embabel.agent.api.common.streaming.StreamingPromptRunnerBuilder;
import com.example.streaming.model.Product;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.stereotype.Service;
import reactor.core.publisher.Flux;
@Service
public class StreamingService {
private final Ai ai;
public StreamingService(Ai ai) {
this.ai = ai;
}
/**
* Example 1: Stream a single structured Product.
*/
public Flux<Product> streamSingleProduct() {
PromptRunner runner = ai.withDefaultLlm();
String prompt = """
Create exactly one fictional electronics product.
Return:
- name
- category
- price
""";
return new StreamingPromptRunnerBuilder(runner)
.streaming()
.withPrompt(prompt)
.createObjectStreamWithThinking(Product.class)
.filter(StreamingEvent::isObject)
.map(StreamingEvent::getObject);
}
/**
* Example 2: Stream multiple structured Product objects.
*/
public Flux<Product> streamMultipleProducts() {
PromptRunner runner = ai.withDefaultLlm();
String prompt = """
Generate exactly three fictional electronics products.
For every product provide:
- name
- category
- price
Produce each product as a separate structured item.
""";
return new StreamingPromptRunnerBuilder(runner)
.streaming()
.withPrompt(prompt)
.createObjectStreamWithThinking(Product.class)
.filter(StreamingEvent::isObject)
.map(StreamingEvent::getObject);
}
/**
* Example 3: Stream reasoning together with the structured Product result.
*/
public Flux<StreamingEvent<Product>> streamWithReasoning() {
PromptRunner runner = ai.withDefaultLlm();
String prompt = """
Recommend one fictional laptop for a software developer.
Think about:
- development workload
- portability
- price
Then produce the recommended Product.
""";
return new StreamingPromptRunnerBuilder(runner)
.streaming()
.withPrompt(prompt)
.createObjectStreamWithThinking(Product.class);
}
/**
* Example 4: Stream reasoning while allowing the LLM
* to call a Java tool before producing the Product.
*/
public Flux<StreamingEvent<Product>> streamWithToolsAndReasoning() {
PromptRunner runner = ai
.withDefaultLlm()
.withToolObject(this);
String prompt = """
Recommend the DevBook Pro laptop.
You MUST use the available product price tool
to obtain its current price.
Think about the product and the price returned
by the tool.
Finally return a Product containing:
- name
- category
- price
""";
return new StreamingPromptRunnerBuilder(runner)
.streaming()
.withPrompt(prompt)
.createObjectStreamWithThinking(Product.class);
}
/**
* Tool available to the LLM.
*/
@Tool(description = "Returns the current price of a product")
public double getProductPrice(String productName) {
return switch (productName.toLowerCase()) {
case "devbook pro" -> 1299.00;
case "novabook air" -> 899.00;
default -> 999.00;
};
}
}
The service begins by injecting Embabel’s Ai abstraction and obtaining the application’s default LLM through ai.withDefaultLlm(). The streamSingleProduct() method requests one structured Product, while streamMultipleProducts() asks the model to generate several products and emits each completed object through a Reactor Flux; in both cases, reasoning events are filtered out using StreamingEvent::isObject. The streamWithReasoning() method instead returns StreamingEvent<Product> directly so that both reasoning and completed object events can be consumed by the caller. Finally, streamWithToolsAndReasoning() makes the service’s @Tool-annotated getProductPrice() method available to the LLM, allowing the model to request price information during the interaction before producing the final structured Product. Together, these methods demonstrate how the same Embabel streaming API can progress from basic structured output to multiple objects, reasoning-aware streaming, and tool-enabled agentic execution.
2.4 Spring Boot Application
The StreamingApplication class serves as the entry point for the Spring Boot application. It starts the Spring application context and enables component scanning so that the streaming service, REST controller, and other required Spring components can be discovered and initialized.
package com.example.streaming;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class StreamingApplication {
public static void main(String[] args) {
SpringApplication.run(StreamingApplication.class, args);
}
}
The @SpringBootApplication annotation combines Spring Boot configuration, auto-configuration, and component scanning. When the main() method calls SpringApplication.run(), Spring Boot starts the application and creates the required beans, including StreamingService and StreamingController. Embabel’s configured Spring integration supplies the AI-related infrastructure used by the service, while Spring WebFlux provides the reactive HTTP layer required to expose the Flux-based streaming endpoints.
2.5 REST Controller for Streaming
The StreamingController exposes the four Embabel streaming scenarios as HTTP endpoints using Spring WebFlux. Each endpoint produces text/event-stream, enabling Server-Sent Events (SSE) so that individual products, reasoning events, and final structured results can be sent to the client incrementally as they become available.
package com.example.streaming;
import com.embabel.agent.api.common.streaming.StreamingEvent;
import com.example.streaming.model.Product;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import reactor.core.publisher.Flux;
@RestController
@RequestMapping("/api/stream")
public class StreamingController {
private final StreamingService streamingService;
public StreamingController(StreamingService streamingService) {
this.streamingService = streamingService;
}
/**
* Stream one Product.
*/
@GetMapping(value = "/single",produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<Product> singleProduct() {
return streamingService.streamSingleProduct();
}
/**
* Stream multiple Product objects.
*/
@GetMapping(value = "/multiple", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<Product> multipleProducts() {
return streamingService.streamMultipleProducts();
}
/**
* Stream reasoning and the final Product.
*/
@GetMapping(value = "/reasoning", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> reasoning() {
return streamingService
.streamWithReasoning()
.map(this::formatEvent);
}
/**
* Stream reasoning while allowing tool execution.
*/
@GetMapping(value = "/tools", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> toolsAndReasoning() {
return streamingService
.streamWithToolsAndReasoning()
.map(this::formatEvent);
}
/**
* Convert StreamingEvent into a value that is easy
* to observe through an SSE client such as curl.
*/
private String formatEvent(StreamingEvent<Product> event) {
if (event.isThinking()) {
return "THINKING: " + event.getThinking();
}
if (event.isObject()) {
return "OBJECT: " + event.getObject();
}
return "";
}
}
The controller delegates all LLM processing to StreamingService and focuses only on exposing the resulting reactive streams over HTTP. The /single endpoint streams one completed Product, while /multiple can deliver multiple completed products as separate stream elements. The /reasoning and /tools endpoints retain StreamingEvent<Product> information and use formatEvent() to distinguish thinking events from completed object events before sending them to the client. Returning Flux directly is important because Spring WebFlux manages the subscription and writes each emitted value as an SSE event, so the controller does not need to call subscribe() manually or wait for the entire LLM response to finish.
2.6 Application Configuration
The application requires only minimal Spring Boot configuration because the streaming service obtains its model through Embabel’s configured default LLM. Provider-specific credentials and model settings should be configured separately, preferably through environment variables or the configuration mechanism recommended by the selected Embabel LLM provider.
spring.application.name=embabel-streaming-demo
The spring.application.name property assigns a name to the Spring Boot application and does not itself configure an LLM. The actual model, provider, API credentials, and any provider-specific options must be supplied through the Embabel starter and provider configuration used by the project. Since StreamingService obtains a PromptRunner using ai.withDefaultLlm(), all four streaming examples automatically use the default LLM configured for the application, avoiding the need to repeat provider or model configuration inside the Java code.
2.7 Running the Application
Once the required Embabel provider and credentials have been configured, start the application normally:
mvn spring-boot:run
The application starts on port 8080 by default. Because these endpoints use Server-Sent Events, the -N option should be used with curl to disable output buffering and make individual events visible as they arrive.
2.8 Streaming a Single Structured Object
Streaming a single structured object is useful when an application expects one typed result from the LLM but still wants to use the same reactive streaming infrastructure as more complex workflows. In this example, Embabel maps the model output to the Product domain type, while the service filters out reasoning events and exposes only the completed object. Because a structured object must contain enough data to be deserialized correctly, the client receives the Product only after its required structured content has been generated. Call the single-object endpoint using:
curl -N http://localhost:8080/api/stream/single
A response is:
data:{"name":"NovaBook Air","category":"Electronics","price":899.99}
The endpoint returns the completed Product as an SSE event. Internally, createObjectStreamWithThinking(Product.class) creates the structured event stream, while filter(StreamingEvent::isObject) removes non-object events and map(StreamingEvent::getObject) extracts the actual Product. This keeps the client-facing response simple while allowing the application to use Embabel’s structured streaming mechanism.
2.9 Streaming Multiple Structured Objects
Streaming multiple structured objects is useful when the LLM generates a collection of independent results that can be consumed individually. Instead of waiting for the entire collection to be generated and returned as a List<Product>, Embabel can emit each completed Product through the reactive stream as soon as that object is available, allowing the client to begin processing or displaying results earlier. Call the multiple-object endpoint using:
curl -N http://localhost:8080/api/stream/multiple
A response is:
data:{"name":"NovaBook Air","category":"Electronics","price":899.99}
data:{"name":"PixelSound Mini","category":"Electronics","price":79.99}
data:{"name":"VisionPad Pro","category":"Electronics","price":649.99}
Each completed Product is emitted as a separate element of the Flux and delivered to the client as an SSE event. The service filters out reasoning events and extracts only structured objects, so the client receives a sequence of typed product results rather than intermediate model output. This approach is particularly useful for recommendations, search results, extraction tasks, or other scenarios where individual results can be used independently without waiting for the complete set.
2.10 Streaming With Reasoning
Streaming with reasoning allows the application to receive intermediate reasoning events in addition to the final structured result. Instead of filtering the stream down to completed Product objects, the service preserves StreamingEvent<Product> instances so that reasoning and object events can be handled separately. This can be useful for observing the progression of an LLM request when the selected model and provider expose reasoning information. Call the reasoning endpoint using:
curl -N http://localhost:8080/api/stream/reasoning
Depending on the selected model and provider, a response like:
data:THINKING: I should consider development performance, portability and price. data:THINKING: A developer laptop should provide sufficient performance while remaining portable. data:OBJECT: Product[name=DevBook Pro, category=Laptop, price=1299.0]
Unlike the single-object and multiple-object examples, the service does not filter StreamingEvent instances. The controller checks each event and formats reasoning events separately from the completed Product. The final object is therefore delivered through the same reactive stream as the reasoning information. Reasoning should be treated as optional because its availability and exact content depend on the capabilities and configuration of the selected model and provider.
2.11 Streaming With Tools and Reasoning
Tool-enabled streaming extends the reasoning workflow by allowing the LLM to request application functionality while generating its response. A single request can therefore involve multiple stages: the model reasons about the task, requests a tool when additional information is required, Embabel executes the tool, and the model continues processing using the returned result before producing the final structured object. Call the tool-enabled endpoint using:
curl -N http://localhost:8080/api/stream/tools
Depending on the model and provider, the client-visible output is:
data:THINKING: I need to obtain the current price of DevBook Pro. data:THINKING: The pricing tool returned a price of 1299.0. data:OBJECT: Product[name=DevBook Pro, category=Laptop, price=1299.0]
In this example, withToolObject(this) makes the service’s tool methods available to the prompt runner, while the @Tool-annotated getProductPrice() method provides the product price. When the model requests this information, Embabel coordinates the tool invocation and makes the returned value available to the subsequent LLM processing. The tool call itself should be distinguished from the streamed result: the tool provides application data or functionality, while the streaming pipeline carries the available reasoning events and ultimately the completed Product. This combination enables agentic workflows in which an LLM can obtain external or application-specific information while still delivering results incrementally.
3. Conclusion
LLM streaming in the Embabel Agentic AI Framework provides a natural way to build responsive AI functionality in Java and Spring applications. At its simplest, an application can stream a single structured domain object instead of waiting for a conventional blocking response. The same mechanism can emit multiple completed objects incrementally, making it possible to process or display results as they become available. When reasoning is supported by the selected model and provider, StreamingEvent<T> allows reasoning information and structured domain objects to travel through the same reactive stream while remaining distinguishable from each other. Tool support extends the model further by allowing an LLM to request application functionality, receive the result, continue its processing, and eventually produce a structured response. The key point is that these capabilities address different concerns. Structured output determines the form of the result, streaming determines how that result is delivered, reasoning provides optional information about intermediate model processing, and tools allow the LLM to interact with application capabilities or external data. Embabel brings these pieces together while still allowing Spring developers to work with familiar concepts such as dependency injection, Java domain objects, Reactor Flux, Spring WebFlux, and Server-Sent Events. For production applications, streaming should be used where incremental delivery provides a meaningful benefit rather than simply because it is available. Applications should also continue to validate LLM-generated data, handle failures and timeouts, select appropriate models, and avoid making critical business logic dependent on optional reasoning output. With those considerations in place, structured streaming provides a practical bridge between reactive Spring applications and agentic LLM workflows.
