AI applications do not always need the same language model for every request. A simple factual question may only require a small, fast model, while a request involving multiple constraints, reasoning steps, or difficult decisions may require a more capable model.
Using the same model for both workloads can create unnecessary cost, latency, and resource consumption. It can also create a reliability problem when an application depends entirely on one model or provider. A better approach is dynamic model routing. Instead of hard-coding one model into the application, a routing layer evaluates each request and chooses the model that is appropriate for that workload.
This article explains automatic model switching and then builds a working example using Quarkus, LangChain4j, and Ollama.
1. Understanding Automatic Model Routing
Automatic model routing means choosing an AI model dynamically instead of configuring one model for every request. A conventional application might send every request directly to a single model. That works well when all requests have roughly the same requirements, but real applications often process a mixture of simple and complex tasks.
For example, consider a travel assistant. A question such as “What currency is used in Japan?” does not require extensive reasoning. A request such as “Plan a ten-day trip across Tokyo, Kyoto, and Osaka while minimizing transportation costs, limiting daily travel time, and providing alternatives for bad weather” is substantially more demanding.
Both requests could be sent to a powerful model, but doing so may waste resources. A routing layer can instead classify the first request as SIMPLE and use a fast model, while sending the second to a reasoning model. The basic idea is to match the model’s capabilities to the workload.
Why Use Multiple Models?
Different models offer different combinations of speed, capability, resource consumption, and response quality. A small model can often handle straightforward requests quickly, while a larger model can provide better results for difficult tasks. This makes a multi-model architecture useful for several reasons.
Cost and resource usage can be reduced by keeping expensive models for requests that actually need them. Latency can also improve because simple requests can be handled by smaller models. At the same time, response quality can improve because complex requests are not forced through a model that may lack the required reasoning capability.
Multiple models can also provide a degree of resilience. If a preferred model becomes unavailable, the application can fall back to another model rather than failing immediately.
2. Choosing a Routing Strategy
There are several ways an application can decide which model to use. The simplest is rule-based routing. The application can inspect characteristics such as prompt length, keywords, or whether the request contains code. This approach is fast and does not require an additional model call, but rules can become difficult to maintain as the application grows.
Another option is LLM-based routing, where a small model examines the request and classifies its complexity. This provides a better understanding of the request’s meaning and avoids maintaining a large collection of application-specific rules.
For this example, we will use an LLM-based classifier. A small Ollama model will classify each request into one of three categories:
- SIMPLE — factual questions, definitions, simple recommendations, and single-step tasks.
- MEDIUM — normal planning, comparisons, content generation, and requests involving several related steps.
- COMPLEX — multi-step reasoning, multiple constraints, detailed planning, ambiguous requirements, and trade-off analysis.
The classification model itself should be lightweight because its purpose is to decide which model should handle the actual request. Using the most powerful model as the classifier would defeat much of the purpose of routing.
3. Setting Up the Application
Create a new Quarkus application with REST, LangChain4j Ollama, and CDI support:
quarkus create app modelrouter \ --package-name=com.jcg.example \ --extension='quarkus-rest,quarkus-langchain4j-ollama,quarkus-arc' \ --java=25 \ --no-code
Move into the project:
cd modelrouter
Before starting the application, make sure Ollama is installed and running:
ollama serve
Then download the models:
ollama pull qwen2.5:0.5b ollama pull qwen2.5:3b ollama pull llama3.2
You can verify the installed models with:
ollama list
4. Configuring Multiple Ollama Models
Quarkus LangChain4j allows multiple model instances to be configured in the same application. Each instance can have its own name, model, temperature, and timeout. Open src/main/resources/application.properties and add:
# Explicit Ollama host
quarkus.langchain4j.ollama.devservices.enabled=false
quarkus.langchain4j.ollama.base-url=http://localhost:11434
# Reasoning lane - default unnamed model
quarkus.langchain4j.ollama.chat-model.model-id=llama3.2
quarkus.langchain4j.ollama.chat-model.temperature=0.7
quarkus.langchain4j.timeout=60s
# Fast lane
quarkus.langchain4j.fast.chat-model.provider=ollama
quarkus.langchain4j.ollama.fast.base-url=${quarkus.langchain4j.ollama.base-url}
quarkus.langchain4j.ollama.fast.chat-model.model-id=qwen2.5:0.5b
quarkus.langchain4j.ollama.fast.chat-model.temperature=0.0
quarkus.langchain4j.ollama.fast.timeout=15s
# Balanced lane
quarkus.langchain4j.balanced.chat-model.provider=ollama
quarkus.langchain4j.ollama.balanced.base-url=${quarkus.langchain4j.ollama.base-url}
quarkus.langchain4j.ollama.balanced.chat-model.model-id=qwen2.5:3b
quarkus.langchain4j.ollama.balanced.chat-model.temperature=0.3
quarkus.langchain4j.ollama.balanced.timeout=30s
The unnamed model uses llama3.2 and represents the reasoning lane. The fast configuration creates a named model backed by qwen2.5:0.5b, while the balanced configuration creates another named model backed by qwen2.5:3b. The important part is that the application now has three independently configured ChatModel instances
5. Creating a Structured Request Classifier
The first step in routing requests is to define a small set of categories that the application can use to describe task complexity. These categories provide a simple contract between the classifier and the routing logic.
public enum TaskComplexity {
SIMPLE,
MEDIUM,
COMPLEX
}
The TaskComplexity enum represents the three routing levels used by the application. SIMPLE is intended for straightforward requests that can be handled by a fast model, MEDIUM represents requests that need more capable processing, and COMPLEX is reserved for tasks that require deeper reasoning.
Using an enum instead of a plain string makes the classification strongly typed and easier to use in the routing logic. The application can use a switch expression to map each value to the appropriate model.
Next, the application needs an AI service that can classify an incoming request and return one of these enum values. Rather than returning an unrestricted text response, the classifier will produce a structured TaskComplexity value that the application can use directly.
@RegisterAiService(modelName = "fast")
public interface RequestClassifier {
@SystemMessage("""
Classify the travel request into exactly one category:
SIMPLE, MEDIUM, or COMPLEX.
SIMPLE:
Factual lookups, definitions, simple recommendations,
and single-step requests.
MEDIUM:
Normal planning, comparisons, content generation,
and requests involving several related steps.
COMPLEX:
Multi-step reasoning, multiple constraints,
detailed planning, ambiguous requirements,
and trade-off analysis.
Choose the lowest category that can reliably
handle the request.
Return only the enum value.
""")
TaskComplexity classify(@UserMessage String request);
}
@RegisterAiService(modelName = "fast") registers the interface as a LangChain4j AI service and connects it to the named fast model. The system message defines the criteria for each complexity level and instructs the model to return only the appropriate category.
The classify() method accepts the user’s request and returns a TaskComplexity enum. This structured result allows the routing logic to select the appropriate model without parsing arbitrary text responses.
6. Selecting a Model at Runtime
With the request classified, the application can now use the result to select the appropriate ChatModel dynamically.
Create ModelSelectionService.java:
@ApplicationScoped
public class ModelSelectionService {
private final RequestClassifier classifier;
private final ChatModel fastModel;
private final ChatModel balancedModel;
private final ChatModel reasoningModel;
private final Event<RoutingDecision> routingEvents;
@Inject
public ModelSelectionService(
RequestClassifier classifier,
@ModelName("fast") ChatModel fastModel,
@ModelName("balanced") ChatModel balancedModel,
ChatModel reasoningModel,
Event<RoutingDecision> routingEvents) {
this.classifier = classifier;
this.fastModel = fastModel;
this.balancedModel = balancedModel;
this.reasoningModel = reasoningModel;
this.routingEvents = routingEvents;
}
public String generate(String request) {
long start = System.currentTimeMillis();
TaskComplexity complexity = classifier.classify(request);
long classificationMillis = System.currentTimeMillis() - start;
ChatModel selectedModel = switch (complexity) {
case SIMPLE ->
fastModel;
case MEDIUM ->
balancedModel;
case COMPLEX ->
reasoningModel;
};
ChatRequest chatRequest = ChatRequest.builder().messages(UserMessage.from(request)).build();
String response = selectedModel.chat(chatRequest).aiMessage().text();
String modelName = switch (complexity) {
case SIMPLE ->
"qwen2.5:0.5b";
case MEDIUM ->
"qwen2.5:3b";
case COMPLEX ->
"llama3.2";
};
routingEvents.fireAsync(
new RoutingDecision(
request,
complexity,
modelName,
classificationMillis
)
);
return response;
}
}
The ModelSelectionService is a CDI application-scoped bean that coordinates classification and model selection. The @ModelName annotation injects the named fast and balanced ChatModel instances, while the default ChatModel represents the reasoning model.
Inside generate(), the classifier first returns a TaskComplexity value. A switch expression then maps SIMPLE, MEDIUM, and COMPLEX to the fast, balanced, and reasoning models respectively.
The selected model receives the original request through a ChatRequest and generates the final response. The second switch records the model name associated with the selected complexity.
Finally, the service publishes a RoutingDecision through fireAsync(). This allows routing information to be observed separately without adding monitoring or logging work to the main response flow.
7. Making Routing Decisions Observable
Once the application can select models dynamically, it is useful to capture those decisions for logging, monitoring, or later analysis. A RoutingDecision record provides a simple structure for this information.
Create RoutingDecision.java:
public record RoutingDecision(
String request,
Complexity complexity,
String selectedModel,
long classificationMillis,
Instant timestamp) {
public RoutingDecision(
String request,
Complexity complexity,
String selectedModel,
long classificationMillis) {
this(
request,
complexity,
selectedModel,
classificationMillis,
Instant.now()
);
}
}
The record stores the original request, its classified complexity, the selected model, the time spent on classification, and the timestamp of the decision.
The RoutingDecision is later published as a CDI event, allowing another component to observe routing decisions without adding monitoring logic directly to the model-selection code.
Create an observer:
@ApplicationScoped
public class RoutingObserver {
private static final Logger LOG = Logger.getLogger(RoutingObserver.class);
public void observe(@ObservesAsync RoutingDecision decision) {
LOG.infof(
"[ROUTING] complexity=%s model=%s classificationMs=%d",
decision.complexity(),
decision.selectedModel(),
decision.classificationMillis()
);
}
}
The router publishes the event with:
routingEvents.fireAsync(
new RoutingDecision(
request,
complexity,
modelName,
classificationMillis
)
);
Because the event is asynchronous, the routing observer does not need to become part of the main response path. This also makes the architecture extensible. A future observer could record metrics or send routing information to an observability system.
Exposing the Application Through REST
Create PlannerResource.java:
@Path("/planner")
public class PlannerResource {
private final ModelSelectionService modelSelectionService;
@Inject
public PlannerResource(ModelSelectionService modelSelectionService) {
this.modelSelectionService = modelSelectionService;
}
@POST
@Consumes(MediaType.TEXT_PLAIN)
@Produces(MediaType.TEXT_PLAIN)
public String plan(String request) {
return modelSelectionService.generate(request);
}
}
The above REST resource simply passes the request to ModelSelectionService. The routing service is responsible for classification, model selection, generation, and publishing the routing event.
Start the application with: ./mvnw quarkus:dev. The application will run on port 8080.
For a simple request, send:
curl -X POST http://localhost:8080/planner \ -H "Content-Type: text/plain" \ -d "What currency is used in Japan?"
The classifier should normally identify this as SIMPLE, causing the request to use qwen2.5:0.5b.
Next, try a medium-complexity request:
curl -X POST http://localhost:8080/planner \ -H "Content-Type: text/plain" \ -d "Create a three-day itinerary for Barcelona with museums, local food, and outdoor activities."
This should normally be classified as MEDIUM and sent to qwen2.5:3b.
Finally, try a more demanding request:
curl -X POST http://localhost:8080/planner \ -H "Content-Type: text/plain" \ -d "I have ten days to visit Tokyo, Kyoto, and Osaka. My budget is limited, I want to minimize transportation costs, I do not want to spend more than four hours traveling on any day, I prefer hotels near major train stations, and I need indoor alternatives for rainy days. Create a day-by-day itinerary and explain the major trade-offs."
This request should normally be classified as COMPLEX and sent to llama3.2.
The classification is still performed by an LLM, so these categories should be treated as routing decisions rather than absolute guarantees.
8. Handling Model Failures
Routing and fallback are related but different concerns.
Routing answers: Which model should handle this request?
Fallback answers: What should happen if that model fails?
For example, if a complex request is routed to llama3.2 but the model times out, the application could retry using the balanced model. A simple fallback implementation could look like this:
private String generateWithFallback(
ChatModel primary,
ChatModel fallback,
ChatRequest request) {
try {
return primary.chat(request)
.aiMessage()
.text();
} catch (RuntimeException exception) {
return fallback.chat(request)
.aiMessage()
.text();
}
}
The fallback strategy should be designed carefully. A model timeout or temporary availability problem may justify a fallback, while invalid input or an application-level validation error usually should not trigger another model request.
Fallbacks can also affect cost. If a request regularly fails on the preferred model and is retried on a second model, the application may end up consuming more resources rather than fewer.
9. Improving the Routing Strategy
The three-level approach provides a good starting point, but a production system can make routing more sophisticated. One improvement is to include confidence in the classifier’s output. Instead of returning only COMPLEX, the classifier could return both a category and a confidence score. A low-confidence SIMPLE classification could then be promoted to the balanced model rather than being sent directly to the smallest model.
Another improvement is to consider model health and system load. A request might normally go to the reasoning model, but if that model is unavailable, the router could temporarily use a balanced model.
The application can also collect metrics such as the number of requests handled by each model, classification latency, generation latency, model failures, fallback frequency, and resource consumption. These metrics help determine whether routing is actually providing the expected benefits.
It is also worth remembering that model routing introduces an additional model call. If every request must first pass through a classifier, the classification latency becomes part of the overall response time. A fast classifier and a carefully designed classification prompt are therefore important.
10. Conclusion
In this article, we explored how automatic model switching can make AI applications more flexible and efficient. Instead of sending every request to one model, the application evaluates the workload and chooses a model that is appropriate for its complexity.
We built a Quarkus application using LangChain4j and Ollama with three model lanes: a fast qwen2.5:0.5b model for simple requests, a balanced qwen2.5:3b model for normal workloads, and llama3.2 for complex reasoning.
The implementation demonstrated how to configure multiple named Ollama model instances, create a structured-output AiService that returns a Java enum, inject named ChatModel beans with @ModelName, select models programmatically at runtime, and publish routing decisions through asynchronous CDI events.
The key idea is that the best model for an AI application is not necessarily the most powerful model for every request. A well-designed routing layer allows the application to use smaller models when they are sufficient and reserve more capable models for workloads that genuinely require them.
11. Download the Source Code
Source and Attribution: The code example in this article is a modified version of the model-routing example originally written by Markus Eisele in his article, Model Routing in Quarkus LangChain4j with Ollama.
You can download the full source code of this example here: build ai applications that switch models automatically
