NVIDIA Triton Inference Server is an open-source platform for deploying and serving machine learning models in production, supporting frameworks such as TensorFlow, PyTorch, ONNX Runtime, and TensorRT. It exposes HTTP and gRPC endpoints that allow applications, including Java-based systems, to send inference requests and receive predictions. Triton provides features such as multi-framework support, concurrent request processing, dynamic batching, optimized inference execution, and model version management, making it easier to integrate and scale machine learning workloads using HTTP APIs or generated Java gRPC clients.
1. Introduction to Triton Java API
The Triton Java API enables Java applications to communicate with NVIDIA Triton Inference Server. A Java application can submit input tensors, request predictions, and process output tensors without implementing the model’s inference logic itself. For example, a Spring Boot application can send customer data to a model hosted by Triton and receive a prediction. The application remains responsible for business logic, while Triton handles model execution. There are two common integration approaches:
- HTTP REST API: Use Java’s built-in `HttpClient` to send JSON inference requests. This is convenient for simple integrations.
- gRPC API: Use generated Java classes based on Triton’s Protocol Buffers definitions. This is useful for strongly typed requests and responses.
The following tutorial uses the HTTP API to keep the Java example simple and easy to run.
2. How to Set Up Triton with Docker
2.1 Prerequisites
Before setting up NVIDIA Triton Inference Server, install Docker, Java 11 or later, and Maven if you plan to build a Maven project. GPU-based inference requires a compatible NVIDIA GPU and the NVIDIA Container Toolkit, while supported models can also run on a CPU.
2.2 Create a Model Repository
Triton loads models from a model repository. Each model has its own directory containing a configuration file and model implementation. In this tutorial, we will create a simple Python model that multiplies every input value by two. Create the following directory structure:
triton-java-demo/
└── model_repository/
└── double_model/
├── config.pbtxt
└── 1/
└── model.py
2.3 Configure the Model
Create the model_repository/double_model/config.pbtxt file with the following configuration:
name: "double_model"
backend: "python"
max_batch_size: 0
input [
{
name: "INPUT"
data_type: TYPE_FP32
dims: [ 4 ]
}
]
output [
{
name: "OUTPUT"
data_type: TYPE_FP32
dims: [ 4 ]
}
]
This configuration defines a model named double_model. It accepts four floating-point values and returns four floating-point values. The max_batch_size: 0 setting means that the declared dimensions represent the complete input tensor without an additional batch dimension.
2.4 Implement the Model
Create the model_repository/double_model/1/model.py file:
import numpy as np
import triton_python_backend_utils as pb_utils
class TritonPythonModel:
def initialize(self, args):
pass
def execute(self, requests):
responses = []
for request in requests:
input_tensor = pb_utils.get_input_tensor_by_name(request, "INPUT")
input_data = input_tensor.as_numpy()
output_data = input_data * 2
output_tensor = pb_utils.Tensor("OUTPUT", output_data.astype(np.float32))
responses.append(pb_utils.InferenceResponse(output_tensors=[output_tensor]))
return responses
def finalize(self):
pass
The Python backend receives each inference request, reads the input tensor, multiplies its values by two, and returns the result as an output tensor.
2.5 Start Triton Using Docker
From the triton-java-demo directory, execute the following command to start Triton Inference Server:
docker run --rm \ -p 8000:8000 \ -p 8001:8001 \ -p 8002:8002 \ -v "$(pwd)/model_repository:/models" \ nvcr.io/nvidia/tritonserver:26.09-py3 \ tritonserver --model-repository=/models
This command starts Triton Inference Server inside a Docker container and mounts the local model repository at /models. It exposes port 8000 for the HTTP inference API, port 8001 for the gRPC inference API, and port 8002 for metrics. This example uses a Triton container with the Python backend; if you choose a different Triton release, ensure that the container image and model configuration are compatible.
2.6 Verify the Server
Open another terminal and execute the following command to check whether the Triton server is ready:
curl http://localhost:8000/v2/health/ready
An HTTP 200 OK response indicates that Triton is ready to accept requests. You can also check whether the model is ready by running:
curl http://localhost:8000/v2/models/double_model/ready
If the model is loaded successfully, this endpoint returns a successful response. Once the server and model are ready, the Java application can send inference requests to the model.
3. Code Example
We will now create a Java application that sends an inference request to `double_model` and prints the model’s output.
3.1 Create a Maven Project
Create a file named `pom.xml`:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>triton-java-demo</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.release>11</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
</project>
No additional Java dependency is required because the example uses the built-in Java HTTP client and JSON serialization is handled manually.
3.2 Create the Java client
Create `src/main/java/com/example/TritonJavaClient.java`:
package com.example;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.Arrays;
public class TritonJavaClient {
private static final String TRITON_URL =
"http://localhost:8000/v2/models/double_model/infer";
public static void main(String[] args) throws Exception {
float[] input = {1.0f, 2.0f, 3.0f, 4.0f};
String inputJson = Arrays.toString(input);
String requestBody = """
{
"inputs": [
{
"name": "INPUT",
"shape": [4],
"datatype": "FP32",
"data": %s
}
],
"outputs": [
{
"name": "OUTPUT"
}
]
}
""".formatted(inputJson);
HttpClient client = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(5))
.build();
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(TRITON_URL))
.timeout(Duration.ofSeconds(30))
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(requestBody))
.build();
HttpResponse<String> response = client.send(
request,
HttpResponse.BodyHandlers.ofString()
);
if (response.statusCode() != 200) {
throw new IllegalStateException(
"Triton inference failed: "
+ response.statusCode()
+ " - " + response.body()
);
}
System.out.println("Input: " + inputJson);
System.out.println("Triton response: " + response.body());
}
}
3.2.1 Code Explanation
The TritonJavaClient class demonstrates how a Java application communicates with NVIDIA Triton Inference Server using its HTTP inference API. The package com.example; statement defines the package, while the imports provide Java’s built-in HTTP client, URI handling, timeout configuration, and array utilities. The TRITON_URL constant specifies the inference endpoint for the double_model model running on localhost:8000. Inside the main() method, a float[] array contains four input values: 1.0, 2.0, 3.0, and 4.0. The Arrays.toString(input) method converts the array into a string representation, which is inserted into the JSON request body using Java’s text blocks and the formatted() method. The JSON payload specifies the input tensor name as INPUT, its shape as [4], its data type as FP32, and the values to process; it also requests the output tensor named OUTPUT. Next, HttpClient.newBuilder() creates an HTTP client with a five-second connection timeout. The HttpRequest.newBuilder() method constructs the request by setting the target URI, a 30-second request timeout, the Content-Type: application/json header, and the HTTP POST method with the JSON payload as its body. The client.send() method sends the request synchronously and stores the server’s response in an HttpResponse<String> object, using BodyHandlers.ofString() to read the response body as a string. The program then checks whether the HTTP status code is 200; if it is not, an IllegalStateException is thrown with the status code and response body to help diagnose the failure. Finally, System.out.println() displays the input values and Triton’s response, which contains the output tensor values produced by the model. Since the example model multiplies each input value by two, the expected prediction is [2.0, 4.0, 6.0, 8.0]. This example illustrates the complete inference workflow, from preparing input data and sending an HTTP request to validating the response and displaying the prediction.
3.3 Code Run and output
Ensure that NVIDIA Triton Inference Server is running and the double_model model is ready. Compile the Java application using Maven and execute the TritonJavaClient class with Java. The application sends the input values [1.0, 2.0, 3.0, 4.0] to Triton, which multiplies each value by two and returns the output tensor. The console output will be:
Input: [1.0, 2.0, 3.0, 4.0]
Triton response: {
"model_name": "double_model",
"model_version": "1",
"outputs": [
{
"name": "OUTPUT",
"datatype": "FP32",
"shape": [4],
"data": [2.0, 4.0, 6.0, 8.0]
}
]
}
The response contains the model name, version, output tensor metadata, and predicted values. Formatting and additional response fields may vary depending on the Triton server version.
4. Conclusion
NVIDIA Triton Inference Server provides a flexible platform for deploying machine learning models and integrating them with Java applications. By using Docker, developers can set up a model-serving environment and expose inference endpoints through HTTP or gRPC. In this tutorial, we created a simple model, configured Triton, and used Java’s built-in HTTP client to send an inference request and receive predictions. This approach can be extended to integrate production machine learning models into Java applications, while Triton’s model management and performance features help support scalable inference workloads.
