Every Java developer knows the comfort of assertEquals. Give a method the same input, and you get the same output. Then you add a large language model to your service, and the build turns red on Monday, green on Tuesday, and red again on Wednesday.
The code did not change, and the model did not break. The output simply varies. In this article, we build a practical evaluation harness for CI, run a small example, and use exact math to decide how many repetitions a test really needs.
Why Normal Assertions Break
It is tempting to set the temperature to zero and call the problem solved. However, the Anthropic SDK reference for the temperature parameter states that even at 0.0 the results will not be fully deterministic. Therefore, exact string comparison is a trap.
The fix is a change of mindset. Instead of asking “is this the exact answer?”, we ask two better questions. First, does each answer satisfy the properties we care about? Second, how often does it do so?
Three Layers of an Evaluation Harness
Layer 1: cheap, deterministic checks
Start with checks that need no second model. Is the output valid JSON? Does it mention the required fact? Is it under the length limit? Does it avoid forbidden words? These run in milliseconds, and they catch most real regressions.
Layer 2: repeated runs with a pass-rate threshold
Next, run the same prompt many times and require a minimum pass rate. JUnit’s @RepeatedTest looks like a natural fit, but there is a catch. As described in this overview of @RepeatedTest, each repetition behaves like a regular test, so one unlucky answer turns the build red. JUnit 5.10 added a failureThreshold that stops after a set number of failures, but it does not compute a pass rate. Consequently, it is simpler to loop inside a single test and assert on the rate yourself.
Layer 3: a model as the judge
Some qualities, such as relevance or factual accuracy, are hard to check with code. Spring AI offers an Evaluator interface for this, with ready-made RelevancyEvaluator and FactCheckingEvaluator classes, as shown in the Spring AI evaluation docs. The docs describe using the AI model itself to evaluate responses. Remember that a judge is also an LLM, so treat its verdict as one more noisy signal, not as ground truth.
A Harness You Can Run
Here is a minimal version of layers 1 and 2. A fake model stands in for the real one, so the example needs no API key. The model answers “Paris” in several different ways, and sometimes gets it wrong. The harness compares exact matching with a property check, then gates on a 95% confidence lower bound. You need Java 17 or newer.
cat > Harness.java <<'JAVA'
import java.util.Random;
public class Harness {
static String fakeModel(Random rng) {
double r = rng.nextDouble();
if (r < 0.50) return "Paris";
if (r < 0.80) return "The capital of France is Paris.";
if (r < 0.92) return "paris";
return "Lyon";
}
static boolean exactMatch(String a) { return a.equals("Paris"); }
static boolean property(String a) {
return a.toLowerCase().contains("paris");
}
static double wilsonLower(int pass, int n) {
double z = 1.96, p = (double) pass / n;
double centre = p + z * z / (2.0 * n);
double margin = z * Math.sqrt(
p * (1 - p) / n + z * z / (4.0 * n * n));
return (centre - margin) / (1 + z * z / n);
}
public static void main(String[] args) {
Random rng = new Random(42);
int n = 50, exact = 0, prop = 0;
for (int i = 0; i < n; i++) {
String answer = fakeModel(rng);
if (exactMatch(answer)) exact++;
if (property(answer)) prop++;
}
System.out.println("exact-match passes: " + exact + "/" + n);
System.out.println("property passes: " + prop + "/" + n);
double lower = wilsonLower(prop, n);
System.out.printf("95%% lower bound: %.2f%n", lower);
System.out.println(lower >= 0.80 ? "GATE: PASS" : "GATE: FAIL");
}
}
JAVA
java Harness.java
It prints the following, and I confirmed the identical output on JDK 17, 21 and 25.
exact-match passes: 27/50 property passes: 48/50 95% lower bound: 0.87 GATE: PASS
Notice the gap. Exact matching passed only 27 of 50 answers, even though 48 were correct. Moreover, the gate uses the Wilson score interval, so a lucky streak cannot fool it. The fixed seed only makes this demo repeatable; a real model gives no such guarantee.
What the Numbers Say
How many repetitions are enough? I calculated it exactly with the binomial distribution. Assume a healthy feature passes 90% of single runs, and a regressed one passes only 70%. Then compare five gating rules.
| Gate | LLM calls | Healthy build fails | Regression caught |
|---|---|---|---|
| 1 run, must pass | 1 | 10.0% | 30.0% |
| 5 runs, all must pass | 5 | 41.0% | 83.2% |
| 20 runs, need 80% | 20 | 4.3% | 76.2% |
| 50 runs, need 80% | 50 | 0.9% | 92.1% |
| 100 runs, need 80% | 100 | 0.1% | 98.4% |
The result is striking. Running once misses 70% of regressions, and requiring all five runs to pass makes healthy builds fail 41% of the time. In contrast, a rate threshold over more runs improves both numbers together, because random noise averages out.
Making It Work in CI
| Stage | What runs | Why |
|---|---|---|
| Every commit | Layer 1 checks, small repetition count | Fast and cheap feedback |
| Nightly | Larger repetition count, model-as-judge | Tighter statistics, higher cost |
| Before release | Full golden set with stored scores | Compare against the last release |
Because LLM calls cost money, spend them where they pay off. Also, store the scores from each run, so you can spot a slow drift that no single run would reveal.
Quick checklist
- Assert on properties, not exact strings.
- Gate on a pass rate over many runs, not on one run.
- Use a confidence bound so luck cannot pass the build.
- Keep model judges for nightly runs and treat them as noisy.
- Record scores per run to catch slow regressions.
What We Have Learned
LLM output varies even at temperature zero, so exact assertions make CI flaky. A good harness checks properties first, then repeats the test and gates on a pass rate, and uses a model judge only where code cannot decide. In our example, exact matching passed 27 of 50 answers while the property check passed 48, and the exact binomial math showed that a 100-run gate with an 80% threshold fails a healthy build only 0.1% of the time while catching 98.4% of regressions. In short, test the rate, not the string, and spend your repetitions where the statistics need them.
