Your load test says the 99th percentile is fine. Meanwhile, a thousand requests quietly vanished from the results.
The Benchmark That Looked Perfect
Picture a load test that reports a 99th percentile of a tenth of a millisecond. The dashboard is green, and the team celebrates. Yet during the test the server froze for a full second, and nobody saw it in the percentiles.
This blind spot has a name. Gil Tene calls it coordinated omission, and he argues in a Mechanical Sympathy post that it can often make percentile data useless. Below, we will see how it works, and then reproduce it in about forty lines of Java.
What Coordinated Omission Means
Omission Is Not the Problem, Coordination Is
According to Tene, leaving out some results is harmless when the omission is random. The trouble starts when the omission is coordinated with the events you are observing. Then you drop mostly the very bad results, and the remaining data looks far better than reality.
How It Happens in a Load Test
Consider a simple load generator. It sends a request, waits for the answer, and only then sends the next one. If the server stalls, the generator also stalls. Consequently, the requests that should have been sent during the stall are never sent, and never measured. The tool records one slow request instead of the thousand that real users would have suffered.
The fix is to measure from the moment a request was supposed to start, not from the moment the tool got around to sending it. Tene’s wrk2 tool does exactly this with a constant request rate. In addition, HdrHistogram offers a recording mode that corrects for the problem when you know the expected interval between measurements. Matt Warren shows the call, recordValueWithExpectedInterval, in his post on .NET garbage collection.
A Demo You Can Run
To make this concrete, I wrote a small program. It plans 10,000 requests at one per millisecond, and each request takes 0.1 ms of work. At request 5,000, the “server” freezes for one second. Every request records two numbers: the naive latency, measured from the actual send, and the honest latency, measured from the intended start time. It needs JDK 11 or newer, and I tested it on JDK 21.
import java.util.Arrays;
import java.util.concurrent.locks.LockSupport;
public class Co {
static void spin(long nanos) {
long end = System.nanoTime() + nanos;
while (System.nanoTime() < end) { }
}
public static void main(String[] args) {
int requests = 10_000; // planned: 1000 requests per second for 10 s
long interval = 1_000_000; // 1 ms between intended sends
long service = 100_000; // 0.1 ms of work per request
double[] naive = new double[requests];
double[] honest = new double[requests];
long start = System.nanoTime();
for (int i = 0; i < requests; i++) {
long intended = start + i * interval;
long now;
while ((now = System.nanoTime()) < intended) LockSupport.parkNanos(intended - now);
long sent = System.nanoTime();
if (i == 5000) {
LockSupport.parkNanos(1_000_000_000L); // the server stalls for 1 second
} else {
spin(service);
}
long done = System.nanoTime();
naive[i] = (done - sent) / 1e6; // what most tools report
honest[i] = (done - intended) / 1e6; // from when the request SHOULD have started
}
report("naive ", naive);
report("honest", honest);
}
static void report(String label, double[] ms) {
double[] s = ms.clone();
Arrays.sort(s);
System.out.printf("%s p50=%.2f p90=%.2f p99=%.2f p99.9=%.2f max=%.2f ms%n", label,
s[s.length / 2], s[(int) (s.length * 0.90)], s[(int) (s.length * 0.99)],
s[(int) (s.length * 0.999)], s[s.length - 1]);
}
}
java Co.java
The program takes about ten seconds. I ran it three times, and the results were nearly identical each time.
What the Numbers Show
| Percentile | Naive (ms) | Honest (ms) |
|---|---|---|
| p50 | 0.10 | 0.20 |
| p90 | 0.10 | 102.3 |
| p99 | 0.13 | 911.4 |
| p99.9 | 1.51 | 992.3 |
| max | 1000.2 | 1000.3 |
The gap is huge. The naive p99 is 0.13 ms, while the honest p99 is about 911 ms. Only the maximum agrees, because the single stalled request is the one request the naive view does record. Note also that the chart below uses a logarithmic axis, since the values span four orders of magnitude.
Why does the honest view look so bad? Because after the freeze, a backlog of about a thousand requests had to wait. Each one was late by a different amount, and users would have felt every one of them.
The timeline makes the mechanism visible. Request 5,000 sits at one second, and each later request is slightly less late, until the backlog clears near request 6,100.
Where This Demo Needs Care
This demo is deliberately simple. It uses one thread, one stall, and a fixed rate. Real systems have many clients, many connections, and messier stalls. Moreover, if real users truly wait for each reply before doing anything else, the right measurement may differ. Most services, however, receive requests from many independent clients who do not wait for each other, and for those the honest view is the one that matches experience.
| Approach | Measures from | Risk |
|---|---|---|
| Closed-loop tool, no correction | Actual send time | Hides stalls, flatters percentiles |
| Constant-rate tool such as wrk2 | Intended send time | Needs a rate you choose |
| HdrHistogram with expected interval | Corrected after the fact | You must know the interval |
Quick checklist
- Ask whether your tool sends at a fixed rate or waits for each reply.
- Measure latency from the intended start time, not the actual send time.
- Report high percentiles such as p99 and p99.9, not only averages.
- Be suspicious of sudden vertical jumps in a percentile chart.
- Rerun with a deliberate stall to test whether your tool notices it.
What We Have Learned
Coordinated omission happens when a load generator slows down along with the system it tests, so the worst requests are never sent or never counted. In my demo, a one-second freeze produced a naive p99 of 0.13 ms but an honest p99 of about 911 ms. The cure is to measure from the intended start time, which tools such as wrk2 and HdrHistogram’s expected-interval recording support. As a result, a stall shows up as the backlog of late requests it really created. Next time a benchmark looks too good, inject a stall and see whether your tool notices.
