I wanted a PromQL cheat sheet I could trust, so I built a small Prometheus box and ran every line of one against it. The first surprise came from the latency query everyone copies: histogram_quantile(0.95, ...) on the classic buckets told me the API’s p95 was 95 ms, while the native histogram for the same requests said 1.2 ms. Neither query errored. One of them was simply wrong by a factor of eighty.
This page is the result: selectors, rate vs irate vs increase, sum by and the other aggregations, vector matching, histograms, the functions I actually use, plus recording and alerting rules checked with promtool check rules and unit tested with promtool test rules. I ran every query here in November 2026 against Prometheus 3.15.1 on an Ubuntu 24.04 VM in my lab, scraping node_exporter 1.12.1 and Prometheus itself, and every output below comes from that box.
PromQL cheat sheet: the quick reference
Every row is the exact query I ran on the lab box, and the last column is the value it returned, rounded, from the first series when a query returned several. The sections after the tables show the full output for the queries that need explaining and where they bite. Metric names come from node_exporter and from Prometheus scraping itself, so you can paste them into your own expression browser unchanged. The two app.jobs.processed rows need my demo exporter, described below. Byte values are raw bytes.
Selectors and matchers
| Query | What it does | Lab result (rounded) |
|---|---|---|
up | Every target, 1 if the last scrape worked, 0 if not | 1 (first of 3 series) |
node_filesystem_avail_bytes{mountpoint="/"} | Exact label match | 16320614400 |
node_cpu_seconds_total{cpu="0", mode!="idle"} | Label does not equal | mode=”iowait” 7.39 (first of 7 series) |
node_filesystem_size_bytes{fstype=~"ext4|vfat"} | Regex match (fully anchored) | mountpoint=”/” 19682557952 (first of 3 series) |
node_network_receive_bytes_total{device!~"lo|veth.*"} | Regex does not match | device=”eth0″ 350097151 |
{__name__=~"node_load(1|5|15)"} | Match on the metric name itself | 0 (first of 3 series) |
{"app.jobs.processed"} | UTF-8 metric name, quoted inside the braces | 1407 |
rate({"app.jobs.processed", "queue.name"="emails"}[1m]) | UTF-8 name plus a quoted UTF-8 label | 3 |
Range vectors, offset and time modifiers
| Query | What it does | Lab result (rounded) |
|---|---|---|
node_load1[1m] | Raw samples from the last minute (range vector) | 4 samples, first 0.01 |
node_memory_MemAvailable_bytes offset 5m | Value as it was 5 minutes ago | 3530756096 |
node_load1 @ 1793615400 | Value at a fixed Unix timestamp (use one inside your own retention) | 0.02 |
rate(node_cpu_seconds_total{cpu="0", mode="user"}[5m * 2]) | Duration expression, no flag needed now | 0.06809 |
node_load1 offset (10m / 2) | Arithmetic in an offset (parentheses required) | 0.14 |
max_over_time(rate(node_network_receive_bytes_total{device="eth0"}[1m])[10m:30s]) | Subquery: worst 1m rate in the last 10m, sampled every 30s | 9.898e+04 |
Counters: rate, irate and increase
| Query | What it does | Lab result (rounded) |
|---|---|---|
rate(node_network_receive_bytes_total{device="eth0"}[5m]) | Per-second average over the window; use it for alerts and dashboards | 1728 |
irate(node_network_receive_bytes_total{device="eth0"}[5m]) | Per-second rate from the last two samples only; spiky | 2261 |
increase(prometheus_http_requests_total{handler="/api/v1/query"}[5m]) | Total growth over the window (extrapolated) | code=”200″ 440 (first of 3 series) |
increase(prometheus_http_requests_total{handler="/api/v1/query", code="200"}[5m] anchored) | No extrapolation; needs promql-extended-range-selectors | 559 |
resets(prometheus_http_requests_total{handler="/api/v1/query", code="200"}[1h]) | How many times the counter went back to zero | 0 |
changes(process_start_time_seconds{job="prometheus"}[1h]) | Restarts, counted from the start-time gauge | 0 |
Aggregation operators
| Query | What it does | Lab result (rounded) |
|---|---|---|
sum by (mode) (rate(node_cpu_seconds_total[5m])) | Keep only the listed labels | mode=”idle” 1.956 (first of 8 series) |
avg without (cpu) (rate(node_cpu_seconds_total{mode=~"user|system|iowait"}[5m])) | Drop the listed labels, keep the rest | mode=”iowait” 0.0003684 (first of 3 series) |
max by (handler) (rate(prometheus_http_requests_total[5m])) > 0 | Busiest rate per handler, zeros filtered | handler=”/alerts” 0.01402 (first of 9 series) |
topk(3, sum by (handler) (rate(prometheus_http_requests_total[5m]))) | Three largest series | handler=”/api/v1/query” 1.677 (first of 3 series) |
bottomk(2, node_filesystem_avail_bytes{fstype!="tmpfs"}) | Two smallest series | mountpoint=”/boot/efi” 102995968 (first of 2 series) |
count by (fstype) (node_filesystem_size_bytes) | Number of series per label value | fstype=”ext4″ 2 (first of 3 series) |
count_values("readonly", node_filesystem_readonly) | Count series per sample value | readonly=”0″ 6 |
quantile(0.5, rate(node_cpu_seconds_total{mode="user"}[5m])) | Median across series (not over time) | 0.01142 |
stddev by (mode) (rate(node_cpu_seconds_total{mode=~"user|system"}[5m])) | Spread across series | mode=”system” 0.0003333 (first of 2 series) |
Binary operators, vector matching and sets
| Query | What it does | Lab result (rounded) |
|---|---|---|
node_filesystem_avail_bytes{fstype=~"ext4|vfat"} / node_filesystem_size_bytes * 100 | Percent free; series pair up on identical labels | mountpoint=”/” 82.92 (first of 3 series) |
node_memory_MemAvailable_bytes / 1024 / 1024 | Scalar arithmetic (MiB) | 3403 |
node_filesystem_avail_bytes{fstype!="tmpfs"} < 1e9 | Filter: keep series under 1 GB | mountpoint=”/boot/efi” 102995968 (first of 2 series) |
node_load1 > bool 0.5 | Return 1 or 0 instead of filtering | 0 |
sum without (code) (rate(prometheus_http_requests_total{code=~"4.."}[5m])) / ignoring (code) rate(prometheus_http_requests_total{code="200"}[5m]) | Match while ignoring one label | handler=”/api/v1/query” 0.1972 (first of 2 series) |
node_filesystem_avail_bytes{mountpoint="/"} * on (instance) group_left (nodename, release) node_uname_info | Many-to-one join, copy labels from the right side | nodename=”box1″ 16320614400 |
up{job="missing"} or vector(0) | Fall back to 0 when the left side is empty | 0 |
node_filesystem_avail_bytes{fstype!="tmpfs"} and node_filesystem_readonly == 0 | Left series that also exist on the right | mountpoint=”/” 16320614400 (first of 3 series) |
up unless up{job="node"} | Left series with no match on the right | job=”prometheus” 1 (first of 2 series) |
Histograms
| Query | What it does | Lab result (rounded) |
|---|---|---|
histogram_quantile(0.95, sum by (le) (rate(prometheus_http_request_duration_seconds_bucket[5m]))) | p95 from classic buckets; always keep le | 0.095 |
histogram_quantile(0.95, sum(rate(prometheus_http_request_duration_seconds[5m]))) | p95 from a native histogram (no _bucket, no le) | 0.001195 |
histogram_count(sum(rate(prometheus_http_request_duration_seconds[5m]))) | Requests per second from a native histogram | 1.874 |
histogram_avg(sum(rate(prometheus_http_request_duration_seconds[5m]))) | Mean observation from a native histogram | 0.0002864 |
histogram_fraction(0, 0.0002, sum(rate(prometheus_http_request_duration_seconds[5m]))) | Share of observations between two values | 0.7284 |
sum(rate(prometheus_http_request_duration_seconds_sum[5m])) / sum(rate(prometheus_http_request_duration_seconds_count[5m])) | Mean from classic _sum and _count | 0.0002864 |
Functions
| Query | What it does | Lab result (rounded) |
|---|---|---|
absent(up{job="mysql"}) | Returns 1 when nothing matches (alert on missing targets) | job=”mysql” 1 |
absent_over_time(up{job="node"}[5m]) | Same, over a whole window (empty while the job is up) | empty (no series) |
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[10m], 24 * 3600) | Linear forecast 24 hours ahead | 1.342e+10 |
deriv(node_filesystem_avail_bytes{mountpoint="/"}[10m]) | Per-second slope of a gauge | -3.423e+04 |
delta(node_memory_MemAvailable_bytes[5m]) | Change of a gauge over the window (extrapolated) | 4.307e+07 |
clamp_max(node_load1 * 100, 50) | Cap values (clamp_min, clamp too) | 0 |
time() - node_boot_time_seconds | Uptime in seconds | 1145 |
max_over_time(node_load1[10m]) | Peak of a gauge over time (also min, avg, sum) | 1.31 |
quantile_over_time(0.9, node_load1[10m]) | p90 of one series over time | 1.241 |
label_replace(up, "host", "$1", "instance", "(.*):.*") | New label from a regex capture | host=”localhost” 1 (first of 3 series) |
label_join(up, "target", "/", "job", "instance") | New label from joined labels | target=”node/localhost:9100″ 1 (first of 3 series) |
sort_desc(sum by (handler) (rate(prometheus_http_requests_total[5m]))) | Order the result (instant queries only) | handler=”/api/v1/query” 1.969 (first of 64 series) |
day_of_week() | 0 is Sunday; also hour(), day_of_month() | 1 |
Ready-made node_exporter queries
| Query | What it does | Lab result (rounded) |
|---|---|---|
100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) | CPU busy percent per host | 2.186 |
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) | Memory used percent | 13.1 |
100 * (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) | Root disk used percent | 17.08 |
rate(node_network_receive_bytes_total{device="eth0"}[5m]) * 8 / 1e6 | Inbound Mbit/s | 0.01382 |
sum(rate(prometheus_http_requests_total{code=~"4..|5.."}[5m])) / sum(rate(prometheus_http_requests_total[5m])) | Error ratio (4xx and 5xx) | 0.1526 |
sum(rate(prometheus_http_request_duration_seconds_bucket{le="0.1"}[5m])) / sum(rate(prometheus_http_request_duration_seconds_count[5m])) | Latency SLO: share of requests under 100 ms | 1 |
One row in those tables needs a feature flag (anchored), and so do the experimental functions further down. The native-histogram rows need no flag but do need scrape_native_histograms: true on the scraped target, as in the lab config in the next section, or they come back empty. Everything else works on a default Prometheus 3 install with no flags, including the quoted UTF-8 selectors and the duration arithmetic.
Setting up the box I queried
If you already run Prometheus, you don’t need my file, but add the two histogram keys shown below to your existing prometheus job, or every native-histogram query on this page comes back empty. Your job names may differ too: our install guide calls the node job node_exporter where I use node. I installed the release tarballs the same way our Prometheus 3 install guide for Ubuntu and Debian does (there is also a separate walkthrough to install Prometheus on Ubuntu 26.04), then added three scrape jobs: node_exporter on 9100, Prometheus itself on 9090, and a tiny exporter on 9200 that serves a metric with dots in its name. The part that matters for this page is the prometheus job, because it turns on native histograms for that one target:
sudo vim /etc/prometheus/prometheus.ymlThis is the exact file the lab ran with, unchanged from the box.
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
- job_name: prometheus
scrape_native_histograms: true
always_scrape_classic_histograms: true
static_configs:
- targets: ["localhost:9090"]
- job_name: node
static_configs:
- targets: ["localhost:9100"]
- job_name: app
static_configs:
- targets: ["localhost:9200"]scrape_native_histograms: true is the switch for native histograms now, and the old native-histograms feature flag does nothing on current releases. I also kept always_scrape_classic_histograms: true so the same requests land in both formats, which is how I could compare them side by side. Drop it on a real server once your dashboards have moved: the classic format costs a series per bucket (twelve per handler here, counting _sum and _count), while the native one is a single series.
Every query output on this page came from promtool query instant, which ships in the Prometheus tarball and talks to the HTTP API. It is quicker than the browser for checking one expression, and you can paste the result into a ticket:
promtool query instant http://localhost:9090 'up'All three targets answered, and the number after @ is the evaluation time as a Unix timestamp.
up{instance="localhost:9100", job="node"} => 1 @[1793615474.885]
up{instance="localhost:9090", job="prometheus"} => 1 @[1793615474.885]
up{instance="localhost:9200", job="app"} => 1 @[1793615474.885]One thing promtool won’t show you is PromQL’s info and warning annotations. They only come back in the JSON from /api/v1/query, which matters for the rate-on-a-gauge mistake covered in the errors section.
Selecting series with label matchers
A selector is a metric name plus label matchers. There are four matcher operators:
=equals,!=does not equal=~matches a regex,!~does not match it
Regex matchers are anchored at both ends, so fstype=~"ext" matches nothing on an ext4 box. You need "ext.*" or the full value. The regex form against my filesystems shows it:
promtool query instant http://localhost:9090 'node_filesystem_size_bytes{fstype=~"ext4|vfat"}'Three filesystems matched, the tmpfs mounts didn’t, and each series still carries every label node_exporter attached:
node_filesystem_size_bytes{device="/dev/sda1", fstype="ext4", instance="localhost:9100", job="node", mountpoint="/"} => 19682557952 @[1793615474.999]
node_filesystem_size_bytes{device="/dev/sda15", fstype="vfat", instance="localhost:9100", job="node", mountpoint="/boot/efi"} => 109395456 @[1793615474.999]
node_filesystem_size_bytes{device="/dev/sda16", fstype="ext4", instance="localhost:9100", job="node", mountpoint="/boot"} => 923156480 @[1793615474.999]Metric names with dots are where Prometheus 3 changed the syntax. UTF-8 names are on by default, but a name that isn’t a legal classic identifier has to be quoted and moved inside the braces. I first tried the obvious form and got a parse error that points at the dot:
query error: bad_data: invalid parameter "query": 1:9: parse error: unexpected character: '.'The quoted form works, and quoted label names with dots work the same way:
promtool query instant http://localhost:9090 'rate({"app.jobs.processed", "queue.name"="emails"}[1m])'My demo exporter adds 3 jobs a second, and the rate agrees with it.
{instance="localhost:9200", job="app", queue.name="emails"} => 2.9999999999999996 @[1793615475.158]Getting that metric into Prometheus was harder than querying it. My first attempt dropped it into node_exporter’s textfile collector, and node_exporter refused the file twice: once with invalid metric name in comment for the unquoted # HELP line, then with invalid metric name "app.jobs.processed" once I quoted it. I gave up on the textfile route and served the metric from a 15-line Python exporter instead. If you’re planning dotted names from OpenTelemetry, check that every hop in your pipeline accepts them before you rename anything.
Choosing between rate, irate and increase
All three take a counter and a range, and they answer different questions. rate() is the per-second average over the whole window, irate() uses only the last two samples in it, and increase() is the total growth over the window. I ran them back to back on the same interface:
promtool query instant http://localhost:9090 'rate(node_network_receive_bytes_total{device="eth0"}[5m])'
promtool query instant http://localhost:9090 'irate(node_network_receive_bytes_total{device="eth0"}[5m])'
promtool query instant http://localhost:9090 'increase(prometheus_http_requests_total{handler="/api/v1/query"}[5m])'The terminal below has all five commands from this section with their output, including the two short-window runs further down:
Same five minutes, same counter, and rate said 1,728 bytes/s while irate said 2,261. Which one is right? Neither is wrong. irate just reported the last 15 seconds, which happened to be busier. That’s why I keep irate for zoomed-in graphs and never alert on it: one quiet or busy scrape flips the answer. The increase result also had a fraction in it (63.1578947368421 requests with a 400 code), which surprises people the first time. Counters only grow in whole numbers, but increase extrapolates to the edges of the window, so fractions are normal.
A graph shows the difference better than single numbers. I ran a 6-second CPU burst every 46 seconds and plotted both functions in one panel, using label_replace to name each line and or to join them. Paste this into the query page’s Graph tab:
label_replace(sum(rate(node_cpu_seconds_total{mode="user"}[5m])), "query", "rate 5m", "", "")
or
label_replace(sum(irate(node_cpu_seconds_total{mode="user"}[5m])), "query", "irate 5m", "", "")Fifteen minutes of it, with the bursts starting at 10:35:
The green irate line jumps to about 0.4 CPU-seconds per second on most bursts and falls back to almost zero between them, and a few bursts show up at half height because they straddled two scrapes. The 5-minute rate climbs slowly and settles near 0.13. Both are correct. If an alert has to catch a 6-second burst, neither line is the answer. A shorter window or max_over_time on a subquery is.
How short can the range be?
Prometheus 3 made range selectors left-open: a sample sitting exactly on the window’s start no longer counts. With my 15-second scrape interval, a [15s] window holds one sample, and a rate needs two. The query doesn’t fail, it just returns nothing:
promtool query instant http://localhost:9090 'rate(node_cpu_seconds_total{cpu="0", mode="user"}[15s])'
promtool query instant http://localhost:9090 'rate(node_cpu_seconds_total{cpu="0", mode="user"}[30s])'The first command printed an empty line and exited 0, while the second gave a value.
{cpu="0", instance="localhost:9100", job="node", mode="user"} => 0.006000000000000227 @[1793615475.439]Two scrape intervals is the floor, and I’d use four in anything you alert on, so one missed scrape doesn’t blank the panel. In Grafana, $__rate_interval does that arithmetic for you, at least four times the scrape interval, as long as the scrape interval set on the data source matches the real one.
Whole-number increases with anchored
The anchored modifier stops the extrapolation, so increase() returns whole numbers. On the current release it sits behind --enable-feature=promql-extended-range-selectors, and without the flag the parser says anchored modifier is experimental and not enabled. With the flag on, I ran both forms 40 ms apart:
promtool query instant http://localhost:9090 'increase(prometheus_http_requests_total{handler="/api/v1/query", code="200"}[5m] anchored)'
promtool query instant http://localhost:9090 'increase(prometheus_http_requests_total{handler="/api/v1/query", code="200"}[5m])'Anchored came back as 559 and the plain one as 572.63.
{code="200", handler="/api/v1/query", instance="localhost:9090", job="prometheus"} => 559 @[1793615581.703]
{code="200", handler="/api/v1/query", instance="localhost:9090", job="prometheus"} => 572.6315789473684 @[1793615581.743]For billing-style counts and “how many requests failed today” panels, the whole number is what people expect to see. Its sibling, smoothed (as in rate(x[5m] smoothed)), sits behind the same promql-extended-range-selectors flag, but I did not run a smoothed query on the lab box, so there is no output for it here. The release candidate for the next minor version lists anchored and smoothed selectors as stable, so the flag should go away soon, though I haven’t tested that build either.
Aggregating with sum by and without
by keeps only the labels you list, while without drops the ones you list and keeps the rest. I reach for without in recording rules, because labels like instance and job survive automatically, and by in dashboards where I want exactly one line per value. This is CPU time per mode across both cores:
promtool query instant http://localhost:9090 'sum by (mode) (rate(node_cpu_seconds_total[5m]))'The values are CPU-seconds per second, so idle near 2.0 on a 2-core VM means the box was almost asleep:
{mode="idle"} => 1.9562807017543857 @[1793615475.479]
{mode="iowait"} => 0.0007368421052631577 @[1793615475.479]
{mode="irq"} => 0 @[1793615475.479]
{mode="nice"} => 0 @[1793615475.479]
{mode="softirq"} => 0.0003157894736842106 @[1793615475.479]
{mode="steal"} => 0.00038596491228070164 @[1793615475.479]
{mode="system"} => 0.015824561403508762 @[1793615475.479]
{mode="user"} => 0.022842105263157858 @[1793615475.479]The order of operations matters more than the syntax. Always rate() first and sum() second, because summing raw counters and then taking a rate breaks as soon as one of them resets. For “top N” views, topk on the aggregated result is what you want:
promtool query instant http://localhost:9090 'topk(3, sum by (handler) (rate(prometheus_http_requests_total[5m])))'My load script hammered /api/v1/query, so it tops the list.
{handler="/api/v1/query"} => 1.6771929824561402 @[1793615475.636]
{handler="/api/v1/query_range"} => 0.1087719298245614 @[1793615475.636]
{handler="/metrics"} => 0.06666666666666667 @[1793615475.636]Without the topk, a plain max by (handler) returned 64 rows, most of them 0 for handlers nobody had called. Appending > 0 cut it to the nine that had traffic, which is the quickest cleanup for a noisy table panel. Remember too that quantile() works across series at one instant. For a percentile of one series over time you want quantile_over_time().
Matching two vectors with on, ignoring and group_left
A binary operator between two vectors pairs series whose labels are identical. When they aren’t, you tell it which labels to compare with on() or ignoring(), and when one side has several series per match, you say which side with group_left or group_right. The common case is copying a label from an info metric:
promtool query instant http://localhost:9090 'node_filesystem_avail_bytes{mountpoint="/"} * on (instance) group_left (nodename, release) node_uname_info'node_uname_info is always 1, so the value is unchanged and the result gains the host name and kernel:
{device="/dev/sda1", fstype="ext4", instance="localhost:9100", job="node", mountpoint="/", nodename="box1", release="6.8.0-106-generic"} => 16320614400 @[1793615476.08]My first try at a 4xx-to-200 ratio per handler failed. I wrote rate(...{code=~"4.."}) / ignoring (code) rate(...{code="200"}), and the left side had two series per handler (400 and 422), so Prometheus refused with many-to-one matching must be explicit (group_left/group_right). Summing the codes away on the left first fixed it:
promtool query instant http://localhost:9090 'sum without (code) (rate(prometheus_http_requests_total{code=~"4.."}[5m])) / ignoring (code) rate(prometheus_http_requests_total{code="200"}[5m])'The query_range handler shows 15.5 errors per success because my load script sends it a broken start parameter on purpose:
{handler="/api/v1/query", instance="localhost:9090", job="prometheus"} => 0.19718309859154928 @[1793615971.124]
{handler="/api/v1/query_range", instance="localhost:9090", job="prometheus"} => 15.522300371533767 @[1793615971.124]Set operators skip the value maths and only compare label sets. The three behave like this in the lab:
or vector(0): keeps a panel at 0 instead of “No data”and: kept the three writable filesystemsunless:up unless up{job="node"}left the prometheus and app targets
With up{job="missing"} or vector(0) the lab returned {} => 0, an empty label set, which is fine for a single stat panel and useless anywhere you need to group by a label afterwards.
Reading histograms: classic buckets vs native
This is the finding from the intro, and the reason I’d check every latency panel you own. Prometheus exposes its own HTTP request duration as both a classic histogram (the _bucket series with an le label) and a native histogram (one series, no le). I asked both for the p95 of the same requests over the same 5 minutes, then asked for the mean:
promtool query instant http://localhost:9090 'histogram_quantile(0.95, sum by (le) (rate(prometheus_http_request_duration_seconds_bucket[5m])))'
promtool query instant http://localhost:9090 'histogram_quantile(0.95, sum(rate(prometheus_http_request_duration_seconds[5m])))'
promtool query instant http://localhost:9090 'histogram_avg(sum(rate(prometheus_http_request_duration_seconds[5m])))'Classic said 95 ms, native said 1.19 ms, and the mean of the same requests was 0.29 ms.
So which number is real? The classic one is an artifact. This histogram’s smallest bucket is le="0.1", and every request in the window fell into it, since that bucket divided by _count returned exactly 1. histogram_quantile assumes observations are spread evenly inside a bucket, so the 95th percentile of “somewhere between 0 and 100 ms” comes out as 95 ms, no matter how fast the requests actually were. Native histograms use exponential buckets that are fine enough at sub-millisecond scale to give a believable answer. If your classic p95 lands on a round fraction of a bucket’s upper bound, like 0.095 against le="0.1", you’re reading the bucket layout, not your latency.
Classic histograms stay honest only if you respect how the function reads them, because histogram_quantile works from bucket counts and nothing else. What I check on every classic latency query now:
sum by (le)in the aggregation, plus any split label such ashandler- Never
sum without (le): the quantile needs the buckets - A bucket boundary near the SLO, since the answer can’t be finer than the buckets
histogram_avgor_sumover_countnext to it as a sanity check
Native histograms drop the le step entirely, and they add functions the classic format can’t do directly:
promtool query instant http://localhost:9090 'histogram_fraction(0, 0.0002, sum(rate(prometheus_http_request_duration_seconds[5m])))'About 73% of requests finished in under 200 microseconds.
{} => 0.7284007851537232 @[1793615476.468]For a classic-bucket SLO the equivalent is one bucket divided by _count, as in the ready-made table. On my box le="0.1" divided by the count returned exactly 1, which is true and also tells you nothing, for the same bucket-layout reason.
Using the functions I reach for most
Most of the function list is niche. These are the ones that show up in my alerts and dashboards, with what each returned in the lab:
absent(up{job="mysql"}):{job="mysql"} => 1, so an alert fires when a whole job disappearspredict_linear(node_filesystem_avail_bytes{mountpoint="/"}[10m], 24 * 3600): 13.4 GB left in 24 h, down from 16.3 GB, while a test loop filled the diskderiv(...[10m])on the same series: -34,232 bytes per secondtime() - node_boot_time_seconds: 1,144.8 seconds of uptimelabel_replace(up, "host", "$1", "instance", "(.*):.*"): addedhost="localhost"to all three targetsmax_over_time(rate(node_network_receive_bytes_total{device="eth0"}[1m])[10m:30s]): peak 1-minute rate of 98,979 bytes/s in the last 10 minutes
The subquery is worth learning. A plain 5-minute rate hides short bursts, and the subquery answers “what was the worst minute in the last ten” without a recording rule. Use deriv and delta on gauges only, because on counters they ignore resets and give nonsense after a restart.
Duration arithmetic now works without a flag. rate(node_cpu_seconds_total[5m * 2]) and offset (10m / 2) both ran on a stock install, which is handy in templated queries. The experimental functions are a different story, because they stay behind --enable-feature=promql-experimental-functions. The ones I tried with the flag on:
limitk(2, ...): two arbitrary series, cheaper thantopksort_by_label(..., "mountpoint"): rows in label ordermad_over_time(node_load1[15m]): median absolute deviation, 0.165ts_of_max_over_time(node_load1[15m]): Unix time of the peakdouble_exponential_smoothing(...[10m], 0.3, 0.3): the renamedholt_winters
Without the flag, Prometheus says exactly which one it needs:
query error: bad_data: invalid parameter "query": 1:1: parse error: limitk() is experimental and must be enabled with --enable-feature=promql-experimental-functionsIf you have old dashboards using holt_winters, they break on Prometheus 3 even with the flag. It was renamed to double_exponential_smoothing, and the old name now fails with unknown function with name "holt_winters". To turn the flags on, edit your existing unit rather than replacing it, because the unit from our install guide also sets retention, the listen address and the OTLP receiver, and losing --storage.tsdb.retention.time drops you back to the 15-day default on the next restart:
sudo vim /etc/systemd/system/prometheus.serviceAppend this one argument to the end of your ExecStart line. If the line is split, end the previous line with a \ first:
--enable-feature=promql-experimental-functions,promql-extended-range-selectorsFor reference, this was the whole unit on my lab box, which is deliberately minimal. Don’t paste it over a production unit:
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--web.enable-lifecycle \
--enable-feature=promql-experimental-functions,promql-extended-range-selectors
Restart=on-failure
[Install]
WantedBy=multi-user.targetReload systemd and restart the service. The journal confirms both flags took effect:
sudo systemctl daemon-reload
sudo systemctl restart prometheus
journalctl -u prometheus --no-pager | grep -i "enabled"Both lines appeared right after the restart.
Nov 02 10:32:40 box1 prometheus[53209]: time=2026-11-02T10:32:40.024Z level=INFO source=main.go:263 msg="Experimental PromQL functions enabled."
Nov 02 10:32:40 box1 prometheus[53209]: time=2026-11-02T10:32:40.024Z level=INFO source=main.go:316 msg="Experimental PromQL extended range selectors enabled."Experimental means the names and behaviour can change between releases, so I would keep these out of alert rules and use them in ad hoc queries until they graduate.
Writing recording rules
A recording rule evaluates an expression on a schedule and stores the result as a new series. I use them for anything a dashboard or alert queries over and over, and for expressions long enough that I don’t want them copied into five places. The naming convention is level:metric:operations. Create the file:
sudo mkdir -p /etc/prometheus/rules
sudo vim /etc/prometheus/rules/recording.ymlThese four rules are the exact file I tested. The CPU rule drops both cpu and mode, so the result carries only instance and job:
groups:
- name: node-recording
interval: 30s
rules:
- record: instance:node_cpu_utilisation:rate5m
expr: 1 - avg without (cpu, mode) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
- record: instance:node_memory_utilisation:ratio
expr: 1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
- record: job:prometheus_http_requests:rate5m
expr: sum by (job, code) (rate(prometheus_http_requests_total[5m]))
- record: job:prometheus_http_request_duration_seconds:p95_5m
expr: histogram_quantile(0.95, sum by (job, le) (rate(prometheus_http_request_duration_seconds_bucket[5m])))My first version used avg without (cpu), which left a stray mode="idle" label on the recorded series. It worked, but every alert built on it inherited that label. Prometheus picks up rule files on a reload. My unit runs with --web.enable-lifecycle, so the HTTP endpoint works; without that flag, use sudo systemctl restart prometheus instead:
curl -s -X POST http://localhost:9090/-/reload
promtool query instant http://localhost:9090 'instance:node_cpu_utilisation:rate5m'The recorded series is queryable like any other, carrying only instance and job:
instance:node_cpu_utilisation:rate5m{instance="localhost:9100", job="node"} => 0.07226315789473658 @[1793616873.813]The last rule records the classic p95, and it dutifully stored 0.09500000000000001. A recording rule makes a bad query cheaper, not correct, so fix the histogram first.
Writing alert rules and testing them with promtool
Alert rules add for (how long the condition must hold before it fires), labels for routing and annotations for the message. keep_firing_for holds an alert open for a while after the condition clears, which stops a flapping CPU alert from paging you three times in ten minutes. Create the alerts file:
sudo vim /etc/prometheus/rules/alerts.ymlThese are the four alerts from the lab, unchanged. The CPU alert builds on the recording rule above, and the disk alert combines a forecast with a floor, so a disk that’s 90% free never pages you just because a log rotated:
groups:
- name: node-alerts
rules:
- alert: InstanceDown
expr: up == 0
for: 2m
labels:
severity: critical
annotations:
summary: "{{ $labels.instance }} ({{ $labels.job }}) is down"
description: "The target has failed every scrape for 2 minutes."
- alert: HighCpuUsage
expr: instance:node_cpu_utilisation:rate5m > 0.85
for: 10m
keep_firing_for: 5m
labels:
severity: warning
annotations:
summary: "CPU on {{ $labels.instance }} at {{ $value | humanizePercentage }}"
- alert: DiskWillFillIn24h
expr: |
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24 * 3600) < 0
and
node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes < 0.25
for: 30m
labels:
severity: warning
annotations:
summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} fills within 24h"
- alert: HighHttpErrorRatio
expr: |
sum by (job) (rate(prometheus_http_requests_total{code=~"5.."}[5m]))
/
sum by (job) (rate(prometheus_http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }} 5xx ratio is {{ $value | humanizePercentage }}"Unit tests live in their own directory. My first run put the test file in rules/, where the rule_files glob would have tried to load it as a rule file and failed the next reload, so keep it out of that path:
sudo mkdir -p /etc/prometheus/tests
sudo vim /etc/prometheus/tests/alerts_test.ymlThe test feeds synthetic series in and states what should fire and when. The second case checks the recording rule’s value (an idle counter growing 6 seconds per minute means 90% busy) and the rendered annotation:
rule_files:
- ../rules/recording.yml
- ../rules/alerts.yml
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
- series: 'up{job="node", instance="box2:9100"}'
values: '1 1 0 0 0 0'
alert_rule_test:
- eval_time: 3m
alertname: InstanceDown
exp_alerts: []
- eval_time: 5m
alertname: InstanceDown
exp_alerts:
- exp_labels:
severity: critical
job: node
instance: box2:9100
exp_annotations:
summary: "box2:9100 (node) is down"
description: "The target has failed every scrape for 2 minutes."
- interval: 1m
input_series:
- series: 'node_cpu_seconds_total{instance="box1:9100", cpu="0", mode="idle"}'
values: '0+6x30'
promql_expr_test:
- expr: instance:node_cpu_utilisation:rate5m
eval_time: 10m
exp_samples:
- labels: 'instance:node_cpu_utilisation:rate5m{instance="box1:9100"}'
value: 0.9
alert_rule_test:
- eval_time: 25m
alertname: HighCpuUsage
exp_alerts:
- exp_labels:
severity: warning
instance: box1:9100
exp_annotations:
summary: "CPU on box1:9100 at 90%"Check the syntax of both rule files, then run the tests. The cd is only there for the short paths on the command line, because promtool resolves the test file’s rule_files relative to the test file itself:
cd /etc/prometheus
promtool check rules rules/recording.yml rules/alerts.yml
promtool test rules tests/alerts_test.ymlBoth files parsed with four rules each, and the tests passed.
Would the test catch a wrong expectation? I changed the expected summary to 95% and ran it again:
The diff shows the expected and actual alert side by side, and the command exited 1, so it works as a CI gate. Put promtool check rules and promtool test rules in the pipeline that ships your rule files, and a typo never reaches a reload. A syntax error looks like this, with the line, the rule and the column:
Checking /tmp/broken.yml
FAILED:
/tmp/broken.yml: 5:15: group "broken", rule 1, "LoadHigh": could not parse expression: 1:24: parse error: unclosed left parenthesisTo see a real alert fire, I stopped the demo exporter on port 9200. InstanceDown went to pending on the next evaluation and to firing two minutes later. Open Alerts in the Prometheus UI and expand the rule:
- 1InstanceDown rule, firing
- 2The down target and its labels
- InstanceDown rule, firing: the rule row turns red with a firing count.
- The down target and its labels: the alert carries
instance="localhost:9200"andjob="app", the labels Alertmanager routes on.
The rule panel shows the raw annotation template, and the rendered summary only appears in the alert itself and in Alertmanager. Querying ALERTS gives the same state from PromQL, which is what I use in Grafana panels that list everything currently firing across a fleet.
Fixing common PromQL errors
These came up in the lab, roughly in the order I hit them while building this page. The two vector errors are in the capture below, and the rest are quoted as Prometheus returned them.
Each heading below is the message as you’ll see it, followed by the cause and the fix I used.
vector cannot contain metrics with the same labelset
I got this from label_replace(rate(node_cpu_seconds_total{mode="idle"}[5m]), "cpu", "all", "cpu", ".*"). Rewriting cpu to one value made both cores’ series identical, and a result can’t hold two series with the same labels. The same error shows up when a function drops __name__ from a selector that matched several metrics, such as rate({__name__=~"node_network_(receive|transmit)_bytes_total"}[5m]). Aggregate first so only one series is left per label set:
promtool query instant http://localhost:9090 'label_replace(sum without (cpu) (rate(node_cpu_seconds_total{mode="idle"}[5m])), "cpu", "all", "", "")'That leaves one series per mode, now labelled cpu="all".
{cpu="all", instance="localhost:9100", job="node", mode="idle"} => 1.8510526315789462 @[1793615971.162]For the multi-metric case, query each metric separately or sum by (__name__) before anything strips the name.
many-to-one matching must be explicit (group_left/group_right)
node_filesystem_avail_bytes * on (instance) node_uname_info fails because the left side has one series per filesystem and the right side one per host. Add group_left (the many side is on the left) and name the labels to copy, as in the vector matching section. If both sides have several series per match, that’s many-to-many and no modifier fixes it: aggregate one side down first.
PromQL info: metric might not be a counter
rate(node_load1[5m]) returned a number and promtool printed no warning, but load average is a gauge and its rate means nothing. The API flags it:
curl -s localhost:9090/api/v1/query --data-urlencode 'query=rate(node_load1[5m])' | python3 -m json.toolThe result is a value plus an infos entry that nobody sees unless they look at the JSON:
{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{
"metric": {
"instance": "localhost:9100",
"job": "node"
},
"value": [
1793615954.048,
"0.005778725614035086"
]
}
]
},
"infos": [
"PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: \"node_load1\" (1:6)"
]
}Use deriv() or delta() for gauges. The check is name-based, though: my real counter app.jobs.processed got the same info because its name doesn’t end in _total. Treat it as a hint, not proof.
An empty result with no error
The quiet one. A range window shorter than two scrape intervals ([15s] at a 15-second scrape) returns nothing and exits 0. So do typos in label values, since a selector that matches no series isn’t an error. When a panel says “No data”, run the bare selector without functions first. If that’s empty too, the problem is the labels, not the maths.
parse error: unexpected character: ‘.’
This is an unquoted UTF-8 metric name, and the fix is to quote it and put it inside the braces, as in {"app.jobs.processed"}.
If you want the theory behind all of this, Prometheus: Up & Running (2nd edition) (paid link) by Julien Pivotto and Brian Brazil is the book I’d hand a new on-call engineer. It covers PromQL, rules and Alertmanager in more depth than any cheat sheet can, though it predates Prometheus 3, so UTF-8 names and stable native histograms come from this page and the docs. Our roundup of Prometheus and Grafana books compares it with the alternatives. For exporters to point these queries at, the node_exporter queries above follow the same patterns as our guide to monitoring Linux server uptime and health. The MySQL and MariaDB exporter setup, the Redis monitoring guide, the Nginx monitoring guide, the PostgreSQL exporter setup and the Docker container stack with cAdvisor all feed the same patterns, and on Kubernetes the kube-prometheus-stack install ships recording rules in exactly this level:metric:operations style. The full function list is in the official PromQL functions reference, and the Prometheus 3 migration guide lists every query behaviour that changed.
What I still don’t know
I haven’t run native histograms under real production load, so I can’t tell you what the extra bucket resolution costs in TSDB size on a busy server. On this box the difference was lost in the noise. I also don’t know whether node_exporter’s textfile collector will learn UTF-8 names, or whether a flag I missed already allows them. And I’d like to know how many teams are alerting on a classic p95 that is really just a bucket boundary. After this lab, my first move on any inherited Prometheus is to check every latency panel against histogram_avg and the bucket layout, because a number that looks plausible is the hardest kind of wrong to notice.
