Monitoring

Install Grafana on Debian 13 and Ubuntu 26.04 with Nginx and SSL

Grafana’s Debian package is a 397 MB download that unpacks to 1.4 GB and starts 14 plugin backends on first start. It also listens on port 3000 on every interface, over plain HTTP, with admin/admin as the login. So we bind it to localhost and put Nginx with a Let’s Encrypt certificate in front before anyone logs in.

Original content from computingforgeeks.com - post 2375

This guide takes you through the full path to install Grafana on Debian and Ubuntu. It covers the Grafana repository, Prometheus on the same host, the Nginx reverse proxy with TLS and a working Node Exporter dashboard. We also measured idle RAM and what one setting gives back. In November 2026 we re-ran the install, the stack, an upgrade with rollback and the tuning on a fresh Debian 13 VM with Grafana 13.3.0, after the first run on Debian 13 and Ubuntu 26.04 in October. The lab served a lab CA certificate where you will have a Let’s Encrypt one.

Prerequisites

Grafana itself is light on CPU, so the sizing question is memory and disk. The package needs about 1.4 GB on disk before any data, and Grafana with its bundled plugin processes sat at roughly 700 MB of resident memory on our idle lab VM. Prometheus is the part that grows, because its RAM and disk scale with active series times retention. A single host scraping itself needs very little, while a server scraping a few hundred nodes wants its own machine. For this guide you need the following.

  • Ubuntu 26.04 LTS or Debian 13 with sudo access
  • 2 vCPU, 2 GB RAM and 10 GB of free disk for Grafana, Prometheus and Node Exporter on one host
  • A domain or subdomain with an A record pointing at the server (any DNS provider works)
  • TCP ports 80 and 443 reachable from the internet for the Let’s Encrypt HTTP-01 challenge

Our lab VM had 2 vCPU and 4 GB of RAM, which is a floor for following along, not a production recommendation. If you’re still deciding how far to take the Prometheus side, our roundup of the best Grafana and Prometheus books covers the ones worth reading before you size a real monitoring server. Grafana has no hard dependency on Prometheus. If you already run Prometheus elsewhere, you can skip its install step and point the data source at that server instead.

Set the shell variables

The domain shows up in the Grafana config, the certificate request and the Nginx file, so set it once along with the contact address Certbot registers on your ACME account. Replace both values with your own.

export SITE_DOMAIN="grafana.example.com"
export ADMIN_EMAIL="[email protected]"

These only live in the current shell. If you reconnect or switch to a root shell with sudo -i, run the two export lines again. Otherwise the later sed and certbot commands run with an empty domain.

Add the Grafana APT repository

Debian and Ubuntu do not package Grafana, so it comes from Grafana Labs’ own repository at apt.grafana.com. The same stable main line works on both distros because the repository is not tied to a release codename.

Running only Ubuntu 26.04? The current Grafana guide for Ubuntu 26.04 adds the same repository and goes further on that release, with both the archive and upstream Prometheus builds, email alerts and a PostgreSQL backend. Stay here if you want one set of steps for Debian and Ubuntu with Nginx in front, and start with the tools the repository setup needs:

sudo apt update
sudo apt install -y apt-transport-https wget gnupg

Download the signing key into /etc/apt/keyrings and make it world-readable so apt can use it when it verifies the repository. These are the exact steps from Grafana’s install docs.

sudo mkdir -p /etc/apt/keyrings
sudo wget -q -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc

Add the repository and refresh the package index.

echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update

Before installing, confirm apt now sees a candidate from apt.grafana.com. An empty Candidate line means the repository line or key is wrong, so fix that before going further.

apt-cache policy grafana | head -4

On our Debian 13 VM the stable repository offered this release:

grafana:
  Installed: (none)
  Candidate: 13.3.0
  Version table:

Ubuntu 26.04 reads the same repository line, and in our October run both distros were offered the identical build, so nothing below changes between the two until the Prometheus packages.

Install Grafana on Debian or Ubuntu

The grafana package is the open source build. Grafana Labs also publishes grafana-enterprise, and its docs now list Enterprise as the default edition. We use the OSS package here because nothing in this guide needs Enterprise, and it keeps the install fully open source. Install it:

sudo apt install -y grafana

Expect this to take a while. On our November Debian run apt fetched 397 MB and the install took about two and a half minutes, mostly download time, and /usr/share/grafana ended up at 1.4 GB. The package does not start the service on its own. Reload systemd, then enable and start it in one go.

sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server

Check the version with sudo. The grafana binary lives in /usr/sbin, which is not on a regular user’s PATH on Debian. Without sudo you get grafana: command not found there, even though the install worked.

sudo grafana server -v

The output names the installed release:

Version 13.3.0 (commit: 6f8ce3c693b493043ebcd4787566d2b6b118b0f0, branch: release-13.3.0#patched)

The older grafana-server -v still answers, but it now prints a deprecation warning pointing you to the grafana server subcommand, so update any scripts that call it. Give the service a minute or two before you test it, or you’ll chase a problem that isn’t there. On the very first start Grafana builds its SQLite database. Our October Debian log showed 719 migrations taking 23.7 seconds, and the November package ran 728 of them in 1 minute 14 seconds on a slower lab disk. Port 3000 stays closed until they finish, so an immediate check looks like a failed install when it is not.

sudo journalctl -u grafana-server | grep 'logger=migrator.*migrations completed'
curl -s http://127.0.0.1:3000/api/health

This is the November Debian VM after one later restart, so the journal holds two lines: the first start that ran all 728 migrations, and a restart that skipped them. A healthy instance then reports the database as ok along with its version.

Nov 05 21:52:26 mon01 grafana[2918]: logger=migrator t=2026-11-05T21:52:26.202552836Z level=info msg="migrations completed" performed=728 skipped=0 duration=1m13.659331968s
Nov 05 21:53:35 mon01 grafana[6110]: logger=migrator t=2026-11-05T21:53:35.381913082Z level=info msg="migrations completed" performed=0 skipped=728 duration=513.66µs
{
  "database": "ok",
  "version": "13.3.0",
  "commit": "6f8ce3c693b493043ebcd4787566d2b6b118b0f0"
}

The two distros behaved the same apart from a few details that matter when you script this. The Prometheus side differs more than Grafana does, because it comes from each distro’s own archive. The table is the October side-by-side; where the November re-run on Debian moved a number, it sits next to the October one.

ItemDebian 13Ubuntu 26.04
Grafana package from apt.grafana.com13.2.4 (Oct), 13.3.0 (Nov)13.2.4 (Oct)
grafana on a normal user’s PATHNo, run it with sudoYes, /usr/sbin is on PATH
First-start database migrations719 in 23.7 s (Oct), 728 in 73.7 s (Nov)719 in 10.4 s (Oct)
Prometheus from the distro archive2.53.32.53.5
node_exporter from the distro archive1.9.01.10.2
ufwNot installedInstalled
Grafana resident memory at idle (14 processes)692 MiB639 MiB
Whole VM in use at idle (MemTotal minus MemAvailable)746 MiB806 MiB

The migration time is disk and CPU bound, not a distro trait, so read it as “expect anywhere from 10 seconds to over a minute” rather than a reason to pick one OS. If curl returns nothing after two minutes, read sudo journalctl -u grafana-server -e. A syntax error in /etc/grafana/grafana.ini and another process on port 3000 are the usual causes at this stage.

Install Prometheus and Node Exporter

Grafana only draws what a data source gives it, so before you can see a dashboard you need Prometheus and a node exporter to scrape. Both distros ship them in their main repositories. The packaged prometheus.yml already scrapes the local node exporter on port 9100 under the job name node, which is exactly what the Node Exporter Full dashboard expects. Install both:

sudo apt install -y prometheus prometheus-node-exporter

The packages start both services straight away, and both listen on every interface. Prometheus has no authentication, so on a server with a public address anyone could query it on port 9090. Only Grafana on the same host needs it, so bind both to the loopback address through their /etc/default files.

sudo sed -i 's|^ARGS=""|ARGS="--web.listen-address=127.0.0.1:9090"|' /etc/default/prometheus
sudo sed -i 's|^ARGS=""|ARGS="--web.listen-address=127.0.0.1:9100"|' /etc/default/prometheus-node-exporter
sudo systemctl restart prometheus prometheus-node-exporter

Give the restart a few seconds before checking, because Prometheus binds its port a moment after the exporter does (on Ubuntu our first ss right after the restart listed only port 9100). Both should then show 127.0.0.1 as the local address instead of *:

sudo ss -tlnp | grep -E '9090|9100'

On Debian 13 the output looked like this.

LISTEN 0      4096       127.0.0.1:9100      0.0.0.0:*    users:(("prometheus-node",pid=12512,fd=3))
LISTEN 0      4096       127.0.0.1:9090      0.0.0.0:*    users:(("prometheus",pid=12517,fd=6))

Give Prometheus 20 seconds to complete a scrape, then ask it which targets it sees. Both should report up. A down node target means the exporter is not running or the port in prometheus.yml does not match.

curl -s http://127.0.0.1:9090/api/v1/targets | grep -o '"health":"[a-z]*"'

Two healthy targets print two lines.

"health":"up"
"health":"up"

The distro packages trail the upstream Prometheus release (the versions are in the table above), which is fine for a dashboard backend. If you want the current upstream build with its own service user, follow the Prometheus install guide for Ubuntu and Debian instead and come back for the Grafana side.

Bind Grafana to localhost and set its public URL

With Nginx in front, nothing outside the server should reach port 3000 directly. Grafana also has to know the HTTPS address it is served from, so redirects and links in alert notifications point to the right place. Three keys in the [server] section of /etc/grafana/grafana.ini handle that. They ship commented out, so this sed uncomments and sets them from your variable:

sudo sed -i -e "s|^;http_addr =.*|http_addr = 127.0.0.1|" \
  -e "s|^;domain = localhost|domain = ${SITE_DOMAIN}|" \
  -e "s|^;root_url = .*|root_url = https://${SITE_DOMAIN}/|" /etc/grafana/grafana.ini

Check that all three lines changed, then restart Grafana.

sudo grep -nE '^(http_addr|domain|root_url) ' /etc/grafana/grafana.ini
sudo systemctl restart grafana-server

You should see three uncommented lines with your own domain in them.

48:http_addr = 127.0.0.1
54:domain = grafana.example.com
62:root_url = https://grafana.example.com/

If a line is missing, the file was already edited and the sed pattern did not match. Open it with sudo vim /etc/grafana/grafana.ini and set the key by hand under [server]. The restart is quick this time because the database migrations already ran, and sudo ss -tlnp | grep 3000 should now show 127.0.0.1:3000.

Get a Let’s Encrypt certificate

Certbot with the Nginx plugin answers the HTTP-01 challenge through Nginx itself, so it works with any DNS provider as long as your A record resolves to this server and port 80 is reachable. Install Nginx and Certbot.

sudo apt install -y nginx certbot python3-certbot-nginx

Turn on the firewall next, before the certificate request, with SSH, HTTP and HTTPS open. Ubuntu already ships ufw and Debian does not install it, so the install line covers both. Allow OpenSSH first so enabling the firewall over an SSH session does not lock you out. If it ever does, the cloud or hypervisor console still works and sudo ufw disable gets you back in. The OpenSSH profile only covers port 22, so if your SSH daemon listens on another port, allow that port instead (for example sudo ufw allow 2222/tcp) before you enable the firewall.

sudo apt install -y ufw
sudo ufw allow OpenSSH
sudo ufw allow "Nginx Full"
sudo ufw --force enable

Ports 3000, 9090 and 9100 stay closed because nothing in that list opens them, and the services only listen on localhost anyway. The UFW firewall guide covers rate-limiting SSH and other rules worth adding on a public box. Now request the certificate. We use certonly so Certbot only fetches the certificate and leaves the Nginx config to us. The deploy hook reloads Nginx after every renewal, so a renewed certificate is actually served.

sudo certbot certonly --nginx -d "${SITE_DOMAIN}" --non-interactive --agree-tos -m "${ADMIN_EMAIL}" --deploy-hook "systemctl reload nginx"

On success Certbot leaves three things behind that the rest of the setup relies on:

  • /etc/letsencrypt/live/<your domain>/fullchain.pem, the certificate chain
  • /etc/letsencrypt/live/<your domain>/privkey.pem, the private key
  • the certbot.timer systemd unit that renews the certificate

If the request fails with an unauthorized or connection timeout message, Let’s Encrypt could not reach port 80 on the address your A record points to. Check the record with dig +short "${SITE_DOMAIN}", then any cloud security group or router port forward in front of the server. Prove renewal works before you forget about it.

sudo certbot renew --dry-run

The dry run uses the Let’s Encrypt staging environment and doesn’t replace your certificate, so it’s safe to repeat after any change to the Nginx config.

Alternative: DNS-01 for private servers

When the server sits on a private network or behind NAT with no inbound port 80, use a DNS-01 challenge instead. Certbot proves ownership by writing a TXT record through your DNS provider’s API, so nothing has to reach the server. Install the plugin for your provider and keep the rest of the guide the same.

DNS providerCertbot plugin packageDebian 13Ubuntu 26.04
Cloudflarepython3-certbot-dns-cloudflareYesYes
Amazon Route 53python3-certbot-dns-route53YesYes
Google Cloud DNSpython3-certbot-dns-googleYesYes
BIND or any RFC 2136 serverpython3-certbot-dns-rfc2136YesYes
DigitalOceanpython3-certbot-dns-digitaloceanNoYes
Linodepython3-certbot-dns-linodeNoYes

The DigitalOcean and Linode plugins are not in the Debian 13 archive, so on Debian you would install Certbot and that plugin from the snap or a pip virtual environment instead of apt. We checked the Ubuntu column with apt-cache policy on the lab VM and the Debian column against the Debian archive index.

With Cloudflare as the example, the token needs Zone DNS Edit permission and goes in a file only root can read. Creating the empty file with the right mode first keeps the token out of your shell history:

sudo apt install -y python3-certbot-dns-cloudflare
sudo install -m 600 /dev/null /etc/letsencrypt/cloudflare.ini
sudo vim /etc/letsencrypt/cloudflare.ini

Put one line in the file with your own token:

dns_cloudflare_api_token = your-token-here

Then request the certificate through the plugin. Substitute your provider’s plugin and credentials file if you use something else.

sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini -d "${SITE_DOMAIN}" --non-interactive --agree-tos -m "${ADMIN_EMAIL}" --deploy-hook "systemctl reload nginx"

A DNS-01 failure usually comes down to the token. One scoped to the wrong zone, or with read-only DNS permission, gets an authorization error from the provider API rather than from Let’s Encrypt. We did not run either Certbot path in the lab, because a lab VM cannot get a public certificate; it used a lab CA certificate at the same paths instead.

Configure Nginx as the reverse proxy

The Nginx site does three jobs, and the WebSocket one is easy to miss because the dashboards still load without it:

  • redirects plain HTTP on port 80 to HTTPS
  • terminates TLS and proxies to Grafana on 127.0.0.1:3000
  • passes Grafana Live WebSocket upgrades on /api/live/

Without the /api/live/ block, live panel updates fail quietly while everything else works, which makes it a confusing thing to debug later. Create the site file:

sudo vim /etc/nginx/sites-available/grafana.conf

Paste this configuration. SITE_DOMAIN_HERE is a placeholder that the next command replaces with your domain, so leave it as it is:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    listen [::]:80;
    server_name SITE_DOMAIN_HERE;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name SITE_DOMAIN_HERE;

    ssl_certificate     /etc/letsencrypt/live/SITE_DOMAIN_HERE/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/SITE_DOMAIN_HERE/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;

    location / {
        proxy_set_header Host $host;
        proxy_pass http://127.0.0.1:3000;
    }

    location /api/live/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_pass http://127.0.0.1:3000;
    }
}

Swap the placeholder for your domain, enable the site and test the syntax before reloading, so a typo never takes down an Nginx that serves other sites.

sudo sed -i "s/SITE_DOMAIN_HERE/${SITE_DOMAIN}/g" /etc/nginx/sites-available/grafana.conf
sudo ln -s /etc/nginx/sites-available/grafana.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Both lines of a good test end in ok and successful.

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Check the redirect and the proxied health endpoint from the server. The first request should return a 301 and the second the same JSON you saw from port 3000. On Ubuntu our very first request still got a 200 from the default site, because systemctl reload returns before the new Nginx workers take over. Run it again and the 301 appears.

curl -sI "http://${SITE_DOMAIN}/" | head -1
curl -s "https://${SITE_DOMAIN}/api/health"

A 502 Bad Gateway here means Nginx is up but Grafana isn’t answering on 127.0.0.1:3000, usually because it is still starting or the http_addr change did not take, and sudo ss -tlnp | grep 3000 tells you which. At this point every service should be active, with only Nginx facing the network and Grafana, Prometheus and Node Exporter on the loopback address:

Terminal showing Grafana version and Grafana, Prometheus and Node Exporter bound to localhost behind Nginx on Debian 13

If port 3000, 9090 or 9100 shows 0.0.0.0 or * instead, the matching sed step didn’t apply, and that service is reachable from outside whenever the firewall is off. The same proxy layout carries over if you later add Loki for logs, which our Grafana Loki setup for Ubuntu and Debian walks through.

Log in and change the admin password

Open https://grafana.example.com with your own domain in a browser. You get the Grafana login page over HTTPS, and the username and password are both admin.

Grafana login page served over HTTPS through the Nginx reverse proxy

Right after the first login Grafana asks for a new password. The form has a Skip button, and skipping it leaves admin/admin working on a server that is now reachable from the internet. Set a long password here and store it in your password manager.

Grafana prompting to change the default admin password after first login

After you submit, Grafana lands on the home page with an empty Recent dashboards panel and the navigation on the left. Dashboards, Connections and Alerting are the three menus you will use most.

Grafana home page after logging in on Debian 13 behind Nginx with HTTPS

If you lose the admin password later, reset it from the shell on the server. The CLI writes straight to the Grafana database, so it works even when nobody can log in.

sudo grafana cli admin reset-admin-password 'Use-A-Long-Passphrase-2026'

It prints a screen of startup log lines, because the CLI loads the full server config to reach the database, and ends with Admin password changed successfully. No restart is needed. We ran it on both lab VMs and then confirmed the new password against the API, which answered with HTTP 200.

Add Prometheus as a data source

Open Connections in the side menu, pick Data sources under it and click Add data source. On a fresh install the page is empty apart from that button, as in our Debian capture.

Grafana Connections menu with Data sources open and the Add data source button on Debian 13
  1. 1Connections
  2. 2Data sources
  3. 3Add data source

Type Prometheus in the filter box and click the Prometheus card. Grafana creates the data source and opens its settings page. The only field you must fill is Prometheus server URL. Enter http://127.0.0.1:9090, the loopback address Prometheus is now bound to.

Grafana Prometheus data source settings with the server URL set to 127.0.0.1:9090

Leave authentication at No Authentication, scroll to the bottom and click Save & test. Grafana runs a test query against the Prometheus API, and a green “Successfully queried the Prometheus API” box means the connection works.

Grafana Prometheus data source Save and test showing Successfully queried the Prometheus API

A red box with a connection refused error means Prometheus is not listening on that address, so compare the URL with the ss output from the Prometheus step. For a Prometheus server on another host, use its address and allow Grafana’s IP to reach port 9090 on that host’s firewall. Every pass through Add data source creates a new entry, and our retries left prometheus-1 and prometheus-2 in the list. Delete the extras so imported dashboards do not pick the wrong one.

Import the Node Exporter Full dashboard

Building panels by hand is the slow way to get started. The community dashboard Node Exporter Full (ID 1860 on grafana.com) already charts CPU, memory, disk, network and systemd metrics from the exporter you installed. Open Dashboards > New > Import, type 1860 in the Grafana.com dashboard URL or ID field below the grafana.com link and click the Load button beside it. The second Load button further down belongs to the JSON box and does nothing with an ID.

Grafana Import dashboard page with Node Exporter Full ID 1860 entered and the Load button
  1. 1Grafana.com dashboard URL or ID
  2. 2Load

Grafana downloads the dashboard JSON from grafana.com, so the server needs outbound HTTPS for this step. Keep the name and folder, click Import, and the dashboard opens with its data source, job and instance selectors already filled from the only Prometheus source on the server. Switch the time range to the last 30 minutes so the panels are not mostly empty on a fresh install.

Grafana Node Exporter Full dashboard showing live CPU, memory and network data from Prometheus

If you see panels that read “No data” while their neighbours draw fine, those are usually metrics the exporter does not collect on a VM, such as hardware temperatures, and can be ignored. If every panel is empty, the Job selector at the top does not match the job_name in /etc/prometheus/prometheus.yml. The packaged config uses node, which the dashboard picks up on its own. From here the next useful pieces are alert rules routed through the Prometheus Alertmanager and exporters for the services on the box, starting with Nginx metrics in Prometheus and Grafana since Nginx is already running here.

Provision the data source from a file

Clicking through the UI is fine for one server. When Grafana is built by Ansible or rebuilt often, define the data source in a YAML file instead, so a fresh install comes up already connected. This section is optional, and the dashboard above works without it. Create the provisioning file.

sudo vim /etc/grafana/provisioning/datasources/prometheus.yaml

Add the data source definition. Grafana reads every YAML file in this directory at startup, and editable: false locks the source against UI edits so the file stays the single source of truth:

apiVersion: 1

datasources:
  - name: Prometheus-provisioned
    type: prometheus
    access: proxy
    url: http://127.0.0.1:9090
    isDefault: true
    editable: false

Grafana runs as the grafana user and must be able to read the file, so give that group read access and restart the service:

sudo chown root:grafana /etc/grafana/provisioning/datasources/prometheus.yaml
sudo chmod 640 /etc/grafana/provisioning/datasources/prometheus.yaml
sudo systemctl restart grafana-server

We ran this on servers that already had a source created earlier, and Grafana did not complain about two defaults. It moved the default flag to the provisioned source and marked it read-only. Check with the API, using the admin password you set:

curl -s -u 'admin:Use-A-Long-Passphrase-2026' http://127.0.0.1:3000/api/datasources | python3 -m json.tool | grep -E '"(name|isDefault|readOnly)"'

On the Ubuntu VM the older source lost the default flag and the provisioned one is locked:

        "name": "prometheus",
        "isDefault": false,
        "readOnly": false
        "name": "Prometheus-provisioned",
        "isDefault": true,
        "readOnly": true

Dashboards you imported keep pointing at the source they were imported with, so on a server that started from the UI, either leave both in place or re-import the dashboard against the provisioned one before deleting the old source. A provisioned source cannot be edited in the UI. Change the YAML and restart instead, which is the point of putting it in a file.

Back up and upgrade Grafana

Grafana keeps its state in two places, and a useful backup takes both:

  • /var/lib/grafana, with grafana.db (users, data sources, dashboards, alert rules) and the plugins directory
  • /etc/grafana, with grafana.ini and the provisioning files

Copy both before you upgrade, because Grafana migrates the database on the first start of a new release and an older package is not guaranteed to run against it. Skip plugins-bundled: it took 456 MB of the 509 MB in /var/lib/grafana on our Ubuntu VM and the package puts it back on reinstall. The plugins directory held 51 MB of app plugins Grafana fetched on its own at first start. The package is close to a 400 MB download, so fetch it while Grafana is still running to keep the outage short:

sudo apt update
sudo apt install --download-only -y grafana

Then stop the service and take the backup:

sudo systemctl stop grafana-server
sudo mkdir -p /var/backups/grafana
sudo tar -czf /var/backups/grafana/grafana-$(date +%F-%H%M).tar.gz --exclude=/var/lib/grafana/plugins-bundled /var/lib/grafana /etc/grafana

Stopping the service first matters because copying a SQLite file while Grafana writes to it can capture a half-written page. Tar warns that it’s removing the leading / from member names, which is normal, and the archive came to 37 MB on our October Ubuntu VM, mostly those app plugins. The same command on the November Debian VM wrote 153 MB, so check free space under /var/backups before you rely on our first figure. The hour and minute in the name keep a second backup on the same day from overwriting the first. With the service still stopped, install the downloaded package and start Grafana again:

sudo apt install --only-upgrade -y grafana
sudo systemctl start grafana-server

In October we ran this exact sequence on the Ubuntu VM after first rolling it back one patch release. With the package already downloaded, the install took 23 seconds and Grafana answered its health check one second after it started, so the outage was about half a minute plus the tar. A patch release ran no migrations.

In November we repeated the sequence on Debian across a minor release, from the last build of the previous line to the current one, and the first start after it ran 9 new migrations in under 7 seconds. That VM’s disk was slow enough that the install alone took two minutes, so time the outage on your own hardware rather than trusting ours. Read the Grafana release notes before any minor or major jump, because that is where breaking changes are listed, then run sudo grafana server -v and open a dashboard to confirm the data still loads.

Upgrade gotcha: [marketplace] is now [plugins_marketplace]

The current Grafana line renames one grafana.ini section, and an upgrade does not carry your edit across. Grafana Labs flagged it as a breaking change in grafana/grafana#132172: [marketplace] becomes [plugins_marketplace], and the environment variable GF_MARKETPLACE_LICENSE_DIRECTORY becomes GF_PLUGINS_MARKETPLACE_LICENSE_DIRECTORY. A grafana.ini first written by the previous release line carries an uncommented [marketplace] header with its only key commented out; files created by earlier releases have no such section unless you added it yourself. The package only writes grafana.ini when it is missing, so the upgrade leaves the file alone and the old header survives. That is harmless until you have set license_directory under it. Then Grafana quietly ignores the value and falls back to its data path, and our journal had no line mentioning the section after the restart.

We reproduced it on Debian by setting license_directory on the previous release and upgrading with the steps above. Check your own file and what Grafana actually loaded; set GF_AUTH to the admin login you chose earlier:

GF_AUTH='admin:Use-A-Long-Passphrase-2026'
sudo grep -n -A3 "^\[marketplace\]" /etc/grafana/grafana.ini
curl -s -u "$GF_AUTH" localhost:3000/api/admin/settings | python3 -c 'import json,sys; d=json.load(sys.stdin); print("marketplace:", d.get("marketplace")); print("plugins_marketplace:", d.get("plugins_marketplace"))'

The two sections disagree. The old one still holds our directory, while the section Grafana now reads has fallen back to /var/lib/grafana:

marketplace: {'license_directory': '/etc/grafana/licenses'}
plugins_marketplace: {'license_directory': '/var/lib/grafana'}

Rename the header in place and restart. The sed only matches the exact header line, so the comments and your key under it stay as they are:

sudo sed -i 's/^\[marketplace\]$/[plugins_marketplace]/' /etc/grafana/grafana.ini
sudo systemctl restart grafana-server

Run the same curl again once Grafana answers. The old section is gone and the new one carries the directory, which is the whole fix:

Terminal on Debian 13 showing the old marketplace section ignored after the Grafana upgrade and plugins_marketplace picking up the licence directory after the rename

On a stock install with no license_directory set, nothing changes in behaviour, and the pull request notes that the features reading this section sit behind a pre-GA feature toggle. Rename it anyway: a leftover header is a setting that silently stops applying the day those features ship. Docker and Kubernetes deployments that pass the environment variable need the new name in their manifests too.

Roll back to the previous build

Rolling back is the same backup in reverse. List the releases the repository still carries to find the one you came from:

apt list -a grafana

The repository keeps older builds, with the installed one marked:

Listing...
grafana/stable,now 13.3.0 amd64 [installed]
grafana/stable 13.2.5 amd64
grafana/stable 13.2.4 amd64
grafana/stable 13.2.3 amd64

Stop Grafana and extract the tarball over /, which puts /var/lib/grafana and /etc/grafana back exactly as they were. Use the file name your backup step printed:

sudo systemctl stop grafana-server
sudo tar -xzf /var/backups/grafana/grafana-2026-11-05-2216.tar.gz -C /

In October we ran this restore on the Ubuntu VM: tar exited 0, Grafana started and both data sources were still listed through the API. On a fresh host, install the grafana package first and then extract the tarball, because the archive leaves out the package’s own files and plugins-bundled. To go back to the older build, install it next. Without --allow-downgrades, apt stops with Error: Packages were downgraded and -y was used without --allow-downgrades. and exit code 100, so pass the flag with the version you picked from the list:

sudo apt install -y --allow-downgrades grafana=13.2.5 #https://github.com/grafana/grafana/releases
sudo systemctl start grafana-server

Our October rollback went one patch release back. The November one crossed the minor release and its 9 new migrations: without the flag apt refused with the error above and Downgrading: 1 in its summary, and with the flag it installed the older build, started cleanly and reported that version from /api/health. It worked because the database came out of the backup taken before the upgrade. Across a minor or major version, restoring that backup before starting the older package is the part you cannot skip. The restore also brings back the old [marketplace] header, so redo the rename when you upgrade again.

Cut Grafana’s idle memory

On a Grafana that only talks to Prometheus, disabling the bundled data source plugins you don’t use halves Grafana’s resident memory, but the RAM the kernel actually gets back is closer to 70 to 100 MiB. The reason is how plugins run: every bundled data source with a backend (Loki, Elasticsearch, MySQL, PostgreSQL, Tempo and nine more on the current release) starts as its own Go process when Grafana boots, whether or not a data source uses it. On our November Debian install those were 14 gpx_ plugin children under the main process, each holding 25 to 40 MiB. CloudWatch is the newcomer; the October build started 13.

We measured idle memory about nine and a half minutes after a Grafana restart, with no browser open. Each figure is the summed RSS of every process in the service’s cgroup, the anonymous part of that RSS from /proc/<pid>/status, and MemTotal minus MemAvailable for the whole VM. Both lab VMs had 2 vCPU and 4 GB of RAM. These figures come from the October build with 13 plugin backends; we did not re-measure in November, so expect the default columns on the current release to carry one more plugin process of roughly 30 to 40 MiB RSS.

MeasurementDebian 13 defaultDebian 13 tunedUbuntu 26.04 defaultUbuntu 26.04 tuned
Grafana processes142142
Grafana RSS692 MiB382 MiB639 MiB341 MiB
Grafana anonymous RSS237 MiB170 MiB183 MiB131 MiB
Whole VM in use746 MiB645 MiB806 MiB738 MiB
Prometheus RSS97 MiB97 MiB90 MiB91 MiB
Node Exporter RSS26 MiB27 MiB24 MiB24 MiB
Nginx RSS (3 processes)28 MiB28 MiB26 MiB26 MiB

Read the RSS line with care. It drops by about 300 MiB on both distros, but most of that was the plugin binaries’ own code pages, which the kernel counts as page cache and can drop under pressure anyway. The anonymous memory and the whole-VM figure are the honest ones. The tuning freed 67 MiB of anonymous memory and 101 MiB of in-use RAM on Debian, and 52 MiB and 68 MiB on Ubuntu. Prometheus, Node Exporter and Nginx didn’t move, as expected, and together they come to about 150 MiB on a host that only scrapes itself.

If you run Grafana on a 1 or 2 GB VPS, the change is worth doing; on a 4 GB server you won’t notice it. The cost is that disabled plugins vanish from the Add data source list, so if you add Loki later you have to take loki back out of the list first. The setting lives in the [plugins] section of grafana.ini, and this sed fills in the commented-out line with every bundled backend except Prometheus.

sudo sed -i "s|^; disable_plugins =.*|disable_plugins = cloudwatch,elasticsearch,grafana-postgresql-datasource,grafana-pyroscope-datasource,influxdb,jaeger,loki,mssql,mysql,opentsdb,stackdriver,tempo,zipkin|" /etc/grafana/grafana.ini
sudo systemctl restart grafana-server

Confirm only the Prometheus backend is left running under the Grafana process.

ps -o rss=,comm= --ppid "$(systemctl show -p MainPID --value grafana-server)"

One child process remains, with its RSS in KiB.

32184 gpx_grafana-pro

If you set disable_plugins on an older release, add cloudwatch to your line after upgrading, or the new backend starts on every boot. In October, with the other twelve disabled, the Prometheus data source still passed its health check through the API on both VMs; in November we confirmed that the list above, CloudWatch included, leaves only the Prometheus backend running. We didn’t re-test the dashboards in the browser after the change. If you later find yourself sizing Prometheus rather than Grafana, cardinality is the number to watch, not the plugin count. A few hundred scraped nodes will outgrow this whole table long before Grafana does.

Keep reading

Upgrade Ubuntu 24.04 to Ubuntu 26.04 LTS (Step by Step) Ubuntu Upgrade Ubuntu 24.04 to Ubuntu 26.04 LTS (Step by Step) UFW Firewall Commands with Examples on Ubuntu 26.04 / 24.04 Security UFW Firewall Commands with Examples on Ubuntu 26.04 / 24.04 Configure Samba File Share on Debian 13 / 12 Debian Configure Samba File Share on Debian 13 / 12 Upgrade a Single Package on Ubuntu / Debian with apt Debian Upgrade a Single Package on Ubuntu / Debian with apt APT Cheat Sheet: apt and apt-get Commands for Ubuntu/Debian Cheat Sheets APT Cheat Sheet: apt and apt-get Commands for Ubuntu/Debian Installing Mono Microsoft’s .NET Framework on Debian 12 Debian Installing Mono Microsoft’s .NET Framework on Debian 12

Leave a Comment

Press ESC to close