Install Prometheus and Grafana on Kubernetes with Helm

On a kubeadm cluster, a default kube-prometheus-stack install comes up with four scrape targets down before you open Grafana once. The etcd, scheduler, controller manager and kube-proxy metrics ports listen on 127.0.0.1 only, so their dashboards stay empty and the bundled TargetDown alert fires after ten minutes. Installing Prometheus and Grafana on Kubernetes with the Helm chart is the easy part, and storage, those targets and HTTPS for Grafana are where your time goes.

Original content from computingforgeeks.com - post 93294

This guide installs the full stack into a monitoring namespace with persistent volumes, a Grafana password kept in a Secret, the kubeadm target fix, a ServiceMonitor and an alert rule for your own app, and an HTTPS Ingress. Every command below was run in October 2026 against kube-prometheus-stack 92.11.0 on a single-node kubeadm cluster running Kubernetes 1.37 on Ubuntu 26.04, so the output and the screenshots come from that box.

How the stack fits together

The chart deploys six workloads, and only one of them is meant to be reached by people. Prometheus pulls metrics from every target on a schedule, Grafana queries Prometheus with PromQL, and Alertmanager receives the alerts Prometheus fires. The Prometheus Operator sits beside them and turns Kubernetes objects (ServiceMonitor, PodMonitor, PrometheusRule) into scrape and rule configuration, which is why you never edit prometheus.yml on this stack. Grafana is the piece you expose, and in this guide it goes out through Traefik with a cert-manager certificate.

The diagram matches the lab: Prometheus on 9090 with a 20Gi volume and 15 days of retention, Grafana on 3000, Alertmanager on 9093, and the control plane ports the kubeadm fix opens further down. Everything else stays on cluster-internal Service addresses.

Prerequisites

You need a working cluster and a workstation that can reach it. Building the cluster is out of scope here. Ours was one control plane node built with our kubeadm guide for Ubuntu 26.04, with Calico as the CNI and the control plane taint removed so workloads could schedule on it. Before you start, check that you have:

• a Kubernetes cluster with kubectl access as cluster admin
• Helm 4 on the machine that runs kubectl (Helm 3 also installs this chart)
• a StorageClass, or the local-path provisioner from the next step
• about 1 GB of free RAM and 25 GB of disk on the node that runs Prometheus

Those numbers are a floor for following along, not a production recommendation. Prometheus memory and disk follow the number of active series multiplied by retention, and the series count grows with every node, pod and container you add. Our one node with 21 pods carried 80,163 active series, Prometheus held about 430 MB of working set and Grafana another 425 MB. On a real cluster, let it run for a day, read prometheus_tsdb_head_series and size RAM and the volume from that trend instead of from this lab. Readers on Amazon EKS can follow the EKS monitoring walkthrough, which swaps local-path for EBS.

Add a StorageClass on kubeadm

A fresh kubeadm cluster has no StorageClass at all, and the values file below asks for one by name. Skip this and Grafana sits in Pending while the operator refuses to create the Prometheus and Alertmanager StatefulSets (the exact error is in the troubleshooting section). Check first.

kubectl get storageclass

On kubeadm the answer is No resources found. Other platforms already ship a class, so what you do depends on where the cluster runs:

• k3s: use local-path, which k3s creates at install
• EKS, GKE or AKS: use the class marked (default) in that output
• kubeadm lab or single node: install local-path as shown next
• kubeadm in production: Longhorn, Rook-Ceph or your SAN’s CSI driver

Whatever you pick, the name goes into the values file in a later step, so note it now. For this lab, Rancher’s local-path provisioner is the quickest fix because it needs no extra disks or daemons. Detect its latest release and apply the manifest.

export LPP_VER=$(curl -sL https://api.github.com/repos/rancher/local-path-provisioner/releases/latest \
  | grep tag_name | head -1 | sed 's/.*"\(v[^"]*\)".*/\1/')
echo "${LPP_VER}"
kubectl apply -f "https://raw.githubusercontent.com/rancher/local-path-provisioner/${LPP_VER}/deploy/local-path-storage.yaml"
kubectl -n local-path-storage rollout status deploy/local-path-provisioner --timeout=180s

The class appears with WaitForFirstConsumer binding, so a volume is only created once a pod that uses it is scheduled. That is the right mode for node-local storage, because the directory has to exist on the node where the pod lands. It also means a PVC sitting in Pending before the pod starts is normal here, not an error.

NAME         PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  7s

Know what you are getting with it. Each volume is a plain directory under /opt/local-path-provisioner on the node, and the data is gone if that node is. The provisioner also ignores the requested size (its README says so), which means a 20Gi claim can grow until the root filesystem fills. That is why the values file sets a retentionSize as well as a time limit, and why you should not run Prometheus on local-path anywhere you care about the history.

Add the prometheus-community Helm repository

The chart is published both as an OCI artifact and in the classic Helm repository. The repository form keeps helm search and helm show short, so add it and refresh the index.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Check which chart version you are about to install. The APP VERSION column is the Prometheus Operator release, not Prometheus itself, which confuses most people the first time they read it.

helm search repo prometheus-community/kube-prometheus-stack --versions | head -4

The newest chart was at the top when we ran it. Pin that version with --version if you need repeatable installs across clusters:

NAME                                      	CHART VERSION	APP VERSION	DESCRIPTION                                       
prometheus-community/kube-prometheus-stack	92.11.0      	v0.94.1    	kube-prometheus-stack collects Kubernetes manif...
prometheus-community/kube-prometheus-stack	92.10.1      	v0.94.1    	kube-prometheus-stack collects Kubernetes manif...
prometheus-community/kube-prometheus-stack	92.10.0      	v0.94.1    	kube-prometheus-stack collects Kubernetes manif...

The chart metadata also declares the oldest Kubernetes release it accepts. Helm refuses to install on anything older, so read it before you plan an install on an old cluster.

helm show chart prometheus-community/kube-prometheus-stack | grep -E '^(version|appVersion|kubeVersion):'

Anything from 1.25 upwards passes this check, so every supported Kubernetes release is fine:

appVersion: v0.94.1
kubeVersion: '>=1.25.0-0'
version: 92.11.0

The version line is the chart release you just found with helm search, and the appVersion line becomes important again when you upgrade the CRDs later.

Create the Grafana admin Secret

The chart can generate a random Grafana password, but then it lives in a Secret named after the release and changes if the release is ever recreated. Creating your own Secret keeps the password out of the values file and out of Helm’s release history. Create the namespace, generate a password and store it.

kubectl create namespace monitoring
export GRAFANA_ADMIN_PASS=$(openssl rand -base64 18)
kubectl -n monitoring create secret generic grafana-admin \
  --from-literal=admin-user=admin \
  --from-literal=admin-password="${GRAFANA_ADMIN_PASS}"
echo "${GRAFANA_ADMIN_PASS}"

Copy the printed password into your password manager now. The export lasts only for this shell, and the later steps read the password back from the Secret if you lose it.

Write the Helm values file

The values file is where this install differs from helm install with defaults. Without it, Prometheus and Alertmanager write to emptyDir volumes and lose everything on the first pod restart, and Prometheus only picks up ServiceMonitors that carry the release label. Create the file.

vim kps-values.yaml

Paste this, which is the exact file the lab install used. Change local-path if your cluster uses another StorageClass, and size the Prometheus claim from your own series count rather than copying our 20Gi.

grafana:
  admin:
    existingSecret: grafana-admin
    userKey: admin-user
    passwordKey: admin-password
  persistence:
    enabled: true
    storageClassName: local-path
    size: 5Gi

prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 18GB
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 2Gi

Each block changes a default that bites later:

• grafana.admin.existingSecret: reads the login from the Secret you just created
• persistence and storageSpec: real volumes for Grafana, Prometheus and Alertmanager
• retention: 15d plus retentionSize: 18GB: whichever limit is hit first deletes the oldest blocks
• the three *SelectorNilUsesHelmValues: false keys: pick up ServiceMonitors, PodMonitors and rules from any namespace, with or without a release label

Keep retentionSize a few GB under the volume size, because the write-ahead log and compaction need headroom on top of the blocks. On local-path that headroom is the only limit you have, since the 20Gi request is not enforced.

Install kube-prometheus-stack

Install the chart into the namespace you created. The time prefix is only there so you can compare your install time with ours.

time helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values kps-values.yaml \
  --wait --timeout 15m

Our first successful install returned in under three minutes, with only part of the image set cached from an earlier failed attempt, and a reinstall with every image already pulled took 55 seconds. The NOTES text at the end tells you to read the password from a prometheus-grafana Secret, which does not exist when you use your own Secret, so ignore that line:

NAME: prometheus
LAST DEPLOYED: Sat Oct 24 06:12:36 2026
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus"

Get Grafana 'admin' user password by running:

  kubectl --namespace monitoring get secrets prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

Access Grafana local instance:

  export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=prometheus" -oname)
  kubectl --namespace monitoring port-forward $POD_NAME 3000

Get your grafana admin user password by running:

  kubectl get secret --namespace monitoring -l app.kubernetes.io/component=admin-secret -o jsonpath="{.items[0].data.admin-password}" | base64 --decode ; echo


Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

real	2m50.818s
user	0m4.833s
sys	0m0.795s

Don’t trust --wait as proof that Prometheus is up. Helm waits for the objects it created, and the Prometheus and Alertmanager StatefulSets are created afterwards by the operator, so on our first install helm install returned while the prometheus-...-prometheus-0 pod was still in PodInitializing. Wait on the pods themselves.

kubectl -n monitoring wait --for=condition=Ready pods --all --timeout=300s
kubectl get pods -n monitoring

Six pods, all Ready. Grafana shows 3/3 because it runs two sidecars that load dashboards and datasources from ConfigMaps, and node-exporter runs as a DaemonSet, so a bigger cluster shows one of those per node:

NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          32s
prometheus-grafana-67f8b59bf9-vnxrf                      3/3     Running   0          34s
prometheus-kube-prometheus-operator-59b4bf45f6-mcfh4     1/1     Running   0          34s
prometheus-kube-state-metrics-7b49cdf8bb-w8z6f           1/1     Running   0          34s
prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   0          32s
prometheus-prometheus-node-exporter-mcc5k                1/1     Running   0          34s

Confirm the three volumes bound to the sizes you asked for. The default kubectl get pvc output prints claim names over 100 characters long, so these custom columns keep it readable.

kubectl get pvc -n monitoring \
  -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,SIZE:.spec.resources.requests.storage,CLASS:.spec.storageClassName

A PVC stuck in Pending here means the class name in the values file does not exist on your cluster:

NAME                                                                                                             STATUS   SIZE   CLASS
alertmanager-prometheus-kube-prometheus-alertmanager-db-alertmanager-prometheus-kube-prometheus-alertmanager-0   Bound    2Gi    local-path
prometheus-grafana                                                                                               Bound    5Gi    local-path
prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0           Bound    20Gi   local-path

The chart also installed ten CustomResourceDefinitions. These are the object types you will create for your own apps, and they matter again at upgrade and uninstall time because Helm leaves them alone.

kubectl get crd | grep monitoring.coreos.com

ServiceMonitor, PodMonitor, PrometheusRule and AlertmanagerConfig are the four most teams use day to day:

alertmanagerconfigs.monitoring.coreos.com               Namespaced   v1alpha1(storage)   2026-10-24T07:12:07Z
alertmanagers.monitoring.coreos.com                     Namespaced   v1(storage)         2026-10-24T07:12:07Z
podmonitors.monitoring.coreos.com                       Namespaced   v1(storage)         2026-10-24T07:12:08Z
probes.monitoring.coreos.com                            Namespaced   v1(storage)         2026-10-24T07:12:08Z
prometheusagents.monitoring.coreos.com                  Namespaced   v1alpha1(storage)   2026-10-24T07:12:08Z
prometheuses.monitoring.coreos.com                      Namespaced   v1(storage)         2026-10-24T07:12:09Z
prometheusrules.monitoring.coreos.com                   Namespaced   v1(storage)         2026-10-24T07:12:10Z
scrapeconfigs.monitoring.coreos.com                     Namespaced   v1alpha1(storage)   2026-10-24T07:12:10Z
servicemonitors.monitoring.coreos.com                   Namespaced   v1(storage)         2026-10-24T07:12:11Z
thanosrulers.monitoring.coreos.com                      Namespaced   v1(storage)         2026-10-24T07:12:11Z

The creation times show all ten arrived in the same few seconds as the chart, which is how you can tell later whether a CRD came from this install or from an older one. The kubectl cheat sheet has more of the get and jsonpath forms used through the rest of this guide.

Fix the kubeadm control plane targets

Right after the install, Prometheus could reach the API server and the kubelet but not etcd, the scheduler, the controller manager or kube-proxy. The reason is that kubeadm binds their metrics endpoints to 127.0.0.1, and Prometheus scrapes them on the node IP, so every scrape ends in connection refused. You lose the etcd, scheduler and controller manager dashboards, and the bundled TargetDown rule fires once a job has had targets down for 10 minutes. This is the Target health page, filtered to targets that are down.

This section only applies to kubeadm. On k3s the control plane runs inside one process, and on EKS, GKE and AKS the control plane is not yours to scrape, so those clusters need the matching components disabled in the values file instead. Confirm the listeners on the control plane node.

sudo ss -tlnp | grep -E ':(2381|10249|10257|10259) '

All four listen on loopback only, which is why the scrapes from the node IP were refused:

LISTEN 0      4096       127.0.0.1:2381       0.0.0.0:*    users:(("etcd",pid=97206,fd=15))
LISTEN 0      4096       127.0.0.1:10249      0.0.0.0:*    users:(("kube-proxy",pid=98460,fd=24))
LISTEN 0      4096       127.0.0.1:10257      0.0.0.0:*    users:(("kube-controller",pid=97199,fd=4))
LISTEN 0      4096       127.0.0.1:10259      0.0.0.0:*    users:(("kube-scheduler",pid=97294,fd=4))

Three of them are static pods, so editing their manifests is enough and the kubelet restarts each one. Editing etcd.yaml restarts etcd, and the API server drops with it for about a minute, so expect TLS handshake timeout errors from kubectl until it is back. Rebind the controller manager, the scheduler and the etcd metrics listener.

sudo sed -i 's/--bind-address=127.0.0.1/--bind-address=0.0.0.0/' \
  /etc/kubernetes/manifests/kube-controller-manager.yaml \
  /etc/kubernetes/manifests/kube-scheduler.yaml
sudo sed -i 's#--listen-metrics-urls=http://127.0.0.1:2381#--listen-metrics-urls=http://0.0.0.0:2381#' \
  /etc/kubernetes/manifests/etcd.yaml

Wait for the API server to answer again before the next step, otherwise the kube-proxy change fails half applied. On our node the gap lasted about a minute. This loop simply polls the readiness endpoint until it answers.

until kubectl get --raw=/readyz >/dev/null 2>&1; do sleep 5; done; echo "API server is back"

kube-proxy reads its settings from a ConfigMap, where metricsBindAddress is empty and therefore defaults to loopback. Set it, then restart the DaemonSet so the pods pick it up.

kubectl -n kube-system get configmap kube-proxy -o yaml \
  | sed 's/metricsBindAddress: ""/metricsBindAddress: "0.0.0.0:10249"/' \
  | kubectl replace -f -
kubectl -n kube-system rollout restart daemonset kube-proxy
kubectl -n kube-system rollout status daemonset kube-proxy --timeout=120s

Run the same ss check again. All four ports should now listen on every address:

LISTEN 0      4096               *:10249            *:*    users:(("kube-proxy",pid=104479,fd=20))
LISTEN 0      4096               *:10259            *:*    users:(("kube-scheduler",pid=103414,fd=4))
LISTEN 0      4096               *:10257            *:*    users:(("kube-controller",pid=103670,fd=4))
LISTEN 0      4096               *:2381             *:*    users:(("etcd",pid=103613,fd=14))

Opening these ports has a cost, so restrict them. The controller manager and scheduler ports need a token. The etcd port 2381 and kube-proxy’s 10249 serve plain HTTP metrics to anyone who can reach the node, so your firewall should allow them only from the pod network and your monitoring hosts. Also expect kubeadm upgrade to undo the three manifest edits: the kubeadm reconfiguration docs warn that an upgrade may overwrite changes made after the cluster was created, so re-run the two sed lines after every upgrade and check the targets again.

On a cluster with several control plane nodes, run the two sed commands on every one of them, because each node has its own static pod manifests. The kube-proxy change is cluster-wide and runs once. For a fix that survives upgrades, put the same flags in the cluster’s kubeadm configuration. In the kubeadm.k8s.io/v1beta4 format, extraArgs is a list of name and value pairs: an entry with name: bind-address and value: 0.0.0.0 for controllerManager and scheduler, plus name: listen-metrics-urls with value: http://0.0.0.0:2381 for etcd.local. Kubeadm patch files passed to kubeadm upgrade --patches do the same job, as described in the kubeadm control plane flags docs. We used the sed route on our single node and did not test either persistent form.

Open Grafana with a port-forward

The quickest way in needs no Ingress, no DNS and no certificate. Forward the Grafana Service to your workstation, which is all you need for a lab or a first look.

kubectl -n monitoring port-forward svc/prometheus-grafana 3000:80

If kubectl runs on the cluster node and not on your laptop, add an SSH tunnel from the laptop to that node (10.0.1.50 in our lab, with your own login in place of user) and keep both running.

ssh -N -L 3000:127.0.0.1:3000 [email protected]

Browse to http://localhost:3000 and sign in as admin with the password you generated. If you lost it, read it back from the Secret:

kubectl -n monitoring get secret grafana-admin -o jsonpath='{.data.admin-password}' | base64 -d; echo

The login page shows the Grafana version in its footer, which is a handy way to confirm which Grafana the chart actually deployed. The chart bumps Grafana on its own schedule, so this version changes with chart upgrades even when you never touch Grafana settings.

A port-forward dies with your terminal and drops whenever the Grafana pod restarts, which happened to ours every time a Helm upgrade changed its config. Use it for setup and debugging, and use the Ingress section further down for anything people rely on.

Explore the bundled Grafana dashboards

The chart provisioned close to 30 dashboards into Grafana, and the Prometheus datasource is already wired in, so there is nothing to import. Open Dashboards from the left menu to see them.

Start with Kubernetes / Compute Resources / Cluster. It breaks CPU and memory down by namespace and compares usage with requests and limits. On our node, about 45 minutes after install, the monitoring namespace used 0.04 cores and 872 MiB, and kube-system used more CPU than everything else combined because etcd and the API server live there.

The CPU Limits tile reads No data because none of our pods set a CPU limit, and that is the kind of gap this dashboard is good at exposing. For host-level numbers, open Node Exporter / Nodes, which shows CPU, load, memory, disk and network for one node at a time. Three dashboards cover most first questions about a cluster: Compute Resources / Cluster for “who is using it”, Compute Resources / Namespace (Pods) for “which pod”, and Node Exporter / Nodes for “is the host itself short of something”.

Check Prometheus targets and Alertmanager

Grafana only shows what Prometheus collected, so when a panel is empty, check Prometheus before you blame the dashboard. Forward the Prometheus and Alertmanager Services the same way as Grafana.

kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 &
kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-alertmanager 9093:9093 &

Open http://localhost:9090 and run sum by (job) (up) in the Query tab. Each row is a scrape job and the value is how many of its targets are up, so a healthy kubeadm node shows every job at 1 or more, including the four control plane jobs fixed above:

Alertmanager on http://localhost:9093 should already show one alert, Watchdog. It fires on purpose all the time, so that an outside dead man’s switch can tell you when the whole alerting pipeline stops. In our screenshot it sits next to the custom alert from the PrometheusRule section, and both are routed to a receiver called null:

That null receiver is the default configuration the chart writes, and it means no alert leaves the cluster until you add a real receiver. We dumped it from the generated Secret to confirm:

kubectl -n monitoring get secret alertmanager-prometheus-kube-prometheus-alertmanager \
  -o jsonpath='{.data.alertmanager\.yaml}' | base64 -d | grep -A4 '^receivers'

One receiver, and it sends nowhere. Every alert, including the custom one later in this guide, ends up there:

receivers:
- name: "null"
route:
  group_by:
  - namespace

Receivers for email, Slack or a webhook go under alertmanager.config in the values file. The routing and receiver syntax is the same as on a standalone server, which our Alertmanager setup guide covers. We did not configure a receiver in this lab.

Monitor your own app with a ServiceMonitor

Your applications get scraped the same way the stack scrapes itself: the app exposes /metrics, a Service selects its pods, and a ServiceMonitor tells the operator which Service port to scrape. We used podinfo, a small Go web app that serves Prometheus metrics on its main port 9898. Create the manifest.

vim podinfo.yaml

It holds a demo namespace, a two-replica Deployment and a Service. The port name http matters, because the ServiceMonitor refers to the port by name, not by number.

apiVersion: v1
kind: Namespace
metadata:
  name: demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  namespace: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      labels:
        app: podinfo
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.15.0
          ports:
            - name: http
              containerPort: 9898
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              memory: 64Mi
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo
  namespace: demo
  labels:
    app: podinfo
spec:
  selector:
    app: podinfo
  ports:
    - name: http
      port: 9898
      targetPort: http

Apply it and wait for both replicas. Two replicas matter here, because the alert rule further down counts healthy podinfo targets and needs a number to fall from:

kubectl apply -f podinfo.yaml
kubectl -n demo rollout status deploy/podinfo --timeout=180s

Now create the ServiceMonitor. It lives in the app’s namespace and selects the Service by its app: podinfo label:

vim podinfo-servicemonitor.yaml

No release label is needed, because the values file told Prometheus to accept ServiceMonitors from any namespace. With the chart defaults you would need release: prometheus on it, and a ServiceMonitor without that label is ignored with no error anywhere. That silent miss is the most common reason a new app never shows up in Prometheus.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: podinfo
  namespace: demo
spec:
  selector:
    matchLabels:
      app: podinfo
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

Apply it. The operator watches ServiceMonitors in every namespace, so there is nothing to restart and no Helm upgrade to run, and it rewrites the Prometheus scrape config on its own within a few seconds:

kubectl apply -f podinfo-servicemonitor.yaml

Within one scrape interval, a new pool named serviceMonitor/demo/podinfo/0 appears on the Target health page with one endpoint per pod. The pool is named after the namespace, the ServiceMonitor and the endpoint index, so you can tell at a glance which object produced which targets.

The same check works from the command line against the Prometheus API, through the port-forward from the previous section. A value of 1 per pod means the scrape succeeded.

curl -sG http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=up{job="podinfo"}'; echo

Both podinfo pods report "1", and the labels show which namespace, pod and Service each sample came from:

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","container":"podinfo","endpoint":"http","instance":"10.244.73.70:9898","job":"podinfo","namespace":"demo","pod":"podinfo-7cdff85d5b-fmdsg","service":"podinfo"},"value":[1792826164.127,"1"]},{"metric":{"__name__":"up","container":"podinfo","endpoint":"http","instance":"10.244.73.69:9898","job":"podinfo","namespace":"demo","pod":"podinfo-7cdff85d5b-rqn26","service":"podinfo"},"value":[1792826164.127,"1"]}]}}

If the pool never appears, the usual cause is a ServiceMonitor that selects the wrong labels or names a port the Service does not have. Check kubectl get endpoints -n demo podinfo first, then the operator log.

Add a PrometheusRule alert

Alert rules are PrometheusRule objects, and the operator loads them into Prometheus without a restart. This rule fires when fewer than two podinfo targets are up for two minutes, which is easy to trigger on purpose by scaling the Deployment down.

vim podinfo-rules.yaml

The for: 2m line is what moves an alert from pending to firing, and it is the main defence against paging someone over a single failed scrape.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: podinfo-rules
  namespace: demo
spec:
  groups:
    - name: podinfo.rules
      rules:
        - alert: PodinfoReplicasLow
          expr: sum(up{job="podinfo", namespace="demo"}) < 2
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "podinfo has fewer than 2 healthy replicas"
            description: "Only {{ $value }} podinfo target(s) are up in namespace demo."

Apply the rule, then scale podinfo to one replica to break it. Scaling down is a safe way to test, because nothing else depends on the demo app and the scrape result drops on the next interval.

kubectl apply -f podinfo-rules.yaml
kubectl -n demo scale deployment podinfo --replicas=1

Here is how it played out on our cluster. We scaled down at 06:26:48 UTC, the alert went pending at about 06:28 once a scrape and a rule evaluation had both seen one target, and it fired at 06:30:11 after the two minutes ran out. Alertmanager stamped it with that firing time. In the Prometheus Alerts tab the rule shows its expression, labels and how long it has been active:

One gap in this demo rule: sum() over no series returns nothing, so with zero replicas the expression is empty and the alert never fires. A production rule should cover total loss too, for example by adding or vector(0) to the sum or a second rule on absent(up{job="podinfo"}). We only tested the rule as shown.

Scale back to two replicas when you have seen it, and the alert resolves on the next evaluation. The rule drops back to inactive in the Alerts tab as soon as the expression stops matching, so you can watch the whole cycle in a few minutes.

kubectl -n demo scale deployment podinfo --replicas=2

Writing rules that page at the right moment is mostly PromQL practice: picking the right rate windows, aggregating away noisy labels and choosing for durations. If you want a structured way to learn that, our roundup of Grafana and Prometheus monitoring books lists the ones worth reading. To alert on endpoints from the outside, pair this stack with Blackbox exporter uptime checks.

Expose Grafana over HTTPS with Traefik and cert-manager

This section is optional, and everything in it is extra to the core install:

• an ingress controller (Traefik here)
• cert-manager to issue and renew the certificate
• a public DNS A record for the Grafana name
• ports 80 and 443 forwarded from the internet to Traefik’s node ports

The default path is a Let’s Encrypt certificate through an HTTP-01 challenge, which needs the last two items. Our lab had neither a public name nor port 80, so we never requested a Let’s Encrypt certificate: the issuer below passed the API server’s validation with --dry-run=server, and the certificate in our checks came from a private CA issuer called lab-ca, which is the option further down for clusters with no public port 80. Set the three values this section uses, with your own name and contact address.

export GRAFANA_DOMAIN="grafana.example.com"
export ADMIN_EMAIL="[email protected]"
export CLUSTER_ISSUER="letsencrypt-prod"

Install cert-manager from its OCI chart with its CRDs, using the latest release tag. The OCI chart is the one cert-manager’s docs now lead with, and crds.enabled=true lets the chart install its own CRDs so you don’t manage them separately.

export CM_VER=$(curl -sL https://api.github.com/repos/cert-manager/cert-manager/releases/latest \
  | grep tag_name | head -1 | sed 's/.*"\(v[^"]*\)".*/\1/')
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version "${CM_VER}" \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true --wait --timeout 5m

Next comes the ingress controller. The community ingress-nginx project stopped shipping releases and security fixes after March 2026, per the Kubernetes retirement notice, so we used Traefik. Without a cloud load balancer, its Service has to be a NodePort, and the current chart takes the type under service.spec.type. We first passed service.type, which the chart ignored without a warning and left a LoadBalancer stuck at <pending>, so copy the flags exactly.

helm repo add traefik https://traefik.github.io/charts
helm repo update traefik
helm upgrade --install traefik traefik/traefik \
  --namespace traefik --create-namespace \
  --set service.spec.type=NodePort \
  --set ports.web.nodePort=30080 \
  --set ports.websecure.nodePort=30443 \
  --wait --timeout 5m
kubectl get svc -n traefik
kubectl get ingressclass

Traefik now answers on 30080 and 30443 on every node, and the chart made it the default IngressClass. If your cluster already has a default controller, add --set ingressClass.isDefaultClass=false to the command above, otherwise the cluster ends up with two defaults:

NAME      TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
traefik   NodePort   10.101.94.57   <none>        80:30080/TCP,443:30443/TCP   14s
NAME                CONTROLLER                      PARAMETERS   AGE
traefik (default)   traefik.io/ingress-controller   <none>       14s

No public port 80 on your cluster? Skip ahead to the private-cluster subsection below. Otherwise, point DNS and the ports at Traefik before you create the issuer, not after. cert-manager runs its own self-check against http://${GRAFANA_DOMAIN}/.well-known/acme-challenge/... before it asks Let’s Encrypt to validate, and it waits there until that URL answers. On your DNS provider, router or cloud load balancer, set up:

• an A record for the Grafana name pointing at your public address
• port 80 forwarded to node port 30080 (HTTP-01 challenge and redirects)
• port 443 forwarded to node port 30443 (Grafana over HTTPS)

Forwarding only 443 is the common mistake here. Nothing on the node listens on port 80 itself, so a forward of 80 to 80 fails the challenge even though the firewall looks open. From a machine outside your network, http:// plus the Grafana name should now return a 404 from Traefik, which proves DNS and port 80 both reach it.

Create the Let’s Encrypt issuer. It uses Traefik to answer the HTTP-01 challenge.

vim letsencrypt-issuer.yaml

Keep the ADMIN_EMAIL_HERE placeholder as it is; the next command fills it from your variable:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ADMIN_EMAIL_HERE
    privateKeySecretRef:
      name: letsencrypt-prod-account
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik

Fill in the email and apply it. The issuer registers an ACME account under that address, so use a mailbox your team owns rather than a personal one:

sed -i "s/ADMIN_EMAIL_HERE/${ADMIN_EMAIL}/" letsencrypt-issuer.yaml
kubectl apply -f letsencrypt-issuer.yaml

The issuer turns Ready once it has registered the ACME account, and it requests nothing until an Ingress asks for a certificate. If your cluster can take the HTTP-01 challenge, skip the next subsection and go straight to the Ingress.

Private cluster with no public port 80: use your own CA

A lab or internal cluster that cannot receive the HTTP-01 challenge still gets HTTPS from a private CA in cert-manager, which is what we ran. Browsers will not trust it until you import the root, so it suits internal tools, not public ones. Public names behind NAT can use a DNS-01 solver instead, which the cert-manager DNS-01 docs cover per DNS provider; we did not test that path. Create the CA issuer file:

vim lab-ca-issuer.yaml

It bootstraps a self-signed issuer, uses it to sign a root certificate stored in the lab-root-ca Secret, and exposes that root as the lab-ca ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-bootstrap
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: lab-root-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: lab-root-ca
  secretName: lab-root-ca
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-bootstrap
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: lab-ca
spec:
  ca:
    secretName: lab-root-ca

Apply it, wait for the root certificate, and switch the issuer variable to lab-ca so the next file picks it up:

kubectl apply -f lab-ca-issuer.yaml
kubectl -n cert-manager wait --for=condition=Ready certificate/lab-root-ca --timeout=60s
kubectl get clusterissuer
export CLUSTER_ISSUER="lab-ca"

Both issuers report Ready almost immediately, because neither talks to anything outside the cluster:

NAME                   READY   AGE
lab-ca                 True    3s
selfsigned-bootstrap   True    4s

To trust Grafana’s certificate in your browser, export the root from the lab-root-ca Secret and import it on the machines that need it:

kubectl -n cert-manager get secret lab-root-ca -o jsonpath='{.data.ca\.crt}' | base64 -d > lab-root-ca.crt

The Grafana name also has to resolve inside your network, through an internal DNS record or a hosts file entry pointing at the node. Without a 443 forward, the browser URL carries the node port, as in https://grafana.example.com:30443.

Create the Grafana Ingress

The Grafana Ingress goes into a second values file, so the core kps-values.yaml stays the same with or without it. Keeping it separate also makes it obvious at upgrade time which file carries the Ingress, because you must pass both files to every later helm upgrade.

vim grafana-ingress-values.yaml

Paste it with both placeholders left in. root_url tells Grafana its public address, so its redirects and links use HTTPS:

grafana:
  ingress:
    enabled: true
    ingressClassName: traefik
    annotations:
      cert-manager.io/cluster-issuer: CLUSTER_ISSUER_HERE
    hosts:
      - GRAFANA_DOMAIN_HERE
    tls:
      - secretName: grafana-tls
        hosts:
          - GRAFANA_DOMAIN_HERE
  grafana.ini:
    server:
      root_url: https://GRAFANA_DOMAIN_HERE

Fill in the name and the issuer from your variables, then upgrade the release with both values files. The order matters only when two files set the same key, and these two do not overlap.

sed -i -e "s/GRAFANA_DOMAIN_HERE/${GRAFANA_DOMAIN}/g" \
  -e "s/CLUSTER_ISSUER_HERE/${CLUSTER_ISSUER}/" grafana-ingress-values.yaml
helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values kps-values.yaml \
  --values grafana-ingress-values.yaml \
  --wait --timeout 10m
kubectl -n monitoring get ingress,certificate

With our lab-ca issuer, cert-manager issued grafana-tls within seconds and Traefik picked it up. With Let’s Encrypt, the certificate stays False until the HTTP-01 challenge passes, which depends on the DNS and port forwards above:

NAME                                           CLASS     HOSTS                 ADDRESS   PORTS     AGE
ingress.networking.k8s.io/prometheus-grafana   traefik   grafana.example.com             80, 443   28s

NAME                                      READY   SECRET        AGE
certificate.cert-manager.io/grafana-tls   True    grafana-tls   28s

If a Let’s Encrypt certificate is still not Ready after a few minutes, look at the ACME objects cert-manager created for it. The describe output names the step that is stuck, and the most common one is in the troubleshooting section below. We could not run these against Let’s Encrypt in our lab:

kubectl -n monitoring get challenges,orders
kubectl -n monitoring describe challenges

Test the certificate Traefik actually serves for that name. With Let’s Encrypt the issuer line names a Let’s Encrypt intermediate; ours names the lab root.

echo | openssl s_client -connect 127.0.0.1:30443 -servername "${GRAFANA_DOMAIN}" 2>/dev/null \
  | openssl x509 -noout -issuer -ext subjectAltName -dates

The SAN matches the Grafana name. Our lab certificate runs for 90 days, cert-manager’s default duration, and cert-manager renews it on its own before it expires:

issuer=CN=lab-root-ca
X509v3 Subject Alternative Name: critical
    DNS:grafana.example.com
notBefore=Oct 24 07:21:37 2026 GMT
notAfter=Jan 22 07:21:37 2027 GMT

Once Grafana is reachable this way, stop using the port-forward for it. Prometheus and Alertmanager have no login of their own, so keep them on port-forwards rather than publishing them through the same Ingress.

Measure what the stack costs at idle

The stack is lighter on CPU than most people expect and heavier on memory, almost all of it in two pods. We read these numbers from Prometheus itself about 20 minutes after install, with the podinfo demo running.

curl -sG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=sort_desc(sum by (pod) (container_memory_working_set_bytes{namespace="monitoring",container!=""}))'; echo

The table puts that output next to the other figures from the same run:

Measurement	Value on k8s01
Pods in the monitoring namespace	6
Prometheus working set	430 MB
Grafana working set	425 MB
Alertmanager, operator, kube-state-metrics, node-exporter	86 MB combined
Whole namespace, memory	941 MB
Whole namespace, CPU (rate over the last few minutes)	0.07 cores
Active series (prometheus_tsdb_head_series)	80,163
Scrape targets	20 across 14 jobs
Prometheus volume used after 22 minutes	15 MB
First successful helm install, mostly cold image cache	2 min 51 s

Grafana’s share surprised us. It briefly reached 685 MB right after a restart while it loaded the provisioned dashboards, then settled at 425 MB, so give its pod a memory limit with headroom rather than one cut to the idle number. Prometheus is the one that grows: its memory tracks active series, so read the series count after a day on your own cluster and size from there.

Upgrade the chart and its CRDs

A chart upgrade has two halves, and Helm only does one of them. helm upgrade rolls the operator, Prometheus, Grafana and the exporters forward, but the ten CRDs stay at whatever version first created them. The chart README says so for Helm 3, and we checked that it still holds with Helm 4. We installed an older 90.x release, upgraded it to the latest chart with the same values file, and the operator moved to the new release while every CRD kept the old one. Work through an upgrade in this order:

1. Read the chart’s UPGRADE.md for every major version you cross
2. Update the CRDs to the operator release of the target chart
3. Run helm upgrade with the same values files as the install
4. Check the pods, then the Target health page

Refresh the repository index first, then check which operator version the CRDs on your cluster were built for. The annotation is the quickest way to see it:

helm repo update prometheus-community
kubectl get crd prometheuses.monitoring.coreos.com \
  -o jsonpath='{.metadata.annotations.operator\.prometheus\.io/version}'; echo

On our upgraded release it still printed the old operator version, even though the running operator image was a release newer:

0.93.1

The chart’s appVersion is the operator release whose CRDs it expects, so read it from the chart and apply that release’s CRD files. The loop below reads it from the newest chart. If you upgrade to a pinned chart instead, add the same --version to the helm show chart command, or you will apply CRDs for the wrong operator release. The command in UPGRADE.md uses kubectl apply --server-side on its own, and under Helm 4 that failed on every file. Helm 4 installs with server-side apply by default, so the field manager helm owns the CRD fields and kubectl refuses to take them over:

error: Apply failed with 3 conflicts: conflicts with "helm":
- .metadata.annotations.controller-gen.kubebuilder.io/version
- .metadata.annotations.operator.prometheus.io/version
- .spec.versions

Our release was installed by Helm 4 from the start. If yours came from Helm 3 through a Helm 3 to Helm 4 migration, the field owners can differ, so run the plain command first and read the error. Adding --force-conflicts transfers those fields to kubectl, which is what you want here, because the CRDs are exactly what Helm will not update. This loop is the one that worked on our cluster:

export OP_VER=$(helm show chart prometheus-community/kube-prometheus-stack | grep '^appVersion:' | awk '{print $2}')
echo "${OP_VER}"
for crd in alertmanagerconfigs alertmanagers podmonitors probes prometheusagents prometheuses prometheusrules scrapeconfigs servicemonitors thanosrulers; do
  kubectl apply --server-side --force-conflicts \
    -f "https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/${OP_VER}/example/prometheus-operator-crd/monitoring.coreos.com_${crd}.yaml"
done

Each file reports serverside-applied, and the annotation check above then printed the new operator version for all ten CRDs. Your existing ServiceMonitors and rules are untouched, because the CRD update only changes the schema they are validated against:

customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/prometheusagents.monitoring.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/scrapeconfigs.monitoring.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com serverside-applied

Now upgrade the release itself. Pass every values file you installed with, or Helm resets the missing settings to chart defaults and your volumes or Ingress go with them. Add --values grafana-ingress-values.yaml to this command if you set up the HTTPS section:

helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values kps-values.yaml \
  --wait --timeout 15m
helm list -n monitoring

Our 90.x release moved to the latest chart in one revision, and every pod restarted onto the new images within half a minute:

NAME      	NAMESPACE 	REVISION	UPDATED                                	STATUS  	CHART                        	APP VERSION
prometheus	monitoring	2       	2026-10-24 07:06:18.583701862 +0000 UTC	deployed	kube-prometheus-stack-92.11.0	v0.94.1    

Our test ran the CRD update after helm upgrade, and the operator logged no errors in between, but upstream asks for the CRDs first, so follow that order. The chart also has a crds.upgradeJob.enabled value that runs a Job to apply the CRDs during the upgrade. We did not test it, so the manual loop above is the path we can vouch for.

Uninstall the stack

helm uninstall removes the workloads and most of what the chart created, but it leaves enough behind that a later reinstall picks up old data. Remove the release, then look at what is still in the namespace:

helm uninstall prometheus -n monitoring
kubectl get pvc,secret -n monitoring

On our cluster the Prometheus and Alertmanager volumes survived, along with three Secrets:

NAME                                                                                                                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/alertmanager-prometheus-kube-prometheus-alertmanager-db-alertmanager-prometheus-kube-prometheus-alertmanager-0   Bound    pvc-0c62c246-a607-469f-b0e8-2d9ff34c1bea   2Gi        RWO            local-path     <unset>                 50m
persistentvolumeclaim/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0           Bound    pvc-59902deb-fdbf-4d4c-84d0-0c268b7b497f   20Gi       RWO            local-path     <unset>                 50m

NAME                                          TYPE                DATA   AGE
secret/grafana-admin                          Opaque              2      53m
secret/grafana-tls                            kubernetes.io/tls   3      36m
secret/prometheus-kube-prometheus-admission   Opaque              3      53m

Grafana’s volume went with the release because Helm created it directly. The other two came from the StatefulSets’ volume claim templates, which the operator created, so Helm never owned them. Everything left after the uninstall:

• the Prometheus and Alertmanager PVCs with their data
• grafana-admin, the Secret you created
• grafana-tls, written by cert-manager
• the admission webhook Secret from the chart’s install hook
• all ten monitoring.coreos.com CRDs
• your own ServiceMonitors and PrometheusRules, such as the podinfo ones in demo

Keep the PVCs if you plan to reinstall and want the metric history back. To remove everything, delete the claims and then the CRDs. Volumes on local-path use the Delete reclaim policy, so deleting a claim deletes its directory on the node, and deleting a CRD deletes every object of that kind in every namespace, so check that nothing else on the cluster uses them first:

kubectl delete pvc -n monitoring --all
for crd in alertmanagerconfigs alertmanagers podmonitors probes prometheusagents prometheuses prometheusrules scrapeconfigs servicemonitors thanosrulers; do
  kubectl delete crd ${crd}.monitoring.coreos.com
done

The operator also manages a kubelet Service in kube-system for scraping, and it may outlive the release. We did not check for it after our uninstall, so look for it before a reinstall under a different release name, where a leftover could produce duplicate kubelet targets:

kubectl -n kube-system get svc | grep kubelet

Delete it if it is there and no other Prometheus uses it. Delete the monitoring namespace last to take the leftover Secrets with it. If you only want to move the stack to a new chart major, skip all of this and use the upgrade path above.

Troubleshoot errors from the lab run

Every error below happened on the lab cluster while writing this guide, and each fix is the one that cleared it. The kubeadm target errors are covered in their own section above, so they only get a pointer here.

Error: storage class “local-path” does not exist

Our first install ran before the cluster had a StorageClass. Grafana’s PVC stayed Pending with ProvisioningFailed, but the more confusing part was that the Prometheus and Alertmanager pods never appeared at all. The operator refuses to create a StatefulSet whose volume claim names a missing class, and the reason is only in its log:

kubectl -n monitoring logs deploy/prometheus-kube-prometheus-operator | grep 'does not exist'

Every reconcile loop repeated the same line, every 10 to 20 seconds in our log:

ts=2026-10-24T06:10:32.779455498Z level=error caller=/workspace/pkg/operator/resource_reconciler.go:699 msg="Unhandled Error" logger=UnhandledError err="sync \"monitoring/prometheus-kube-prometheus-prometheus\" failed: storage class \"local-path\" does not exist"

We fixed it by uninstalling, creating the class and installing again, and the next install came up clean. The operator retries the sync on its own (the line repeats in the log), so creating the class or correcting the name with helm upgrade should also recover a running release, but we did not test that path.

Error: dial tcp 10.0.1.50:10257: connect: connection refused

This is the kubeadm control plane target problem, and the same message appears for ports 10259, 2381 and 10249. The fix is the rebind in the control plane section. If it comes back after a cluster upgrade, kubeadm upgrade has most likely regenerated the static pod manifests, so run the sed lines again.

Error: net/http: TLS handshake timeout right after editing etcd.yaml

Saving /etc/kubernetes/manifests/etcd.yaml restarts etcd, and the API server restarts behind it. Our kube-proxy ConfigMap command ran during that gap and failed with this message plus the server was unable to return a response in the time allotted. Nothing was broken. Wait for /readyz to answer as shown above and run the command again. Any kubectl port-forward sessions die during the same window and have to be restarted.

Challenge stuck at “Waiting for http-01 challenge propagation: failed to perform self check GET request”

We did not hit this one, because our lab never requested a Let’s Encrypt certificate. It is the failure the cert-manager ACME troubleshooting docs describe for HTTP-01, and the setup in this guide is exposed to it. cert-manager fetches the challenge URL itself before it lets Let’s Encrypt validate, and while that self-check fails, kubectl -n monitoring describe challenges shows this reason. Work through the causes in order:

• the A record does not resolve to your public address yet
• port 80 is not forwarded to node port 30080
• the cluster cannot reach its own public address (no NAT loopback, or split DNS)

The first two are fixed at the DNS provider and the router. For the third, the docs point to the solver’s waitInsteadOfSelfCheck option, which skips the self-check and waits a set time instead, or to a DNS-01 solver.

Error from server (NotFound): secrets “prometheus-grafana” not found

That is what you get if you follow the NOTES printed by helm install while using grafana.admin.existingSecret. The chart never creates its own admin Secret in that case. Read the password from grafana-admin, the Secret you created.

Traefik Service stays at EXTERNAL-IP <pending>

A LoadBalancer Service on a bare-metal or VM cluster with no load balancer implementation never gets an address. We hit it because --set service.type=NodePort is silently ignored by the current Traefik chart. Use --set service.spec.type=NodePort, or install MetalLB if you want real LoadBalancer addresses on bare metal.