Deploy Keycloak on Kubernetes with the Operator and CloudNativePG

The Keycloak Operator turns a three-pod Keycloak into one custom resource. Keeping that Keycloak up when a node dies takes a few settings the operator does not apply for you. We powered off the node under the PostgreSQL primary four times: with the defaults every login failed for 16 minutes, and with all of them in place the outage was 80 seconds.

Original content from computingforgeeks.com - post 173471

This guide builds the whole stack on a three-node k3s cluster. CloudNativePG runs a replicated PostgreSQL, cert-manager handles TLS through the Traefik ingress, and the Keycloak Operator runs the Keycloak pods. On top sit a realm loaded from a KeycloakRealmImport, a permanent admin with TOTP, and an optional Grafana login through the KeycloakOIDCClient resource. If one host is all you need, the Keycloak on Docker or Podman guide is less work, and the Keycloak vs authentik vs Zitadel vs Authelia comparison covers whether Keycloak is the right identity server at all. Everything below was run in October 2026 on three Ubuntu 26.04 nodes with k3s v1.36, Keycloak Operator 26.8.0 and CloudNativePG 1.30.1, then replayed on a single Ubuntu 24.04 node.

How the pieces fit

Traefik terminates TLS and is the only thing that faces clients. The Keycloak Operator owns the Keycloak StatefulSet, CloudNativePG owns the database, and each of them keeps one pod per node so a single node failure never takes out a whole tier.

Keycloak keeps sessions in its embedded Infinispan cache, which the operator clusters across the pods with JGroups (mTLS between pods is on by default). PostgreSQL holds everything durable: realms, users, clients, credentials. Lose the database and Keycloak stops serving logins, which is why most of this guide is about the database path.

What you need

The core path needs a cluster of three Linux nodes, a DNS name for Keycloak and a shell with kubectl on one of the nodes. Each optional section lists its own extras.

Section	Needed for	Extra things you need
Core path (k3s to groups claim)	Everyone	3 nodes, one DNS A record, ports 80 and 443 reachable from the internet (or a Cloudflare zone for DNS-01)
Grafana login	Testing SSO with a real app	A second DNS name for Grafana
Backup and upgrades	Everyone running this for real	Somewhere to keep dump files

Size the nodes from the workload, not from our lab. The Keycloak HA guide requests 2 CPU and 1250 MB of memory per pod in its example, sized with its CPU and memory sizing guide. CPU grows with logins per second, because every password login runs the password hash iterations of the realm. PostgreSQL needs RAM for its cache and storage with fast fsync, because user, client and credential changes are written there. Keep the three nodes identical, since both the etcd quorum and the PostgreSQL replicas run at the pace of the slowest member, and keep them on one low-latency network. Our nodes had 4 vCPU, 8 GB RAM and a 40 GB disk each, which is a floor for following along, not a production recommendation.

Set the shell variables

Two values change from reader to reader: the Keycloak hostname and the email the ACME account uses for expiry notices. Export them on the node where you will run kubectl:

export KC_HOST="keycloak.example.com"
export ACME_EMAIL="[email protected]"

These exports last for the current shell only. Run the block again after you reconnect. Point an A record for the hostname at the node IPs (all three, or the one your load balancer uses) before you request a certificate.

Allow the TCP retry sysctl in k3s

This step exists because of the failover drills further down. When a node loses power, the Keycloak pods keep TCP connections to the dead PostgreSQL primary, and Linux retries a stalled connection for about 15 minutes before it gives up. Lowering net.ipv4.tcp_retries2 to 5 cuts that to seconds. The setting is per network namespace, so the host value doesn’t reach the pods. Kubernetes also treats it as an unsafe sysctl, which the kubelet must allow first.

There is a trade-off. At 5, the kernel gives up after about 12.6 seconds of retransmissions, well below the 100 seconds RFC 1122 recommends, so a network stall that long also breaks healthy connections. The pool reopens them, which costs little for Keycloak. If your network has longer stalls, 8 (about 100 seconds) is a middle ground that still beats 15 minutes.

Create the k3s config file on every node before installing k3s (or on an existing cluster, one node at a time):

sudo mkdir -p /etc/rancher/k3s
sudo vim /etc/rancher/k3s/config.yaml

Add the kubelet allowlist:

kubelet-arg:
  - "allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding,net.ipv4.tcp_retries2"

Keep the two forwarding entries. k3s already passes an allowlist with net.ipv4.ip_forward for its ServiceLB pods. A kubelet-arg replaces that list. It does not extend it. Our first attempt allowed only tcp_retries2, and every ServiceLB pod failed with this event, which leaves nothing listening on ports 80 and 443:

Warning   SysctlForbidden   pod/svclb-traefik-b4e835d7-lt7cb   forbidden sysctl: "net.ipv4.ip_forward" not allowlisted

On kubeadm clusters, only net.ipv4.tcp_retries2 goes into the kubelet configuration as allowedUnsafeSysctls, since there is no ServiceLB list to keep. If Pod Security Admission enforces the baseline or restricted level on the keycloak namespace, it rejects unsafe sysctls too, so label that namespace pod-security.kubernetes.io/enforce=privileged. On an existing k3s cluster, restart k3s on one node at a time after editing the file, so etcd keeps its quorum:

sudo systemctl restart k3s

Install the k3s cluster

Skip this section if you already run a k3s cluster with the config file above. Otherwise, the first server starts the embedded etcd cluster. Generate a join token and keep it:

TOKEN=$(openssl rand -hex 24)
echo "$TOKEN" > ~/.k3s-token
chmod 600 ~/.k3s-token
curl -sfL https://get.k3s.io | K3S_TOKEN="$TOKEN" sh -s - server --cluster-init --write-kubeconfig-mode 644

The other two servers join it. Run this on each of them, with the token from the first node and its IP in place of 10.0.1.51:

curl -sfL https://get.k3s.io | K3S_TOKEN="paste-the-token-here" sh -s - server --server https://10.0.1.51:6443 --write-kubeconfig-mode 644

Back on the first node, give your user a kubeconfig and install Helm, which CloudNativePG and Grafana use later. The k3s on Ubuntu 26.04 guide covers the single-node variant and the firewall ports in more depth.

mkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
chmod 600 ~/.kube/config
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4 | bash
kubectl get nodes -o wide

All three servers should report Ready with the control-plane and etcd roles:

NAME    STATUS   ROLES                AGE     VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION             CONTAINER-RUNTIME
k8s01   Ready    control-plane,etcd   2m47s   v1.36.5+k3s1   10.0.1.51     <none>        Ubuntu 26.04.1 LTS   7.0.0-15-generic (amd64)   containerd://2.3.4-k3s1.36
k8s02   Ready    control-plane,etcd   37s     v1.36.5+k3s1   10.0.1.52     <none>        Ubuntu 26.04.1 LTS   7.0.0-15-generic (amd64)   containerd://2.3.4-k3s1.36
k8s03   Ready    control-plane,etcd   13s     v1.36.5+k3s1   10.0.1.53     <none>        Ubuntu 26.04.1 LTS   7.0.0-15-generic (amd64)   containerd://2.3.4-k3s1.36

On a freshly cloned VM, wait for cloud-init status to report done before installing. Our Ubuntu 24.04 template rebooted once during its first-boot package upgrade and cut the k3s installer off with Failed to wait for response: Connection reset by peer. Rerunning the installer after the reboot was enough.

Install cert-manager and an ACME issuer

cert-manager issues the certificate that Traefik serves for Keycloak. Install the latest release straight from its manifest:

CM_VER=$(curl -s https://api.github.com/repos/cert-manager/cert-manager/releases/latest | grep tag_name | cut -d '"' -f 4)
echo "cert-manager ${CM_VER}"
kubectl apply -f "https://github.com/cert-manager/cert-manager/releases/download/${CM_VER}/cert-manager.yaml"
kubectl -n cert-manager rollout status deploy/cert-manager-webhook --timeout=180s

The HTTP-01 issuer is the simplest choice: the ACME server fetches a token over port 80 through Traefik, so the hostname must resolve to your nodes from the internet. Create the issuer file:

vim cluster-issuer-http01.yaml

Paste the issuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ACME_EMAIL_HERE
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik

ACME_EMAIL_HERE is a placeholder, because the file is not a shell script. Swap in your email and apply it:

sed -i "s/ACME_EMAIL_HERE/${ACME_EMAIL}/" cluster-issuer-http01.yaml
kubectl apply -f cluster-issuer-http01.yaml
kubectl get clusterissuer letsencrypt

If your nodes are not reachable from the internet, use DNS-01 instead. For a zone on Cloudflare, create an API token with two permissions, Zone DNS Edit and Zone Zone Read, scoped to all zones or to the zone that holds your hostname. cert-manager reads it from a Secret in its own namespace. Read the token without echoing it, so it stays out of your shell history:

read -rs CF_API_TOKEN
kubectl -n cert-manager create secret generic cloudflare-api-token --from-literal=api-token="${CF_API_TOKEN}"
unset CF_API_TOKEN
vim cluster-issuer-dns01-cloudflare.yaml

Only the solver differs from the HTTP-01 issuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ACME_EMAIL_HERE
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

Apply it in place of the HTTP-01 file. Both issuers carry the letsencrypt name, so the rest of the guide works with either:

sed -i "s/ACME_EMAIL_HERE/${ACME_EMAIL}/" cluster-issuer-dns01-cloudflare.yaml
kubectl apply -f cluster-issuer-dns01-cloudflare.yaml
kubectl get clusterissuer letsencrypt

Our lab nodes sit on a private network that no public ACME server can reach, so the lab certificates came from a cert-manager CA issuer with the same letsencrypt name. Both ACME issuers above passed kubectl apply --dry-run=server against the cluster. For a cluster with its own internal CA, the FreeIPA ACME with cert-manager guide shows the same pattern with a private ACME server, and the cert-manager install guide goes deeper into issuers.

Run PostgreSQL with CloudNativePG

Keycloak supports PostgreSQL as a first-class database, and CloudNativePG gives you a primary with two streaming replicas, automatic failover and a -rw Service that always points at the current primary. The Keycloak high availability guides use it too.

Run two operator replicas, spread over different nodes. With the chart default of one replica, the operator can die with the node that holds the primary. Nothing promotes a replica until Kubernetes evicts and reschedules the operator, and that takes over five minutes. The drill section has the numbers. Create a values file:

vim cnpg-operator-ha-values.yaml

Two replicas, at most one per node:

replicaCount: 2
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: cloudnative-pg

Install the operator with those values:

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update
helm upgrade --install cnpg cnpg/cloudnative-pg --namespace cnpg-system --create-namespace -f cnpg-operator-ha-values.yaml --wait
kubectl -n cnpg-system get pods -o wide

The two operator pods land on different nodes, and only one holds the leader lease at a time:

NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
cnpg-cloudnative-pg-7964b6b8b-btdpw   1/1     Running   0          62m   10.42.0.30   k8s01   <none>           <none>
cnpg-cloudnative-pg-7964b6b8b-z74cg   1/1     Running   0          54m   10.42.2.25   k8s03   <none>           <none>

Next comes the database cluster itself. Create the namespace and the manifest:

kubectl create namespace keycloak
vim keycloak-db.yaml

Three instances, a keycloak database owned by a keycloak role, and required anti-affinity so no two instances share a node:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: keycloak-db
  namespace: keycloak
spec:
  instances: 3
  storage:
    size: 10Gi
  bootstrap:
    initdb:
      database: keycloak
      owner: keycloak
  postgresql:
    parameters:
      max_connections: "200"
  affinity:
    enablePodAntiAffinity: true
    topologyKey: kubernetes.io/hostname
    podAntiAffinityType: required
  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      memory: 1Gi

max_connections is 200 because each of the three Keycloak pods opens a fixed pool of 30 connections. That makes 90, and the realm import Job starts its own Keycloak with another pool. The remaining headroom covers CloudNativePG, replication and your own psql sessions. Apply it and wait for the cluster:

kubectl apply -f keycloak-db.yaml
kubectl -n keycloak wait cluster/keycloak-db --for=condition=Ready --timeout=600s
kubectl -n keycloak get cluster keycloak-db

Our cluster took just under three minutes to bring up all three instances:

NAME          AGE    INSTANCES   READY   STATUS                     PRIMARY
keycloak-db   3m2s   3           3       Cluster in healthy state   keycloak-db-1

CloudNativePG generated the application credentials in the keycloak-db-app Secret and a CA in keycloak-db-ca. Keycloak reads both, so nobody types a database password.

Install the Keycloak Operator

The official operator Helm chart is still experimental and installs the operator cluster-wide, so this guide uses the kustomize manifests, which watch only the namespace you apply them to (OLM clusters can use OperatorHub instead). Detect the latest release and apply its manifests into the keycloak namespace:

KC_VER=$(curl -s https://api.github.com/repos/keycloak/keycloak/releases/latest | grep tag_name | cut -d '"' -f 4)
echo "Keycloak ${KC_VER}"
kubectl -n keycloak apply -k "github.com/keycloak/keycloak-k8s-resources/kubernetes?ref=${KC_VER}"
kubectl -n keycloak rollout status deploy/keycloak-operator --timeout=300s

The kustomize URL needs git on the node. The operator registers four CRDs, and its version decides which Keycloak image the Keycloak CR runs:

Keycloak 26.8.0
customresourcedefinition.apiextensions.k8s.io/keycloakoidcclients.k8s.keycloak.org created
customresourcedefinition.apiextensions.k8s.io/keycloakrealmimports.k8s.keycloak.org created
customresourcedefinition.apiextensions.k8s.io/keycloaks.k8s.keycloak.org created
customresourcedefinition.apiextensions.k8s.io/keycloaksamlclients.k8s.keycloak.org created
deployment "keycloak-operator" successfully rolled out

Create the admin secrets

The operator creates two accounts on first start, from Secrets you provide: a temporary admin user for the console, and a service-account client that the operator uses for realm imports and client resources. Both get random secrets:

kubectl -n keycloak create secret generic keycloak-bootstrap-admin --from-literal=username=temp-admin --from-literal=password="$(openssl rand -base64 24)"
kubectl -n keycloak create secret generic keycloak-admin --from-literal=client-id=keycloak-operator-admin --from-literal=client-secret="$(openssl rand -hex 32)"

The bootstrap admin is meant to be replaced, and a later section does exactly that. The service-account client stays: it is also the recovery path if every human admin gets locked out.

Keycloak verifies the PostgreSQL server certificate against the CloudNativePG CA, so copy the CA into a ConfigMap the Keycloak pods can mount:

kubectl -n keycloak create configmap keycloak-db-ca --from-literal=ca.crt="$(kubectl -n keycloak get secret keycloak-db-ca -o jsonpath='{.data.ca\.crt}' | base64 -d)"

The ConfigMap is a copy. If you ever rebuild or restore the CloudNativePG cluster, it gets a new CA, so delete and recreate the ConfigMap with the same command and restart Keycloak, or every pod fails the TLS check against the database.

Deploy Keycloak

Request the certificate first, so the ingress has its TLS secret when Keycloak comes up. The Certificate manifest:

vim keycloak-certificate.yaml

One DNS name, issued by the letsencrypt ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: keycloak-tls
  namespace: keycloak
spec:
  secretName: keycloak-tls
  dnsNames:
    - KC_HOST_HERE
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer

Fill in the hostname, apply it and wait for the certificate:

sed -i "s/KC_HOST_HERE/${KC_HOST}/" keycloak-certificate.yaml
kubectl apply -f keycloak-certificate.yaml
kubectl -n keycloak wait certificate/keycloak-tls --for=condition=Ready --timeout=300s

The ingress from the operator answers plain HTTP on port 80 as well, and Keycloak served its login pages over it in our first run. A Traefik middleware turns those requests into redirects:

vim redirect-https.yaml

Paste the middleware:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: redirect-https
  namespace: keycloak
spec:
  redirectScheme:
    scheme: https
    permanent: true

Apply it:

kubectl apply -f redirect-https.yaml

Now the Keycloak resource. Create the file:

vim keycloak.yaml

This is the exact manifest from the lab:

apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: keycloak
  namespace: keycloak
spec:
  instances: 3
  db:
    vendor: postgres
    host: keycloak-db-rw
    database: keycloak
    usernameSecret:
      name: keycloak-db-app
      key: username
    passwordSecret:
      name: keycloak-db-app
      key: password
    poolMinSize: 30
    poolInitialSize: 30
    poolMaxSize: 30
  http:
    httpEnabled: true
  hostname:
    hostname: KC_HOST_HERE
  proxy:
    headers: xforwarded
  ingress:
    enabled: true
    className: traefik
    tlsSecret: keycloak-tls
    annotations:
      traefik.ingress.kubernetes.io/router.middlewares: keycloak-redirect-https@kubernetescrd
  scheduling:
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: keycloak
            app.kubernetes.io/instance: keycloak
  bootstrapAdmin:
    user:
      secret: keycloak-bootstrap-admin
    service:
      secret: keycloak-admin
  features:
    enabled:
      - client-admin-api:v2
  networkPolicy:
    enabled: true
    http:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: kube-system
        podSelector:
          matchLabels:
            app.kubernetes.io/name: traefik
      - podSelector:
          matchLabels:
            app.kubernetes.io/name: keycloak-operator
  additionalOptions:
    - name: metrics-enabled
      value: "true"
    - name: db-tls-mode
      value: verify-server
  truststores:
    cnpg:
      configMap:
        name: keycloak-db-ca
  unsupported:
    podTemplate:
      spec:
        securityContext:
          sysctls:
            - name: net.ipv4.tcp_retries2
              value: "5"

Most of it is plain configuration. These fields matter before you change anything:

• scheduling.topologySpreadConstraints with DoNotSchedule: the default spread from the operator is only ScheduleAnyway, and our first deployment put two of the three pods on one node. Twice, in two separate deployments.
• unsupported.podTemplate sets tcp_retries2 inside the pods, which is the fix the k3s step prepared for. The operator documents unsupported fields as outside its support guarantees.
• db-tls-mode: verify-server with the cnpg truststore encrypts the JDBC connections and checks the server certificate, as the Keycloak HA guide recommends. Equal pool sizes (30, 30, 30) allow statement caching, again per that guide.
• proxy.headers: xforwarded makes Keycloak trust the scheme and host Traefik passes along, and httpEnabled lets Traefik talk to the pods on 8080 inside the cluster.
• networkPolicy limits port 8080 to Traefik and the operator pod, which calls the Keycloak admin API for the client resources.
• features.enabled: client-admin-api:v2 powers the KeycloakOIDCClient resource in the optional Grafana section. It does nothing if you never create one.

Fill in the hostname and apply:

sed -i "s/KC_HOST_HERE/${KC_HOST}/" keycloak.yaml
kubectl apply -f keycloak.yaml
kubectl -n keycloak wait keycloak/keycloak --for=condition=Ready --timeout=900s

Verify the deployment

Three Keycloak pods on three nodes, next to the three database instances:

kubectl -n keycloak get pods -o wide

Every node carries one Keycloak pod and one PostgreSQL instance. The two database restarts come from the failover drills later in this guide, when we powered off nodes, not from Keycloak:

NAME                                 READY   STATUS    RESTARTS   AGE     IP           NODE    NOMINATED NODE   READINESS GATES
keycloak-0                           1/1     Running   0          6m2s    10.42.1.46   k8s02   <none>           <none>
keycloak-1                           1/1     Running   0          6m35s   10.42.0.33   k8s01   <none>           <none>
keycloak-2                           1/1     Running   0          7m5s    10.42.2.31   k8s03   <none>           <none>
keycloak-db-1                        1/1     Running   1 (51m ago)   86m     10.42.2.18   k8s03   <none>           <none>
keycloak-db-2                        1/1     Running   0             40m     10.42.1.35   k8s02   <none>           <none>
keycloak-db-3                        1/1     Running   1 (87m ago)   3h1m    10.42.0.24   k8s01   <none>           <none>
keycloak-operator-6b756d5894-cv7nf   1/1     Running   0          68m     10.42.2.22   k8s03   <none>           <none>

Check that the database connections really use TLS. The query counts the Keycloak connections by TLS version on the primary:

PRIMARY=$(kubectl -n keycloak get cluster keycloak-db -o jsonpath='{.status.currentPrimary}')
kubectl -n keycloak exec "$PRIMARY" -c postgres -- psql -tAc "select s.ssl, s.version, count(*) from pg_stat_ssl s join pg_stat_activity a using (pid) where a.usename = 'keycloak' group by 1,2;"

All 90 pooled connections (three pods with 30 each) are on TLS 1.3:

t|TLSv1.3|90

From any machine that resolves the hostname, port 80 should redirect and the discovery document of the master realm should carry the https issuer:

curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" "http://${KC_HOST}/realms/master"
curl -s "https://${KC_HOST}/realms/master/.well-known/openid-configuration" | grep -o '"issuer":"[^"]*"'

A 301 to https and an issuer under your hostname mean Traefik, the certificate and the proxy headers all line up. From a fresh apply, the Keycloak resource reached Ready in about two minutes in our lab.

Replace the bootstrap admin

The temp-admin account has no MFA and a password sitting in a Secret. Replace it with a named admin that must enrol TOTP on first login, then delete it.

kcadm.sh runs inside a Keycloak pod, so no extra tooling is needed on the node. Log it in as the bootstrap admin, with the password piped over stdin so it never lands on a command line:

kubectl -n keycloak get secret keycloak-bootstrap-admin -o jsonpath='{.data.password}' | base64 -d | \
  kubectl -n keycloak exec -i keycloak-0 -- bash -c 'read -r KC_CLI_PASSWORD; export KC_CLI_PASSWORD; /opt/keycloak/bin/kcadm.sh config credentials --server http://localhost:8080 --realm master --user temp-admin --config /tmp/kcadm.config'

A shell function saves typing for the rest of this guide:

kcadm() { kubectl -n keycloak exec -i keycloak-0 -- /opt/keycloak/bin/kcadm.sh "$@" --config /tmp/kcadm.config; }

Create the permanent admin with a random password in its own Secret, require TOTP, and grant the admin role of the master realm. Swap [email protected] for a mailbox you actually read, because password reset mail for this account goes there:

export KC_ADMIN_USER="kcadmin"
kubectl -n keycloak create secret generic keycloak-permanent-admin --from-literal=username="${KC_ADMIN_USER}" --from-literal=password="$(openssl rand -base64 24)"
kcadm create users -r master -s username="${KC_ADMIN_USER}" -s enabled=true -s [email protected] -s emailVerified=true -s "requiredActions=[\"CONFIGURE_TOTP\"]"
kubectl -n keycloak get secret keycloak-permanent-admin -o jsonpath="{.data.password}" | base64 -d | kubectl -n keycloak exec -i keycloak-0 -- bash -c "read -r P; /opt/keycloak/bin/kcadm.sh set-password -r master --username ${KC_ADMIN_USER} --new-password \"\$P\" --config /tmp/kcadm.config"
kcadm add-roles -r master --uusername "${KC_ADMIN_USER}" --rolename admin

Read the password back when you are ready to log in:

kubectl -n keycloak get secret keycloak-permanent-admin -o jsonpath='{.data.password}' | base64 -d; echo

Open https://keycloak.example.com/admin/ (your hostname), sign in as kcadmin, and Keycloak demands an authenticator app before the console opens:

Don’t delete temp-admin until kcadmin has enrolled TOTP and reached the console. That order keeps a way back in if the authenticator setup goes wrong. Once it works, switch kcadm to the service-account client of the operator, because a TOTP-protected human account cannot log in from a script. The client secret comes from the keycloak-admin Secret, again over stdin:

kubectl -n keycloak get secret keycloak-admin -o go-template='{{index .data "client-id" | base64decode}}{{"\n"}}{{index .data "client-secret" | base64decode}}{{"\n"}}' | \
  kubectl -n keycloak exec -i keycloak-0 -- bash -c 'read -r CID; read -r KC_CLI_CLIENT_SECRET; export KC_CLI_CLIENT_SECRET; /opt/keycloak/bin/kcadm.sh config credentials --server http://localhost:8080 --realm master --client "$CID" --config /tmp/kcadm.config'

Keycloak logs in as the service account:

Logging into http://localhost:8080 as service-account-keycloak-operator-admin of realm master

Now delete the bootstrap admin:

TID=$(kcadm get users -r master -q username=temp-admin -q exact=true --fields id --format csv --noquotes)
kcadm delete users/$TID -r master
kcadm get users -r master --fields username --format csv --noquotes

Only the permanent admin is left:

kcadmin

If kcadmin ever loses its authenticator, the service-account login above still works. Look up the user and its credentials first:

KID=$(kcadm get users -r master -q username="${KC_ADMIN_USER}" -q exact=true --fields id --format csv --noquotes)
kcadm get users/${KID}/credentials -r master --fields id,type

Delete the credential of type otp by its id, then put the enrolment back as a required action:

kcadm delete users/${KID}/credentials/paste-otp-credential-id-here -r master
kcadm update users/${KID} -r master -s 'requiredActions=["CONFIGURE_TOTP"]'

Don’t skip the second command. The OTP step in the default browser flow is conditional: a user without an OTP credential just skips it. When we deleted the credential alone, a password was enough to reach the admin console. With the required action set, the same login went to the TOTP setup page. The /tmp/kcadm.config file lives inside the pod, so rerun the service-account login after a pod restart.

Import a realm

A KeycloakRealmImport creates a realm from a manifest through a one-shot Job. Passwords stay out of the manifest: the import substitutes placeholders from Secrets. Create those first:

kubectl -n keycloak create secret generic apps-realm-users --from-literal=alice-password="$(openssl rand -base64 18)" --from-literal=bob-password="$(openssl rand -base64 18)"
vim realm-apps.yaml

The realm has brute-force protection, a 12-character password policy, two groups and two users:

apiVersion: k8s.keycloak.org/v2beta1
kind: KeycloakRealmImport
metadata:
  name: apps-realm
  namespace: keycloak
spec:
  keycloakCRName: keycloak
  placeholders:
    ALICE_PASSWORD:
      secret:
        name: apps-realm-users
        key: alice-password
    BOB_PASSWORD:
      secret:
        name: apps-realm-users
        key: bob-password
  realm:
    realm: apps
    displayName: Example Apps
    enabled: true
    sslRequired: external
    loginWithEmailAllowed: true
    duplicateEmailsAllowed: false
    bruteForceProtected: true
    failureFactor: 5
    passwordPolicy: "length(12) and notUsername and notEmail"
    groups:
      - name: grafana-admins
      - name: grafana-viewers
    users:
      - username: alice
        email: [email protected]
        firstName: Alice
        lastName: Admin
        enabled: true
        emailVerified: true
        groups:
          - /grafana-admins
        credentials:
          - type: password
            value: "${ALICE_PASSWORD}"
            temporary: false
      - username: bob
        email: [email protected]
        firstName: Bob
        lastName: Viewer
        enabled: true
        emailVerified: true
        groups:
          - /grafana-viewers
        credentials:
          - type: password
            value: "${BOB_PASSWORD}"
            temporary: false

Leave clientScopes out of a realm import. When our first version listed a single custom groups scope there, the new realm got only that scope and offline_access, without profile, email, roles or basic, so every login lacked the usual claims. Adding the scope afterwards (next section) keeps the built-in ones.

kubectl apply -f realm-apps.yaml
kubectl -n keycloak wait keycloakrealmimport/apps-realm --for=condition=Done --timeout=600s
kubectl -n keycloak get jobs

The import Job finished in 22 seconds, and the running Keycloak pods picked up the realm without a restart:

NAME         STATUS     COMPLETIONS   DURATION   AGE
apps-realm   Complete   1/1           22s        23s

The import only creates. If the realm exists, it is not overwritten, so the CR is not a way to manage the realm over time. The Keycloak docs recommend deleting the CR after the import, which also removes the finished Job:

kubectl -n keycloak delete keycloakrealmimport apps-realm

The realm and both users stayed in place after the delete. We checked with kcadm get realms and kcadm get users -r apps.

The login page of the realm carries its display name, which is the quickest visual check:

Add a groups claim

Applications map Keycloak groups to their own roles through a groups claim, and Keycloak doesn’t send one by default. In a new shell, define the kcadm function again first, and after a pod restart rerun the service-account login. Then create a client scope with a group membership mapper and make it a realm default:

SCOPE_ID=$(kcadm create client-scopes -r apps -s name=groups -s protocol=openid-connect -s "attributes.\"include.in.token.scope\"=true" -i)
kcadm create client-scopes/${SCOPE_ID}/protocol-mappers/models -r apps -s name=groups -s protocol=openid-connect -s protocolMapper=oidc-group-membership-mapper -s "config.\"claim.name\"=groups" -s "config.\"full.path\"=false" -s "config.\"id.token.claim\"=true" -s "config.\"access.token.claim\"=true" -s "config.\"userinfo.token.claim\"=true"
kcadm update realms/apps/default-default-client-scopes/${SCOPE_ID}

Do this before you create clients. A realm default scope only attaches to clients created after it: in our realm the new grafana client got groups, while the built-in admin-cli did not. full.path=false sends grafana-admins rather than /grafana-admins, which keeps application role rules short.

The core setup is done: three Keycloak pods behind TLS, a replicated database, a realm with users and a groups claim, and an admin protected by TOTP. The Grafana section is optional. Backups, failover and upgrades apply to everyone.

Optional: log in to Grafana through Keycloak

This section needs a second DNS name pointing at the cluster. Set it first:

export GRAFANA_HOST="grafana.example.com"

The KeycloakOIDCClient resource creates the client in Keycloak from the cluster, with the client secret taken from a labelled Secret. It is a preview API (v2alpha1), so expect its fields to change between releases. Create the manifest:

vim grafana-oidc-client.yaml

The Secret and the client together:

apiVersion: v1
kind: Secret
metadata:
  name: grafana-oidc
  namespace: keycloak
  labels:
    operator.keycloak.org/kind: KeycloakOIDCClient
stringData:
  client-secret: GRAFANA_CLIENT_SECRET_HERE
---
apiVersion: k8s.keycloak.org/v2alpha1
kind: KeycloakOIDCClient
metadata:
  name: grafana
  namespace: keycloak
spec:
  keycloakCRName: keycloak
  realm: apps
  client:
    displayName: Grafana
    enabled: true
    loginFlows:
      - STANDARD
    redirectUris:
      - https://GRAFANA_HOST_HERE/login/generic_oauth
    webOrigins:
      - https://GRAFANA_HOST_HERE
    auth:
      method: client-secret
      secretRef:
        name: grafana-oidc
        key: client-secret

Keep enabled: true. Our first version left it out, and the operator created the client with enabled: false, which no application can log in through. Generate the secret, fill both placeholders and apply:

export GRAFANA_CLIENT_SECRET="$(openssl rand -hex 32)"
sed -i -e "s/GRAFANA_HOST_HERE/${GRAFANA_HOST}/g" -e "s/GRAFANA_CLIENT_SECRET_HERE/${GRAFANA_CLIENT_SECRET}/" grafana-oidc-client.yaml
kubectl apply -f grafana-oidc-client.yaml
unset GRAFANA_CLIENT_SECRET
kubectl -n keycloak wait keycloakoidcclient/grafana --for=jsonpath='{.status.conditions[?(@.type=="HasErrors")].status}'=False --timeout=120s
kubectl -n keycloak get keycloakoidcclient grafana -o jsonpath='{.status.conditions[?(@.type=="HasErrors")].status}{"\n"}'

False means the operator created the client without errors. The v2 admin API behind this resource has no protocol mappers, so the groups claim has to come from a realm client scope. Deleting the CR deletes the client in Keycloak, so treat the CR as the source of truth for the client from now on.

Confirm the claim before wiring Grafana. Keycloak can generate an example ID token for any user and client:

CID=$(kcadm get clients -r apps -q clientId=grafana --fields id --format csv --noquotes)
UID_=$(kcadm get users -r apps -q username=alice -q exact=true --fields id --format csv --noquotes)
kcadm get clients/${CID}/evaluate-scopes/generate-example-id-token -r apps -q userId=${UID_} -q scope=openid | grep -A2 '"groups"'

The token for alice carries her group:

  "groups" : [ "grafana-admins" ],
  "preferred_username" : "alice",
  "given_name" : "Alice",

Grafana runs in its own namespace, so copy the client secret over:

kubectl create namespace grafana
kubectl -n keycloak get secret grafana-oidc -o jsonpath='{.data.client-secret}' | base64 -d | kubectl -n grafana create secret generic grafana-oauth --from-file=client-secret=/dev/stdin
vim grafana-values.yaml

The Helm values turn on generic OAuth, hide the local login form and map the groups claim to Grafana roles:

ingress:
  enabled: true
  ingressClassName: traefik
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    traefik.ingress.kubernetes.io/router.middlewares: keycloak-redirect-https@kubernetescrd
  hosts:
    - GRAFANA_HOST_HERE
  tls:
    - secretName: grafana-tls
      hosts:
        - GRAFANA_HOST_HERE
persistence:
  enabled: true
  size: 2Gi
envValueFrom:
  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET:
    secretKeyRef:
      name: grafana-oauth
      key: client-secret
grafana.ini:
  server:
    root_url: https://GRAFANA_HOST_HERE
  auth:
    disable_login_form: true
    signout_redirect_url: https://KC_HOST_HERE/realms/apps/protocol/openid-connect/logout?client_id=grafana
  auth.generic_oauth:
    enabled: true
    name: Keycloak
    allow_sign_up: true
    auto_login: false
    client_id: grafana
    scopes: openid email profile
    auth_url: https://KC_HOST_HERE/realms/apps/protocol/openid-connect/auth
    token_url: https://KC_HOST_HERE/realms/apps/protocol/openid-connect/token
    api_url: https://KC_HOST_HERE/realms/apps/protocol/openid-connect/userinfo
    use_pkce: true
    email_attribute_path: email
    login_attribute_path: preferred_username
    name_attribute_path: name
    groups_attribute_path: groups
    role_attribute_path: contains(groups[*], 'grafana-admins') && 'Admin' || 'Viewer'
    role_attribute_strict: true

role_attribute_strict: true refuses a login that matches no role rule, instead of quietly creating a Viewer. Fill in both hostnames and install the chart. The Grafana Helm chart now lives in the grafana-community repository. The newest chart in the old grafana repository still ships Grafana 12.3.

sed -i -e "s/GRAFANA_HOST_HERE/${GRAFANA_HOST}/g" -e "s/KC_HOST_HERE/${KC_HOST}/g" grafana-values.yaml
helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update
helm install grafana grafana-community/grafana -n grafana -f grafana-values.yaml --wait

Our lab added tls_client_ca pointing at the lab CA, because Grafana calls the Keycloak token endpoint and must trust its certificate. With a public ACME certificate, the system trust store already does. Open https://grafana.example.com:

Alice and bob each logged in through the realm, and Grafana assigned their roles from the groups claim:

The admin row is the local admin from the chart. With the login form disabled it only works through the API, and its password sits in the grafana Secret, which makes it the break-glass account if Keycloak is down. The client also shows up in the client list of the realm, created by the operator:

Keycloak high availability: what happens when a node dies

We tested failover the hard way: a probe ran a password login for bob every half second through Traefik, and we powered off the VM holding the PostgreSQL primary with qm stop. No graceful shutdown, no TCP resets. Four runs, each with one more fix:

Run	Setup	Logins failing	Recovered on its own
1	Defaults	16 min (07:57 to 08:13)	Yes, slowly
2	JDBC socketTimeout=30	10 min, one pod stuck	No, one pod stayed unready
3	tcp_retries2=5 in the pods	6 min 25 s	Yes, the moment the new primary was up
4	tcp_retries2=5 plus 2 CNPG operator replicas	80 s	Yes

The database was never the slow part. CloudNativePG promoted a replica in about 71 seconds in the first run. Keycloak stayed down for 15 minutes longer because its pods kept TCP connections to the IP of the dead primary, and nothing ever answers or resets those.

These are server-side numbers. The probe ran on a surviving node and sent its logins to Traefik on that node, so it never depended on the dead one. A client that resolves your hostname to the IP of the dead node fails until it tries another address, and plain multiple A records give no health checks. Put a load balancer, a kube-vip virtual IP or health-checked DNS in front of the nodes if clients must ride out a node loss too.

Error: “No executor queue space remaining”

The Keycloak pods logged this after the primary vanished, and their readiness probes timed out. A thread dump showed why: the readiness check validates a pooled connection, the validation blocks in a socket read on the dead connection, and every new health check queues behind it until the executor queue is full. A JDBC socketTimeout made it worse in run 2: one pod never recovered. With the default tcp_retries2 of 15, Linux retransmits unacknowledged data on a silent connection for about 15 minutes before it gives up. At 5, it gives up after roughly a dozen seconds, the pool opens a fresh connection through keycloak-db-rw, and readiness comes back.

Failover waited for the CloudNativePG operator

Run 3 fixed Keycloak but still failed for over six minutes, and the CloudNativePG events showed the failover starting late. The single operator pod had been on the dead node. Kubernetes first waits for the node to miss its heartbeats (40 to 50 seconds, depending on the release) before marking it NotReady, then lets pods tolerate the unreachable taint for another 300 seconds before evicting them. A new operator only started after more than five minutes, and only then did it promote a replica. With two operator replicas, the standby took the leader lease 59 seconds after the power-off:

Rolling updates were cheaper still: a configuration change that rolled all three Keycloak pods cost one failed request (a 502) in 397. To run the same drill on your own cluster, find the node that runs the primary first:

kubectl -n keycloak get pod "$(kubectl -n keycloak get cluster keycloak-db -o jsonpath='{.status.currentPrimary}')" -o jsonpath='{.spec.nodeName}{"\n"}'

Cut power to that node from your hypervisor or cloud console, not with shutdown: a clean shutdown sends TCP resets and hides exactly the problem this section is about. If you run helm upgrade on the CloudNativePG chart while a node is down, the new operator pod can sit in Pending and Helm marks the release failed, which is what happened to us. Run the upgrade again once the node is back.

Back up the database

Everything Keycloak knows lives in PostgreSQL, so a database dump is a full backup of realms, users, clients and credentials. A logical dump from the primary takes one command:

PRIMARY=$(kubectl -n keycloak get cluster keycloak-db -o jsonpath='{.status.currentPrimary}')
kubectl -n keycloak exec "$PRIMARY" -c postgres -- pg_dump -Fc -d keycloak > keycloak-$(date +%F).dump
ls -lh keycloak-*.dump

Our realm with two users and a handful of clients produced a 325 KB file. A backup only counts once it restores, so we restored it into a scratch CloudNativePG cluster. Create its manifest:

vim restore-test-db.yaml

One instance, an empty keycloak database and a new name:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: keycloak-db-restore
  namespace: keycloak
spec:
  instances: 1
  storage:
    size: 10Gi
  bootstrap:
    initdb:
      database: keycloak
      owner: keycloak

Start it, load the dump and count the users per realm:

kubectl apply -f restore-test-db.yaml
kubectl -n keycloak wait cluster/keycloak-db-restore --for=condition=Ready --timeout=600s
kubectl -n keycloak exec -i keycloak-db-restore-1 -c postgres -- pg_restore --no-owner --role=keycloak -d keycloak < keycloak-$(date +%F).dump
kubectl -n keycloak exec keycloak-db-restore-1 -c postgres -- psql -d keycloak -c "select r.name as realm, count(u.id) as users from realm r left join user_entity u on u.realm_id = r.id group by r.name order by r.name;"

Both realms came back with their users, in under a second:

 realm  | users
--------+-------
 apps   |     2
 master |     2
(2 rows)

The dump holds password hashes and OTP secrets, so store it like a credential. For point-in-time recovery, CloudNativePG streams WAL to object storage through its Barman Cloud plugin, and Keycloak documents a scheduled S3 backup setup for this exact stack. We didn’t test that path here, and the dump above is the minimum. The Patroni and HAProxy guide covers the same failover questions for PostgreSQL outside Kubernetes.

Upgrade Keycloak

The operator version decides the Keycloak image, and the new operator rolls every Keycloak resource that does not pin spec.image. Take a dump first, then apply the operator manifests of the release you are moving to. Set the version yourself here instead of asking the GitHub API for the newest one, so an upgrade never jumps a minor release by accident:

KC_VER="26.8.0"   # pin: target release from https://github.com/keycloak/keycloak/releases
kubectl -n keycloak apply -k "github.com/keycloak/keycloak-k8s-resources/kubernetes?ref=${KC_VER}"
kubectl -n keycloak rollout status deploy/keycloak-operator --timeout=300s

How the pods roll depends on spec.update.strategy. The default, RecreateOnImageChange, scales the StatefulSet to zero on any image change, even a patch release, so every upgrade is a short outage. To let the operator decide, add this to keycloak.yaml under spec and apply it before the upgrade:

  update:
    strategy: Auto

The Auto strategy runs a Job that checks whether old and new pods can share the cluster, and only rolls when they can. We measured it on a throwaway two-pod Keycloak, moving from 26.7.5 to 26.8.0. The check Job took 32 seconds (18 of them pulling the old image), then refused a rolling update and recorded why:

RecreateUpdateUsed=True Incompatible changes detected. Check update job for details.
[feature-compatibility] Rolling Update is not available. 'feature-compatibility.login-failures' is incompatible: null -> {"enabled":true,"version":2,"updatePolicy":"SHUTDOWN"}.

Both pods went down together, and requests failed for about 25 seconds. That is the normal case for a minor-version upgrade: Keycloak only promises rolling updates within a patch stream, so plan minor upgrades for a quiet window and expect a short outage. Within a patch stream, Auto should choose a rolling update instead, though we only measured the minor jump. The Keycloak rolling updates guide warns that Auto can misjudge a resource that uses unsupported.podTemplate, as ours does for the sysctl. Our template changes only a kernel setting, but test the first upgrade on a copy before trusting it in production. Take a fresh dump before every upgrade, because a minor release can migrate the database schema and there is no downgrade path.