A kubeadm cluster doesn’t fade out when its certificates lapse. It stops on the anniversary of kubeadm init, because every client certificate kubeadm signs is valid for exactly one year. kubectl fails with x509: certificate has expired or is not yet valid, the controller-manager loses its lease, and a worker whose kubelet client certificate also ran out drops to NotReady. A kubernetes certificate expired outage is at least dated and predictable, so once you know which file each error points at, the recovery is short.
We start from the error text, map each line to the certificate behind it, then renew the control plane with kubeadm certs renew all, restart the static pods and refresh admin.conf. After that we rebuild a worker’s kubelet.conf end to end, the part the official troubleshooting page compresses into seven terse steps. We built a two-node Kubernetes v1.37.2 cluster on Ubuntu 24.04 in November 2026 and moved both clocks 370 days forward, so every error line and output below comes from that cluster breaking and recovering.
How kubeadm certificates expire
kubeadm creates three certificate authorities at init time (ca, etcd-ca and front-proxy-ca) and signs everything else from them. The CAs get ten years, while every leaf certificate gets one year, and the failure lands on a single day because all leaf certificates were issued in the same second and so expire in the same second. Clusters that run kubeadm upgrade at least once a year never notice, because the upgrade renews every control-plane certificate as a side effect.
Kubelets are the exception to that clock. kubeadm sets rotateCertificates: true in the KubeletConfiguration, so each kubelet requests a fresh client certificate from the controller-manager’s signer before the old one runs out and stores it under /var/lib/kubelet/pki/. That rotation only works while the kubelet is running and can still authenticate. In our lab the control-plane kubelet on cp01 rotated as expected, while worker01, which we had stopped to play a powered-off node, kept its original certificate and came back with a dead one. The diagram shows which certificate each connection presents:
Each arrow is a TLS connection, and each one fails with a slightly different message when the certificate on either end has expired. The table below lists the files kubeadm certs renew all rewrites, plus the kubelet certificate it does not touch.
| Certificate | File | Who presents it | What fails when it expires |
|---|---|---|---|
| apiserver | /etc/kubernetes/pki/apiserver.crt | kube-apiserver, serving on 6443 | Every client that verifies the server, kubectl first |
| admin.conf | /etc/kubernetes/admin.conf (embedded) | kubectl as kubernetes-admin | kubectl gets 401 |
| controller-manager.conf, scheduler.conf | /etc/kubernetes/*.conf (embedded) | Controller-manager, scheduler | No reconciling, no scheduling, CSR signing stops |
| apiserver-etcd-client | /etc/kubernetes/pki/apiserver-etcd-client.crt | kube-apiserver talking to etcd | etcd refuses new API server connections |
| etcd-server, etcd-peer, etcd-healthcheck-client | /etc/kubernetes/pki/etcd/ | etcd | API server cannot verify etcd, HA members cannot peer |
| apiserver-kubelet-client | /etc/kubernetes/pki/apiserver-kubelet-client.crt | kube-apiserver calling kubelets on 10250 | kubectl logs and exec fail |
| front-proxy-client | /etc/kubernetes/pki/front-proxy-client.crt | Aggregation layer | Aggregated APIs such as metrics.k8s.io |
| kubelet client | /var/lib/kubelet/pki/kubelet-client-current.pem | kubelet on every node | Node goes NotReady; not renewed by kubeadm |
On worker nodes, the last row matters most. kubeadm certs check-expiration doesn’t list kubelet.conf at all, because the docs treat it as self-rotating. A check run on the control plane can therefore report every certificate healthy while a worker sits NotReady with an expired client certificate, which is why we check the kubelet separately below.
What each x509 error is telling you
Every expiry message on this cluster comes from the same line in the Go crypto/x509 package, which prints x509: certificate has expired or is not yet valid: followed by current time ... is after ... with two RFC 3339 timestamps. The wording is identical whichever certificate failed, so the useful information is where the line appears and which side of the connection logged it. When kubectl is dead, you read the API server through the container runtime instead. kubeadm doesn’t install crictl, so add the cri-tools package from the same pkgs.k8s.io repository first:
sudo apt-get install -y cri-toolsThen filter the log of the API server container for x509 lines, which shows who is still knocking with an old certificate:
sudo crictl logs $(sudo crictl ps -a --name kube-apiserver -q | head -1) 2>&1 | grep -i x509 | tail -5On our expired cp01 the API server was rejecting the clients that still presented year-old certificates (lines shortened at the certificate serial):
E1106 01:51:35.873297 1 authentication.go:76] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2027-11-06T01:51:35Z is after 2027-11-01T01:43:25Z, verifying certificate SN=8728276683957971603, ...
E1106 01:51:36.113533 1 authentication.go:76] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2027-11-06T01:51:36Z is after 2027-11-01T01:43:25Z, verifying certificate SN=115837504332588788, ...In the terminal on cp01 these lines are several hundred characters long (shortened again here), which is why the tail -5 matters on a busy API server:
On a worker, the kubelet journal is the place to look, and filtering it for certificate words keeps the noise down:
sudo journalctl -u kubelet --since "10 minutes ago" --no-pager | grep -iE 'x509|bootstrap|expired|Unauthorized' | tail -20Four messages came out of our lab, and each H3 below is the string you’ll find in one of those places.
Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid
kubectl printed this, so the expired certificate is the one the API server presented: apiserver.crt. kubectl never got far enough to send its own credentials. Here is kubectl get nodes on cp01, the first of five retries and then the final line:
E1106 01:52:26.466410 11109 memcache.go:381] "Couldn't get current server API group list" err="Get \"https://10.0.1.10:6443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2027-11-06T01:52:26Z is after 2027-11-01T01:43:25Z"
Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2027-11-06T01:52:26Z is after 2027-11-01T01:43:25ZThis is how it looked on the cp01 console, with the retry lines filtered out:
The controller-manager logged the same string while trying to hold its leader lease, which is why nothing reconciles during the outage. Its log showed "Error retrieving lease lock" err="Get \"https://10.0.1.10:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: ... every two to three seconds. etcd, by contrast, logged no x509 errors at all: it was still running with the certificates it loaded a year earlier, so its log is not where the outage shows.
error: You must be logged in to the server (the server has asked for the client to provide credentials)
Here the TLS handshake succeeded, which means the server certificate is fine, but the API server rejected the client certificate in your kubeconfig with a 401. Some kubectl versions and commands print (Unauthorized) instead. We hit it twice in the lab, both times with the new apiserver.crt already on disk and the stale ~/.kube/config still in place:
E1106 01:53:14.661826 11417 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)On a kubeadm cluster this is almost always a copy of admin.conf that went stale, and the fix is a fresh copy, covered further down.
part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired
The kubelet source file bootstrap.go logs this at startup when the client certificate it was about to use has passed its NotAfter date. kubeadm removes the bootstrap kubeconfig once a node has joined, so the kubelet’s fallback finds nothing and the process exits. These are the matching lines from worker01’s journal after we started its kubelet again (config-dump lines dropped; the next restart ten seconds later printed the same three):
Nov 06 01:52:44 worker01 kubelet[10318]: I1106 01:52:44.652061 10318 server.go:1036] "Client rotation is on, will bootstrap in background"
Nov 06 01:52:44 worker01 kubelet[10318]: E1106 01:52:44.653023 10318 bootstrap.go:267] "Unhandled Error" err="part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2027-11-01 01:41:08 +0000 UTC" logger="UnhandledError"
Nov 06 01:52:44 worker01 kubelet[10318]: E1106 01:52:44.653060 10318 run.go:72] "command failed" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"On worker01 the three lines repeat with every restart attempt:
The message names kubelet.conf, but on our worker that file only pointed at /var/lib/kubelet/pki/kubelet-client-current.pem; the date at the end of the line is the NotAfter of that PEM. systemd restarts the kubelet every ten seconds and it fails the same way each time (Active: activating (auto-restart) (Result: exit-code) in systemctl status kubelet), so the node stays NotReady until you rebuild the file.
x509: certificate has expired or is not yet valid: current time … is before …
Read the last clause carefully. “is after” means a real expiry, while “is before” means the NotBefore date of the certificate sits in the future from the point of view of this machine, which is a clock problem and not a certificate problem. A node whose clock jumped backwards, or a freshly renewed control plane talking to a node that is minutes behind, produces exactly this. Check the clock before renewing anything:
timedatectl statusYou want System clock synchronized: yes on every node. Ours read no with NTP service: inactive because we turned NTP off to move the clock; on a real cluster that combination is the thing to fix with chrony or systemd-timesyncd first, because renewing certificates on a host with the wrong time just mints a new set with the wrong validity window.
Which certificates expired on your cluster
kubeadm reads the expiry dates straight from the files on disk, so this check works even when the API server is down. It first tries the kubeadm-config ConfigMap and, on a broken cluster, prints a fallback warning before carrying on. Run it on a control-plane node:
sudo kubeadm certs check-expirationEvery leaf certificate on cp01 showed <invalid> in the RESIDUAL TIME column, including super-admin.conf, which exists only on the node where kubeadm init ran. The CAs still had eight years left:
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Nov 01, 2027 01:43 UTC <invalid> ca no
apiserver Nov 01, 2027 01:43 UTC <invalid> ca no
apiserver-etcd-client Nov 01, 2027 01:43 UTC <invalid> etcd-ca no
apiserver-kubelet-client Nov 01, 2027 01:43 UTC <invalid> ca no
controller-manager.conf Nov 01, 2027 01:43 UTC <invalid> ca no
etcd-healthcheck-client Nov 01, 2027 01:43 UTC <invalid> etcd-ca no
etcd-peer Nov 01, 2027 01:43 UTC <invalid> etcd-ca no
etcd-server Nov 01, 2027 01:43 UTC <invalid> etcd-ca no
front-proxy-client Nov 01, 2027 01:43 UTC <invalid> front-proxy-ca no
scheduler.conf Nov 01, 2027 01:43 UTC <invalid> ca no
super-admin.conf Nov 01, 2027 01:43 UTC <invalid> ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Oct 29, 2036 01:43 UTC 8y no
etcd-ca Oct 29, 2036 01:43 UTC 8y no
front-proxy-ca Oct 29, 2036 01:43 UTC 8y noOn the cp01 terminal the <invalid> column is the one to scan, since every row in it is a certificate kubeadm will renew:
The “Falling back to default configuration” line is expected on a dead cluster and harmless here. If the EXTERNALLY MANAGED column says yes for anything, kubeadm can’t renew that certificate because it does not hold the signing key. Because the kubelet certificate is missing from that output, check it directly on each node with openssl:
sudo openssl x509 -noout -subject -enddate -in /var/lib/kubelet/pki/kubelet-client-current.pemOn worker01 the subject was right and the date had passed five days earlier:
subject=O = system:nodes, CN = system:node:worker01
notAfter=Nov 1 01:41:08 2027 GMTA notAfter in the past means that node needs the rebuild further down. If the file is missing, or /etc/kubernetes/kubelet.conf carries client-certificate-data instead of a path, the same rebuild applies. For the same check against any other PEM file or remote endpoint, our openssl expiration guide has the variants.
How kubeadm certs renew all fixes the control plane
The renewal re-signs every leaf certificate with the CA keys already sitting in /etc/kubernetes/pki, reusing the existing Common Name, Organization and SANs from the old files rather than reading the kubeadm-config ConfigMap. That’s what makes it safe on a cluster whose API is down, because nothing it signs depends on the cluster answering. Back up the whole directory first, since a renewal overwrites files in place:
sudo cp -a /etc/kubernetes /etc/kubernetes.bak-$(date +%F)With the backup in place, renew everything kubeadm manages:
sudo kubeadm certs renew allAfter the same fallback warning, kubeadm names each of the 11 certificates and ends with the instruction most people skip:
...
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
certificate embedded in the kubeconfig file for the super-admin renewed
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.On the cp01 terminal the same run ends on the line you have to act on:
One thing surprised us. Seconds after the renewal, before any restart, kubectl stopped printing the x509 error and switched to the credentials error from earlier. kube-apiserver had already picked the new serving certificate up from disk, so the TLS handshake passed and only the stale client certificate in ~/.kube/config failed. That partial recovery is easy to misread as “fixed”. etcd, the controller-manager and the scheduler still hold their old certificates in memory, and the upstream docs say plainly that dynamic reload is not supported for all components, so the restart is still required.
Why the static pods need a restart
Control-plane components run as static pods, which the kubelet starts from YAML files in /etc/kubernetes/manifests/, not the API server. So kubectl delete pod can’t restart them. The documented method is to take the manifests away from the kubelet, wait for it to stop the pods, then give them back. Move them to a directory outside the manifests path:
sudo mkdir -p /etc/kubernetes/manifests-paused
sudo mv /etc/kubernetes/manifests/*.yaml /etc/kubernetes/manifests-paused/The docs suggest waiting 20 seconds (the kubelet’s fileCheckFrequency), but kube-apiserver takes longer to drain. Check with crictl:
sudo crictl ps --name 'kube-apiserver|kube-controller-manager|kube-scheduler|etcd'At 25 seconds our API server was still up while the other three had gone:
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
be682b2421356 7a918dfe48696 12 months ago Running kube-apiserver 0 4b2ee4896eb4e kube-apiserver-cp01 kube-systemAt 45 seconds only the header row printed. Wait for that empty list, then put the manifests back and the kubelet recreates each pod with the renewed certificates:
sudo mv /etc/kubernetes/manifests-paused/*.yaml /etc/kubernetes/manifests/
sudo rmdir /etc/kubernetes/manifests-pausedRun the same crictl ps about 40 seconds later and all four should be back with ATTEMPT 0 and a fresh CREATED time:
Moving all four at once is fine on a single control-plane node, since the cluster is already down. On an HA control plane, the section further down explains why you go one node at a time instead.
Why admin.conf has to be copied again
The renewal rewrote /etc/kubernetes/admin.conf, but kubectl reads ~/.kube/config, which on most kubeadm clusters is a copy made on install day. That copy still embeds the expired client certificate, so after the restart kubectl still printed You must be logged in to the server (the server has asked for the client to provide credentials). Replace it:
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configNow ask the cluster for its nodes:
kubectl get nodes -o wideThe control plane answered again, and the output separates the two problems cleanly. cp01 was Ready because its own kubelet had rotated, while worker01 stayed NotReady:
A second kubeadm certs check-expiration now shows every leaf certificate at Nov 05, 2028 with 364d remaining. If you handed copies of admin.conf to CI jobs or colleagues, those copies are expired too and need the new file. kubeadm also renews super-admin.conf, the break-glass credential in the system:masters group. Keep that one on the node rather than in a personal kubeconfig.
How to rebuild an expired kubelet.conf on a worker
A worker with a dead client certificate cannot request a new one, because the request itself has to be authenticated. The way out is to sign a fresh client certificate for that node’s identity on a control-plane node, which holds ca.key, and hand it to the kubelet as a one-time credential. The upstream troubleshooting page lists this as seven steps without the commands, so the full sequence follows in order:
- Back up the worker’s
kubelet.confand PKI directory, and note the API server endpoint it used. - Sign a new kubeconfig for
system:node:<name>on a control-plane node. - Install it on the worker and restart the kubelet so it writes
kubelet-client-current.pem. - Repoint
kubelet.confat that rotating file and restart again.
The node name must match exactly what kubectl get nodes shows, because the Node authorizer in the API server ties the certificate’s CN to that Node object. On the worker, back up and look at the old file before touching anything:
sudo mkdir -p /root/kubelet-backup
sudo cp -a /etc/kubernetes/kubelet.conf /var/lib/kubelet/pki /root/kubelet-backup/
sudo grep -E 'server:|client-certificate|client-key' /etc/kubernetes/kubelet.confworker01’s file was a normal kubeadm join result, pointing at the rotating PEM that had expired:
server: https://10.0.1.10:6443
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pemNote the server: URL (on an HA cluster it’s the load balancer address). If your file shows client-certificate-data instead, the node never rotated at all, and the same fix applies. Now switch to the control-plane node and export the two values the next commands reuse:
export NODE=worker01
export APISERVER=https://10.0.1.10:6443Generate a kubeconfig whose client certificate carries the organization system:nodes and the CN system:node:worker01, which is the identity the kubelet authenticates as. The umask 077 keeps the file, which carries a private key, readable only by you from the moment it is written:
umask 077
sudo kubeadm kubeconfig user --org system:nodes --client-name "system:node:${NODE}" > "kubelet-${NODE}.conf"Without a --config file, kubeadm fills the server and cluster name from its defaults, and the docs warn that you may need to adjust both by hand. Compare the generated server line with the one you noted on the worker:
grep -E 'server:|cluster:|name:' "kubelet-${NODE}.conf"On our single control-plane cluster the defaults matched, so the file was usable as generated:
- cluster:
server: https://10.0.1.10:6443
name: kubernetes
cluster: kubernetes
name: system:node:worker01@kubernetes
- name: system:node:worker01If the server URL differs, which is common when the cluster sits behind a load balancer in front of the API server, set it to the worker’s original value:
sed -i "s#server: .*#server: ${APISERVER}#" "kubelet-${NODE}.conf"We copy the file over SSH and delete the local copy straight after, because it holds a private key for the node identity. Replace 10.0.1.11 with the worker’s address:
scp "kubelet-${NODE}.conf" 10.0.1.11:/tmp/kubelet.conf
rm -f "kubelet-${NODE}.conf"Back on the worker, remove the old kubeconfig and the dead client certificates, then install the new file with root-only permissions. The sh -c wrapper matters because /var/lib/kubelet is mode 0750, so your own shell can’t expand the glob:
sudo sh -c 'rm -f /etc/kubernetes/kubelet.conf /var/lib/kubelet/pki/kubelet-client*'
sudo install -o root -g root -m 600 /tmp/kubelet.conf /etc/kubernetes/kubelet.conf
sudo rm -f /tmp/kubelet.conf
sudo systemctl restart kubeletThe kubelet treats the embedded certificate as a bootstrap credential: it authenticates with it, files a CSR for a proper rotating certificate, and writes the result to the PKI directory. Give it ten seconds and list the directory:
sudo ls -l /var/lib/kubelet/pki/A new timestamped PEM and the kubelet-client-current.pem symlink appeared within those ten seconds, with no manual CSR approval:
total 12
-rw------- 1 root root 1110 Nov 6 01:55 kubelet-client-2027-11-06-01-55-37.pem
lrwxrwxrwx 1 root root 59 Nov 6 01:55 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2027-11-06-01-55-37.pem
-rw-r--r-- 1 root root 2270 Nov 1 2026 kubelet.crt
-rw------- 1 root root 1675 Nov 1 2026 kubelet.keyUntil that symlink exists, the next step has nothing to point at, so don’t rush it. If it never appears, check the kubelet journal for a new error before continuing.
The kubelet now rotates from that file, but kubelet.conf still embeds the certificate you installed, and that embedded copy never changes. On a restart after it expires, the kubelet would load the stale embedded certificate and fail again. Swap the two embedded lines for references to the rotating file, which is the edit that turns a one-year fix into a permanent one:
sudo sed -i \
-e 's#client-certificate-data: .*#client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem#' \
-e 's#client-key-data: .*#client-key: /var/lib/kubelet/pki/kubelet-client-current.pem#' \
/etc/kubernetes/kubelet.conf
sudo systemctl restart kubeletConfirm the edit landed, the kubelet stayed up and the certificate it now uses is the new one:
sudo grep -E 'client-certificate|client-key' /etc/kubernetes/kubelet.conf
systemctl is-active kubelet
sudo openssl x509 -noout -subject -enddate -in /var/lib/kubelet/pki/kubelet-client-current.pemOn worker01, with the PKI listing from the previous step on top, both client lines name the rotating file, the service is active, and the certificate runs for another year:
The NotAfter lands five minutes short of a full year after the file timestamp because the signer in the controller-manager backdates NotBefore to absorb clock skew. Finish on the control-plane node:
kubectl get nodes
kubectl get csrworker01 was Ready within a minute of the kubelet.conf swap, and the CSR it filed during the restart (csr-7mclp) shows Approved,Issued:
The list also tells the history of the cluster: csr-q8ffz is worker01’s original join, and csr-xfcq6 is the cp01 kubelet rotating 35 days before expiry, which is exactly what the worker missed while it was off. (After our clock jump the cp01 kubelet only noticed once restarted; a kubelet running through a real year rotates by itself.) Auto-approval works because kubeadm binds nodes to the role that lets the controller-manager approve their own client certificates. A CSR stuck in Pending usually means the controller-manager is still running with its old certificate, so go back and check the static pod restart. Repeat the whole sequence for each NotReady worker, changing NODE and the scp target each time.
What changes on an HA control plane
Certificates live on each control-plane node separately, and kubeadm certs renew all only touches the node it runs on. The kubeadm docs are explicit that you run it on every control-plane node. Run the backup, renewal and static pod restart on one node, wait until its API server and etcd member are healthy, then move to the next. Restarting every etcd member at the same time costs you quorum for as long as the slowest one takes to come back.
Any control-plane node can run the worker rebuild, since all of them share the same CA. The server: line in the worker’s kubelet.conf must keep pointing at the load balancer or VIP, never at one control-plane node directly. If you built the cluster from our kubeadm HA cluster guide, that is the VIP or HAProxy address on 6443. Our lab had a single control plane, so this section follows the kubeadm docs rather than a captured run.
How to stop it happening next year
Upgrades are the cheapest prevention, because kubeadm renews every control-plane certificate during one, and a cluster that follows the kubeadm upgrade process at least once a year never reaches its expiry date. Kubernetes ships a minor release roughly every four months, so a team that stays within support upgrades often enough by default. The clusters that hit this outage are the stable ones nobody wanted to touch, such as a home Kubernetes cluster that just worked for twelve months.
Beyond upgrades, these are the settings and checks that catch it early:
- Expiry alert:
openssl x509 -checkend 2592000exits non-zero when a certificate expires within 30 days, so a daily cron job againstapiserver.crton each control-plane node is enough to page someone. - Kubelet metric: each kubelet exports
kubelet_certificate_manager_client_ttl_seconds, and an alert on a low value catches a worker whose rotation has stalled before it goes NotReady. - Validity period: new clusters can set
certificateValidityPeriodin the v1beta4 ClusterConfiguration (default8760h). A longer value pushes the date out, but it also extends the life of a leaked admin credential, so treat it as a trade, not a fix. - Long-off nodes: a worker that is powered off past its certificate date always comes back broken, as worker01 did, so check its PEM with openssl before you boot it back into service.
We keep the cron check short enough to show in full. It prints a line only when the API server certificate has less than 30 days left:
sudo openssl x509 -checkend 2592000 -noout -in /etc/kubernetes/pki/apiserver.crt || echo "apiserver.crt expires within 30 days on $(hostname)"Against the freshly renewed certificate it printed Certificate will not expire and exited 0, so the echo stayed silent. New clusters built with our kubeadm install guide for Ubuntu start that one-year clock on install day, so it’s worth putting this check in place the same afternoon. Renewing kubeadm certificates is also a common CKA practice task, and several of the Kubernetes certification books on our list walk through the PKI layout in more depth than the docs do.
Where this recovery breaks down
Everything above assumes the cluster CAs are still valid and their keys are on a control-plane node. When either assumption fails, kubeadm certs renew all has nothing to sign with and the recovery turns into a different job:
- Expired CA: the CAs last ten years, and renewing leaf certificates against an expired CA produces certificates nobody trusts. Rotating the CA itself is a manual, multi-step process covered in the upstream manual-rotation docs.
- External CA: if
check-expirationmarks a certificate EXTERNALLY MANAGED, kubeadm doesn’t haveca.key. Generate CSRs withkubeadm certs generate-csrand have whoever owns the CA sign them. - Kubelet serving certificate: the certificate the kubelet presents on port 10250 is self-signed by default, because
serverTLSBootstrapis off. With it turned on, serving CSRs are not auto-approved, so each one needskubectl certificate approve.
For the ordinary case, a year-old cluster that nobody upgraded, the order is what matters: check the clock, renew, restart the static pods, copy admin.conf, then rebuild each NotReady worker’s kubelet.conf and point it back at the rotating file. The upstream kubeadm certificate management page remains the reference for flags that change between releases.
