Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
This article shows you how to deploy Cluster Health Monitor in Azure Kubernetes Service (AKS).
Cluster Health Monitor helps you detect AKS managed component issues earlier and improve resilience by enabling automatic remediation for unhealthy CoreDNS pods in specific scenarios. It runs periodic checks for components such as CoreDNS, metrics server, and API server, and exposes results as Prometheus metrics for alerting.
Important
Cluster Health Monitor is an AKS-managed checker for system components. It doesn't replace Azure Monitor, Managed Prometheus, or your existing monitoring stack.
Important
AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:
Overview
Cluster Health Monitor is an AKS-managed add-on deployed in the kube-system namespace. It includes two monitors:
Control plane and add-on monitor runs ongoing in-cluster checks to validate control plane and add-on components that affect cluster operations and upgrade reliability. It evaluates the following signals:
- DNS resolution success rate to detect unhealthy DNS pods that can affect service discovery.
- API server connectivity to detect failures reaching the control plane from within the cluster.
- Metrics server availability to detect failures in metrics collection.
On-Demand Monitor runs node health checks automatically when a node is provisioned or rebooted. You can also trigger a check manually on any existing node. It evaluates:
- Pod scheduling to confirm a standard pod can be scheduled on the node.
- Pod networking to confirm pods can reach other pods.
These monitors are automatically enabled on AKS Automatic clusters running version 1.34 or later. Cluster Health Monitor exposes check results as Prometheus metrics on port 9800, so you can scrape and alert on these signals in your existing monitoring pipeline.
Note
Cluster Health Monitor is automatically enabled on AKS Automatic clusters running Kubernetes 1.34+ and can't be enabled on earlier Kubernetes versions in AKS Automatic.
Before you begin
Install the Azure CLI version 2.73.0 or later. Run
az --versionto check your version. To install or upgrade, see Install Azure CLI.Install the
aks-previewAzure CLI extension version 21.0.0b8 or later:az extension add --name aks-previewIf you already installed the extension, update it to the latest version.
az extension update --name aks-previewUse the AKS preview managed clusters API version 2026.01.02-preview for this feature.
Enable Cluster Health Monitor
Enable control plane and add-on monitor
This feature enables in-cluster monitoring of control plane and add-on components. It also remediates unhealthy CoreDNS pods if they stay unhealthy for five minutes. For more information, see How CoreDNS remediation works.
To enable control plane and add-on monitor on a new cluster:
az aks create -g myResourceGroup -n myCluster --enable-continuous-control-plane-and-addon-monitor
To enable control plane and add-on monitor on an existing cluster:
az aks update --resource-group myResourceGroup --name myCluster --enable-continuous-control-plane-and-addon-monitor
Verify control plane and add-on monitor
After you enable the feature, you can verify the deployment status and metric endpoint exposure.
Verify that the Cluster Health Monitor Deployment is running in kube-system.
kubectl get deployment -n kube-system cluster-health-monitor
Disable control plane and add-on monitor
az aks update -g myResourceGroup -n myCluster --disable-continuous-control-plane-and-addon-monitor
Understand metrics exposed by the control plane and add-on monitor
Each check returns one of the following statuses:
| Status | Meaning |
|---|---|
Healthy |
The component is operating normally. |
Unhealthy |
The check failed. The error code in the metric indicates the failure reason. |
Unknown |
The result is inconclusive, for example during pod startup or when a dependency is unavailable. |
Error code reference
Cluster Health Monitor exposes metrics on port 9800 on the cluster-health-monitor deployment in the kube-system namespace. You can scrape these metrics with Prometheus and use them to detect control plane and add-on health issues. When a check reports Unhealthy, Cluster Health Monitor includes an error code in the metric labels.
| Checker | Error code | What it indicates |
|---|---|---|
| APIServer | APIServerCreateError, APIServerGetError, APIServerDeleteError |
The API server check failed during create, get, or delete operations for the synthetic test resource. |
| APIServer | APIServerCreateTimeout, APIServerGetTimeout, APIServerDeleteTimeout |
The API server check exceeded timeout thresholds during create, get, or delete operations. |
| CoreDNS / LocalDNS | ServiceNotReady, PodsNotReady |
DNS infrastructure wasn't ready when the check ran. |
| CoreDNS / LocalDNS | ServiceError, PodError, LocalDNSError |
DNS queries failed with non-timeout errors. |
| CoreDNS / LocalDNS | ServiceTimeout, PodTimeout, LocalDNSTimeout |
DNS queries timed out. |
| MetricsServer | MetricsServerUnavailable |
The metrics-server API couldn't be reached or returned an error. |
| MetricsServer | MetricsServerTimeout |
The metrics-server API request timed out. |
How CoreDNS remediation works
The control plane and add-on monitor in Cluster Health Monitor include a CoreDNS remediation capability that restores DNS health while minimizing risk to DNS availability. Before taking action, it evaluates per-pod DNS health check results, how long each pod is unhealthy, the health state of other CoreDNS pods, and recent remediation history.
It deletes an unhealthy CoreDNS pod only when all of the following conditions are true:
- Exactly one CoreDNS pod is continuously unhealthy for at least five minutes.
- At least one other CoreDNS pod is healthy.
- No CoreDNS remediation event fired in the last one hour.
When all conditions are met, it deletes the unhealthy pod. Kubernetes then recreates it. This approach restores DNS capacity while keeping at least one healthy replica serving traffic and avoids repeated remediation cycles during ongoing incidents.
Enable On-Demand Monitor
To enable On-Demand Monitor on a new cluster:
az aks create --resource-group myResourceGroup --name myCluster --enable-on-demand-monitor
To enable On-Demand Monitor on an existing cluster:
az aks update --resource-group myResourceGroup --name myCluster --enable-on-demand-monitor
Verify On-Demand Monitor
To verify that On-Demand Monitor is enabled, run:
kubectl get crd checknodehealths.clusterhealthmonitor.azure.com -o yaml
Disable On-Demand Monitor
az aks update --resource-group myResourceGroup --name myCluster --disable-on-demand-monitor
Check node health status with On-Demand Monitor
AKS automatically triggers a node health check when a node is provisioned or rebooted. Like the control plane and add-on monitor, On-Demand Monitor reports node health as Healthy, Unhealthy, or Unknown.
Manually trigger a node health check
To manually trigger a health check on an existing node, deploy a CheckNodeHealth custom resource to your cluster. See the CheckNodeHealth CRD definition on GitHub for the resource schema. The custom resource is automatically cleaned up after six hours. Refer to the following example to trigger a health check on a specific node:
kubectl apply -f - <<'EOF'
apiVersion: clusterhealthmonitor.azure.com/v1alpha1
kind: CheckNodeHealth
metadata:
name: my-node-check
spec:
nodeRef:
name: aks-nodepool1-12345678-vmss000000
EOF
To check the health check status, run:
kubectl get CheckNodeHealth my-node-check -o yaml
kubectl describe node <node-name>
In the output, look for the kubernetes.azure.com/NodeHealthy condition and check its Status and Message fields.
Note
Nodes running CoreDNS pods might show a status Unknown for this check. The health check validates cross-node connectivity by using a CoreDNS ping and therefore can't run on the node hosting the CoreDNS pod.
How node remediation works
When On-Demand Monitor detects an unhealthy node, AKS automatically remediates it. The remediation behavior depends on how the node was provisioned:
- Node auto-provisioning nodes are remediated automatically based on node auto-provisioning's remediation logic.
- Other nodes — AKS progressively increases the scope of remediation: restarting the node after five minutes, reimaging it after 10 minutes, and redeploying the VM after 15 minutes. Each action is attempted once per node provisioning event.