Highlights

• Kubernetes 1.36 is the default minor version
• Ansible environment moves to Ansible 12 / ansible-core 2.19, with breaking changes to conditionals and templating. See Urgent Upgrade Notes before upgrading.
• New operating system support includes Ubuntu 26.04, openSUSE Leap 16.0, and openSUSE MicroOS with containerd.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Ansible environment (action required): Kubespray now requires ansible-core >=2.19.0,<2.20.0 and ships with Ansible 12. Before running playbooks, recreate or update the virtual environment on the Ansible control host using this release's requirements.txt. Ansible 2.19 requires conditional expressions to return booleans; -e key=value supplies strings. Pass boolean overrides as JSON (for example, -e '{"drain_nodes": true, "drain_fallback_enabled": true}') or use unquoted booleans in YAML inventory or extra-vars files (-e @vars.yml). Review custom Jinja templates in inventories and integrated playbooks for compatibility with the templating changes described in the Ansible 12 porting guide. The documentation examples have been updated accordingly. (#13468, @VannTen; #13526, @shashankvarma499)
• Node Feature Discovery (action required): The default version is now 0.19.0. Remove node_feature_discovery_enable_nodefeature_api, node_feature_discovery_master_crd_controller, and node_feature_discovery_master_instance from inventory overrides. gRPC mode and executable hooks under source.d are no longer supported. Custom NodeFeatureRule templates must no longer use env, expandenv, or getHostByName; review and migrate affected configurations before upgrading. (#13512, @vashian)
• Pinned Helm versions (action required): Helm now defaults to 3.22.0. Checksums for Helm 3.16.x, 3.17.x, and 3.18.x have been removed. If inventory overrides pin helm_version to one of these versions, remove the override to use the default or select an available version from the retained 3.20.x, 3.21.x, or 3.22.x series. (#13535, @guoard)
• Cilium Gateway API: Gateway API CRDs now default to 1.6.2, meeting Cilium 1.20's minimum requirement of Gateway API 1.6.1. If you enable Cilium's Gateway API support and pin gateway_api_version below 1.6.1, update or remove that override before upgrading. Gateway API 1.6.0 and 1.6.1 checksums are also available, but 1.6.0 does not meet Cilium 1.20's Gateway API requirement. (#13534, @yankay)
• etcd metrics discovery: The etcd-metrics Service now uses EndpointSlice instead of Endpoints to avoid duplicate metrics. If you use this Service, update scrapers that still discover Endpoints. With Prometheus Operator, set spec.serviceDiscoveryRole: EndpointSlice on the relevant ServiceMonitor or Prometheus resource, and grant Prometheus permission to get, list, and watch endpointslices in the discovery.k8s.io API group. Verify etcd scrape targets after upgrading. (#12846, @jmeza-xyz)
• Debian 11 and macvlan: Debian 11 (Bullseye) is no longer supported; migrate to a supported operating system before upgrading. macvlan support on Debian has also been removed because it was not functional on Debian 12 or 13. macvlan is no longer tested in CI. (#13483, @VannTen)
• Debian 13 with Docker: This combination is no longer supported or tested because Debian 13 removed apt-key, which the Docker repository setup still requires. Use containerd on Debian 13, or remain on an operating system supported by the Docker role. (#13499, @VannTen)
• Local volume provisioner: Entries in local_volume_provisioner_storage_classes now use the upstream storageClassMap camelCase keys. Update inventory overrides: host_dir -> hostDir, mount_dir -> mountDir, volume_mode -> volumeMode, fs_type -> fsType, and block_cleaner_command -> blockCleanerCommand. (#13504, @VannTen)
• Cilium image overrides: Rename cilium_hubble_certgen_image_{repo,tag} to cilium_certgen_image_{repo,tag} and cilium_hubble_envoy_image_{repo,tag} to cilium_envoy_image_{repo,tag} in inventory overrides. (#13260, @guoard)
• Control-plane eviction thresholds: eviction_hard_control_plane has been removed. Configure eviction_hard under group_vars/kube_control_plane/ instead. (#13400, @tophac)

Changes by Kind

Feature

• Add additional_sysctl_min to raise sysctl values only when they are below a configured minimum. Expand the default minimums for common networking and container-density settings. additional_sysctl retains its exact-value behavior. (#13224, @yankay)
• Add kubeadm_backup_retention_count to clean up old kubeadm backups and temporary upgrade files under /etc/kubernetes/tmp/. The default, -1, disables cleanup; 0 removes all matching files and directories, and positive values retain that many recent entries of each type. (#13415, @wangsifei99)
• Add openeuler_repo_baseurl to select an alternative openEuler package mirror. (#13134, @yankay)
• Add an opt-in admission readiness check before kubeadm upgrades with kubeadm_upgrade_preflight_check_enabled (default: false). Configure retries and delay with kubeadm_upgrade_preflight_check_retries and kubeadm_upgrade_preflight_check_delay. A failed check warns but does not block the upgrade. (#13140, @felipe88alves)
• Add authentication for custom CNI Helm repositories with custom_cni_chart_repository_username and custom_cni_chart_repository_password. (#13294, @A-Soltani255)
• Add openSUSE Leap 16.0 support and representative KubeVirt CI coverage. (#13473, @NikKumar811)
• Add openSUSE MicroOS support for containerd-based clusters, including transactional package installation and reboot handling. Docker and CRI-O are not supported on MicroOS. (#13449, @yankay)
• Add optional apt package cache cleanup during system upgrades with system_upgrade_autoclean: true on Debian-based systems. (#13327, @Jeroen0494)
• Add optional Prometheus metrics for the kube-vip static pod with kube_vip_metrics_enabled and kube_vip_metrics_port. (#13229, @isumitsolanki)
• Add Terraform provisioning for OpenNebula 7.2+ in contrib/terraform/opennebula, including automatic Ansible inventory generation. (#13390, @razvan01)
• Add Ubuntu 26.04 support. (#13443, @yankay)
• Enable Flannel's nftables backend for Flannel 0.25.5 or newer when kube_proxy_mode: nftables. The nft utility must be installed on the nodes. (#13291, @rtsui-harmonicinc)
• Make controller-manager node CIDR allocation configurable with kube_controller_manager_allocate_node_cidrs. (#13239, @modzilla99)
• Make Kubernetes 1.36 the default minor version. See Component Versions below for the selected patch version. (#13219, @tmurakam)
• Update Cilium to 1.20.2. (#13515, @guoard)
• Update DNS autoscaler (cluster-proportional-autoscaler) to 1.10.3. (#13529, @vashian)
• Update nerdctl to 2.3.5. (#13437, @sandaruwijewardhana)
• Update the default Flannel version to 0.28.9. (#13436, @dhakshin-raghav)
• Use CRI-O 1.36 by default with Kubernetes 1.36. See Component Versions below for the selected patch version. (#13244, @tmurakam)

Bug or Regression

• Add missing RBAC permissions for calico-kube-controllers when using the Kubernetes datastore. (#13375, @taiko19xx)
• Advertise IPv4 as well as IPv6 service cluster IPs through Calico BGP when calico_advertise_cluster_ips: true. (#13503, @VannTen)
• Allow facts to be gathered and cached for reachable hosts when another inventory host is unreachable. (#13438, @suii2210)
• Apply cilium_agent_scrape_port, cilium_operator_scrape_port, and cilium_hubble_scrape_port to the Cilium Helm values so these settings take effect. (#13370, @somaz94)
• Create /opt/cni/bin with root ownership to fix Cilium 1.19.x CNI installation failures. Reject Cilium configurations with a non-root `kube_o...

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  Make kubernetes v1.35 default. cgroup V1 is no longer supported upstream by default, and must be enabled with kubelet_fail_cgroup_v1: false. (#12812, @mzaian)
• Action required
  Support for ingress-nginx ingress controller is removed, as the project has been retired by upstream (#12767, @jmeza-xyz)
• Action required
  The Kubernetes Dashboard addon has been removed from Kubespray because the upstream project is being archived and is no longer maintained. The configuration dashboard_enabled: true is no longer supported. (#12858, @neo502721)
• Adds automated validation role to detect removed variables at playbook start. Playbooks will abort on removed vars. Users must migrate away from the listed removed variables to avoid playbook failures. (#12942, @Srishti-j18)
• Replace ssh_bastion_confing__name with ssh_bastion_config_name (#13046, @scolley31)

Changes by Kind

Feature

• A new configuration option kubelet_static_pod_path has been added which can be used to configure path of static pod manifests OR even to disable staticPodPath setting in kubelet by setting it as empty (STIG recommendation for worker nodes) (#12433, @shaleenbathla)
• Add Structured AuthenticationConfiguration config file support for kube-apiserver (#13035, @chadswen)
• Add support for kube-vip BGP source selection in Kubespray by exposing kube_vip_bgp_sourceip and kube_vip_bgp_sourceif (mutually exclusive). (#13044, @Kirillov-AN)
• Bump CNI plugins to v1.9.1. (#13198, @karimzakzouk)
• Bump Flannel to v0.28.4. (#13199, @karimzakzouk)
• Bump Prometheus Operator CRD to 0.88.1 (#12968, @hcc429)
• Bump containerd default version to 2.2.3. (#13174, @yankay)
• Cilium template apiVersion upgrade to v2 (#13095, @guoard)
• Coredns service name can now be customized with coredns_svc_name (#12951, @VannTen)
• Enable create_namespace option for custom_cni with helm (#13061, @guoard)
• Increase fs.inotify.max_user_instances to 8192 by default (#13075, @rptaylor)
• Increased StartLimitBurst to 10 for docker service to prevent cri-dockerd restart throttling (#13005, @Tushar240503)
• Make kubernetes v1.35.4 default. (#13193, @mzaian)
• Support for Fedora 41 (#12138, @tico88612)
• Support for Fedora 42 (#12989, @labaq)
• Update etcd to 3.6.8
  Upgrade note: You need to update etcd to >=3.5.26 (Kubespray 2.30.0) before upgrading existing cluster (#12634, @tmurakam)
• Updated the default Calico version to v3.31.5. (#13196, @Srishti-j18)
• Upgrade Dockerfile base image from Ubuntu 22.04 to 24.04 (#12935, @guoard)
• Upgrade OpenStack Cloud Controller Manager to v1.35.0 (#12972, @shuan1026)
• Upgrade ansible from 10.7.0 to 11.13.0 (#12903, @guoard)
• Upgrade metrics-server to v0.8.1 (#13189, @mzaian)
• Use kubeadm:cluster-admins group for admin kubeconfig generation. (#12998, @Srishti-j18)
• [etcd] Default version to 3.6.10 for Kubernetes 1.35. (#13190, @mzaian)
• [runc] Default version to 1.4.2. (#13192, @mzaian)

Design

• Container engine are now included rather than imported, to avoid lots of always skipped tasks during playbook execution (#12946, @Tushar240503)
• Network plugins are now included rather than imported, to avoid lots of always skipped tasks during playbook execution (#12933, @Tushar240503)

Documentation

• Offline files repo is expected to use original domains as top-level directories. (#12684, @bbaassssiiee)
• Remove RHEL8 documentation (#13017, @VannTen)

Bug or Regression

• Add service RBAC for Calico Kubernetes datastore so Calico can monitor LoadBalancer IP allocation (#12928, @mickenordin)
• Calico KDD CRDs are now downloaded from https://github.com/projectcalico/calico/blob/master/manifests/crds.yaml instead of the Calico tarball. (#12985, @Srishti-j18)
• Enable metalink for openEuler repos to auto-select fastest mirror, fixing slow package installs outside China (#13094, @yankay)
• Fix OCI CCM deployment failure caused by incorrect template filename in lookup (#13151, @amoghazy)
• Fix broken NO_PROXY variable (#12981, @VannTen)
• Fix calico missing RBAC permissions for kube-controller-manager to access tiers in manifest installs, which was preventing proper resource garbage collection. (#13100, @guoard)
• Fix calico missing staged policy permissions for api server (#13101, @guoard)
• Fix cilium_enable_prometheus variable having no effect by wiring it to the Helm values template. (#13142, @yankay)
• Fix crash when CiliumBGPAdvertisement is defined without a labels key. (#13149, @karimzakzouk)
• Fix drain tasks failing with UNREACHABLE when drain_timeout exceeds the Ansible SSH connection timeout by using async/poll. (#13081, @0xMH)
• Fix terraform openstack compute image_id and update openstack_blockstorage_volume_v3 (#12910, @HauptJ)
• Fixed Gateway API v1.4.1 unexpected checksum change and add test (#13006, @labaq)
• Fixed openeuler metalink 24.03LTS wrong url (#13144, @tico88612)
• Improved validation for kube_version in validate_inventory role. Users will now receive a clear error message if a specified version is missing from the checksums dictionary, preventing cryptic "AnsibleUndefined" errors during the download phase. (#13071, @jannickk)
• Introduce a timeout of 5 minutes on package installation, customizable using pkg_install_timeout (#12878, @VannTen)
• Kubeadm patches not present in the kubeadm_patches variable are now removed from the managed nodes automatically (#13019, @VannTen)
• Make etcd node removal idempotent (#12949, @VannTen)
• Skip package management when there is no packages to install or remove (#13015, @VannTen)
• Update volumesnapshotclass to v1 (#12775, @viktor-f)

Other (Cleanup or Flake)

• Kubeadm config templates are now v1beta4 only; v1beta3 support has been removed. (#13027, @Srishti-j18)
• Local kubeconfig is now copied from /etc/kubernetes/admin.conf when available. (#12997, @Srishti-j18)
• Nifcloud terraform provider support is removed, as the provider is no longer available. (#12936, @VannTen)
• Remove unused cilium_enable_bpf_clock_probe variable. We can use cilium_extra_values to set it if required (#13050, @shaleenbathla)
• Removed netchecker support (#13058, @VannTen)
• The internal kube-config used by control plane components on control plane nodes now points to the local apiserver (default kubeadm behavior)
  This fixes the incorrect version skew between control plane components and apiserver during upgrade (#12870, @VannTen)

Component versions

• kubernetes 1.35.4
• etcd 3.6.10
• docker 28.3
• containerd 2.2.3
• cri-o 1.35.0
• runc 1.4.2
• cni-plugins 1.9.1
• calico 3.31.5
• cilium 1.19.3
• flannel 0.28.4
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.2.2
• kube-vip 1.0.3
• cert-manager 1.15.3
• coredns (derived from kube_major_version)
• argocd 2.14.21
• helm 3.18.4
• metallb 0.13.9
• registry 2.8.1
• nerdctl 2.2.2
• metrics-server 0.8.1
• kata-containers 3.7.0
• aws-ebs-csi-plugin 0.5.0
• azure-csi-plugin 1.10.0
• cinder-csi-plugin 1.30.0
• gcp-pd-csi-plugin 1.9.2
• local-path-provisioner 0.0.32
• local-volume-provisioner 2.5.0
• node-feature-discovery 0.16.4

New Contributors

Welcome and thank you to all the first-time contributors to Kubespray!

@0xMH @amoghazy @botszhuang @celestina-amadi-moniepoint @HauptJ @hcc429 @jannickk @karimzakzouk @Kirillov-AN @labaq @neo502721 @PangQingcheng @Rayui1225 @scolley31 @shuan1026 @sirzzang @Srishti-j18 @thc1006 @tim80411 @Tushar240503 @uchiha-vivek @voidquark @yangminglintw @Zeratyl06

All Contributors

Thank you to everyone who contributed to this release!

@0xMH @amoghazy @bbaassssiiee @botszhuang @celestina-amadi-moniepoint @chadswen @cyclinder @ErikJiang @eshutov @guoard @HauptJ @hcc429 @jannickk @jmeza-xyz @karimzakzouk @Kirillov-AN @labaq @liggitt @mickenordin @mzaian @neo502721 @PangQingcheng @Rayui1225 @rptaylor @scolley31 @shaleenbathla @shuan1026 @sirzzang @Srishti-j18 @thc1006 @tico88612 @tim80411 @tmurakam @Tushar240503 @uchiha-vivek @VannTen @viktor-f @voidquark @yangminglintw @yankay @Zeratyl06

Changes by Kind

Feature

• Default to Kubernetes v1.30.6 (#11673, @robertvolkmann)
• Bump ingress-nginx to 1.11.5 and certgen to 1.5.2 (#12081, @tico88612)
• Default to containerd 1.7.23 (#11643, @robertvolkmann)
• Update runc to v1.1.14 and nerdctl to 1.7.7 (#12817, @rptaylor)

Bug or Regression

• Always copy cert generation scripts to first etcd to pick up fixes on existing clusters (#11615, @k8s-infra-cherrypick-robot)
• Fix: cannot stop & remove all cri containers via remove_node.yml (#11637, @k8s-infra-cherrypick-robot)
• Fix kubecontrollersconfigurations list permission (#12038, @k8s-infra-cherrypick-robot)
• Workaround missing etcd certs on control plane node (#12193, @k8s-infra-cherrypick-robot)
• Move 'pretend certificates' after cert distribution (#12222, @VannTen)
• Fix manage-offline-container-images.sh get image_id (#12315, @k8s-infra-cherrypick-robot)
• Fix PodSecurity Admission "empty" definition handling (#12477, @k8s-infra-cherrypick-robot)

Components

• kubernetes v1.30.6
• etcd v3.5.16
• docker v26.1
• containerd 1.7.23
• cri-o v1.30.3
• cni-plugins v1.2.0
• calico v3.28.1
• cilium v1.15.4
• flannel v0.22.0
• kube-ovn v1.12.21
• kube-router v2.0.0
• multus v3.8
• weave v2.8.7
• kube-vip v0.8.0
• cert-manager v1.14.7
• coredns v1.11.1
• ingress-nginx v1.11.5
• krew v0.4.4
• argocd v2.11.0
• helm v3.15.4
• metallb v0.13.9
• registry v2.8.1
• aws-ebs-csi-plugin v0.5.0
• azure-csi-plugin v1.10.0
• cinder-csi-plugin v1.30.0
• gcp-pd-csi-plugin v1.9.2
• local-path-provisioner v0.0.24
• local-volume-provisioner v2.5.0
• node-feature-discovery v0.16.4

Announcement

Ingress NGINX is sunset. This is the final version supporting Ingress NGINX. We'll remove ingress-nginx in future versions.
For more information, please see the Kubernetes blog.

Kubernetes Dashboard is archived. This is the final version supporting Kubernetes Dashboard. We'll remove Kubernetes Dashboard in future versions. For more information, please see the announcement.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  containerd_discard_unpacked_layers is now applied only for containerd < 2.1 to avoid warnings with the Transfer Service used in newer versions. (#12821, @guoard)
• Cilium k8sServiceHost and k8sServicePort are now derived from kube_apiserver_global_endpoint instead of being auto-detected, so users must ensure this endpoint is correctly set and reachable from all nodes. (#12624, @r3m8)

Changes by Kind

API Change

• Removed deprecated runtime_engine and runtime_root keys from containerd configuration. (#12820, @guoard)

Feature

• Crio: add option pull_progress_timeout (#12555, @pedro-peter)
• Fixed kube_vip_version variable for managing kube-vip version instead of kube_vip_image_tag (#12835, @ThisIsQasim)
• RockyLinux 10 support (experimental) (#12355, @tico88612)
• Support kubernetes v1.34.1 (#12549, @i-yasuda)
• The default Openstack security groups now allow ICMPv6 for general IPv6 functionality. (#12805, @rptaylor)
• Update kube-vip to v1.0.3 (#12815, @LawiK974)
• Upgrade Gateway API to 1.4.0 (#12714, @tico88612)
• Upgrade containerd/nerdctl default version to 2.2.1 (#12825, @guoard)
• contrib/collection.sh will create and install the source as collection. (#12660, @bbaassssiiee)

Documentation

• Fix(cilium): prevent installation failure with unprivileged agent (#12628, @r3m8)
• The default namespace for MetalLB resources is now metallb-system. (#12860, @kittywaresz)

Bug or Regression

• Action-required
  Adding a control plane node in first place is now explicitly unsupported, the new control plane should be placed at the end of your kube_control_plane group (#12636, @clwluvw)
• Add a wait after control plane joining to prevent a kubeadm preflight error when upgrading the control plane configuration at the same time. (#12794, @VannTen)
• Deployments using Calico in vxlan mode without eBPF but with localhost load balancers will now work. (#12598, @rickerc)
• Fix Calico apiserver RBAC permissions for Kubernetes 1.33+ (#12654, @rickerc)
• Fix Cilium loadBalancer.mode rendering in Kubespray values template. (#12701, @intojhanurag)
• Fix RBAC for calico using the etcd datastore (#12828, @LawiK974)
• Fix automatic certs renew with systemd timer (#12876, @VannTen)
• Fix broken upgrade path/control plane node rotation for cluster using calico in etcd datastore mode with separate etcd.
  etcd_cert_dir_mode is deleted (always use 0700) (#12908, @VannTen)
• Fix kubeadm init retry after first failure on cluster creation (#12785, @VannTen)
• Fix(calico): Add missed rbac verb watch for hostendpoints (#12641, @jmeza-xyz)
• Fixed an issue in the config.json.j2 template where the CRI-O registry authentication configuration could render invalid JSON when multiple crio_registry_auth entries were defined, resulting in duplicate top-level auths keys in the generated config. (#12845, @accuROAMC)
• Removing external etcd member (not stacked with control plane) should now work without erroring out because the node is not in the kubernetes cluster (#12682, @VannTen)
• Update CSI components image version (#12627, @xin053)
• [feat] Setting timezone under SELinux. (#12436, @bbaassssiiee)
• apiserver_loadbalancer_domain_name default to loadbalancer_apiserver.address if defined (#12872, @VannTen)

Other (Cleanup or Flake)

• Old versions of several components are removed (#12735, @VannTen)
• Remove left-over 'master' tags (#12795, @VannTen)
• Use cilium native CNI chaining for portmap plugin instead of manually writing /etc/cni/net.d/000-cilium-portmap.conflist (#12814, @ThisIsQasim)

Components

• kubernetes 1.34.3
• etcd 3.5.26
• docker 28.3
• containerd 2.2.1
• cri-o 1.34.4
• cni-plugins 1.8.0
• calico 3.30.6
• cilium 1.18.6
• flannel 0.27.3
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.2.2
• kube-vip 1.0.3
• cert-manager 1.15.3
• coredns 1.12.1
• ingress-nginx 1.13.3
• argocd 2.14.5
• helm 3.18.4
• metallb 0.13.9
• registry 2.8.1
• aws-ebs-csi-plugin 0.5.0
• azure-csi-plugin 1.10.0
• cinder-csi-plugin 1.30.0
• gcp-pd-csi-plugin 1.9.2
• local-path-provisioner 0.0.32
• local-volume-provisioner 2.5.0
• node-feature-discovery 0.16.4

Changes by Kind

Bug or Regression

• Fix Calico apiserver RBAC permissions for Kubernetes 1.33+ (#12695, @k8s-infra-cherrypick-robot)
• Fix Cilium loadBalancer.mode rendering in Kubespray values template. (#12705, @k8s-infra-cherrypick-robot)
• Fix(calico): Add missed rbac verb watch for hostendpoints (#12644, @k8s-infra-cherrypick-robot)
• Removing external etcd member (not stacked with control plane) should now work without erroring out because the node is not in the kubernetes cluster (#12685, @k8s-infra-cherrypick-robot)

Components

• kubernetes 1.33.7
• etcd 3.5.25
• docker 28.3
• containerd 2.1.5
• cri-o 1.33.7
• cni-plugins 1.8.0
• calico 3.30.5
• cilium 1.18.4
• flannel 0.27.3
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.2.2
• kube-vip 0.8.0
• cert-manager 1.15.3
• coredns 1.12.0
• ingress-nginx 1.13.3
• argocd 2.14.5
• helm 3.18.4
• metallb 0.13.9
• registry 2.8.1
• aws-ebs-csi-plugin 0.5.0
• azure-csi-plugin 1.10.0
• cinder-csi-plugin 1.30.0
• gcp-pd-csi-plugin 1.9.2
• local-path-provisioner 0.0.32
• local-volume-provisioner 2.5.0
• node-feature-discovery 0.16.4

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  /etc/hosts/ is no longer populated with all cluster nodes (#12382, @VannTen)
• Action required
  Add support for coredns_affinity to change affinity of coredns deployments, defaulting to the upstream coredns deployment's one.
  The coredns deployment's node affinity has been removed, so the coredns pods will no longer be scheduled into control-planes by default. (#11994, @HoKim98)
• Action required
  Remove support for weave network plugin (#12230, @anshuman-agarwala)
• Action required
  The tag 'master' is removed, replaced by the tag 'control-plane' (#12228, @VannTen)
• Action required
  conntrack_modules is removed; the list of conntrack modules to try to load is instead hardcoded, since there is no reason to have any other values. (#12475, @VannTen)
• Action required
  drop support for cri-o on ubuntu20. (#12233, @VannTen)

Changes by Kind

Feature

• A new sysctl_ignoreerrors value has been added, it allows to ignore errors about unknown keys that may be raised by sysctl (#12514, @bidorffOL)
• A new configuration option kubelet_static_pod_path has been added which can be used to configure path of static pod manifests OR even to disable staticPodPath setting in kubelet by setting it as empty (STIG recommendation for worker nodes) (#12433, @shaleenbathla)
• Add cilium_install_extra_flags variable (#12262, @tmurakam)
• Add external_openstack_lbaas_member_subnet_id: str (not set by default), to define a specific member-subnet-id for the openstack load balancers (#12267, @voondo)
• Add support for containerd_extra_runtime_args variable to allow injection of additional runtime configuration options into containerd CRI plugin section. (#12247, @Ujstor)
• Add support for kubeadm_image_repo variable to change kubernetes core image repository (e.g. kube-apiserver, kube-proxy). (#12128, @HoKim98)
• Add the possibility to use any values from Cilium Helm Chart (#12375, @cleman95)
• Added Prometheus Operator CRDs installation (#12441, @tico88612)
• Adds support for installing containerd as a static binary.
  Bump containerd to 2.1.3, runc to 1.3.0, nerdctl to 2.1.2 (#12377, @yankay)
• Bump ansible to 10.7.0 (#11924, @tico88612)
• Calico supports nftable mode (#12255, @tico88612)
• Control plane health check retries for apiserver, scheduler, and controller-manager are now configurable via control_plane_health_retries (default: 60). (#12452, @aman4433)
• Feat: Support certificate validity period config in kubeadm v1beta4 (#12272, @ErikJiang)
• Feat: add support for crio additional mounts (#12561, @mahendra77024)
• Introduce crio_runtime_switch boolean to allow users to switch the crio runtime by removing pods and stopping crio and kubelet during upgrade ; otherwise crio has problems when trying to work with pods created with the old runtime. (#12008, @mahendra77024)
• Introduced coredns_replicas to alter coredns deployment replicas when enable_dns_autoscaler is set to false. (#12387, @clwluvw)
• Redeploy coredns and nodelocaldns when their configurations change. (#12401, @atobaum)
• Remove --auth-anonymous if kube_api_anonymous_auth is undefined. (#12353, @psychomantys)
• Support Debian 13 Trixie (#12456, @tico88612)
• Support for custom header configuration in containerd registry mirrors via inventory and role variables. Users can now specify headers (e.g., Authorization) for registry mirrors in hosts.toml. (#12368, @pando85)
• Support kubernetes v1.33.1 (#12199, @tmurakam)
• Update cni-plugin to 1.8.0 (#12551, @tmurakam)
• Update load balancers versions to Nginx 1.28.0, Haproxy 3.1.7 (#12178, @guoard)
• Upgrade external snapshot CRD to v0.15.0 (#12308, @tico88612)
• Upgrade multus cni from 4.1.0 to 4.2.2 (#12495, @ThisIsQasim)
• [calico] Update default calico to v3.30.3 (#12523, @tmurakam)
• [flannel] upgrade to 0.26.7 (#12260, @tico88612)
• [ingress-nginx] upgrade controller to version 1.13.3 (#12604, @mzaian)

Design

• Show node to be upgraded/uncordoned in upgrade/uncordon confirmation prompt when using upgrade_node_confirm or upgrade_node_post_upgrade_confirm (#12399, @MatthiasLohr)

Bug or Regression

• Add argocd_install_checksum: str, to define the checksum of argocd_install_url (#12266, @voondo)
• Add missing addresses in kube-apiserver certificate SAN. (#12413, @hhk7734)
• Bugfix: skip etcd cert extraction if cilium identity uses crd (#12565, @mahendra77024)
• Fix Cilium installation issues (caused by templating syntax errors) when certain non-default features (encryption, etc.) are enabled (#12280, @spantaleev)
• Fix Hubble-Relay peer discovery in clusters using non-default cluster name by properly configuring clusterDomain in Cilium Helm values (#12346, @mertcancam)
• Fix cilium installation role to render cilium_config_extra_vars into helm values (#12335, @atobaum)
• Fix cilium_policy_audit_mode variable (#12569, @guoard)
• Fix error when using kubeadm_ignore_preflight_errors: ['all'] (#12606, @VannTen)
• Fix ingress-nginx DaemonSet and Service templates rendering TCP/UDP ports as strings, which prevented correct export of TCP/UDP services via NGINX ingress controller. (#12442, @MahdadGhasemian)
• Fix invalid PodSecurity admission configuration when kube_pod_security_use_default: false (#12439, @AMacedoP)
• Fix scale.yml problems with cached IP facts (#12243, @fox0430)
• Fix the Cilium cluster, which is upgraded from 2.27 to 2.28 will break
  Fix helm release re-use message when installing repeatedly (#12254, @tico88612)
• Fix the issue of etcd node addition failure caused by incorrect ETCD_INITIAL_CLUSTER configuration. (#12342, @liuxu623)
• Fix(kubeadm): Conditionally add --skip-phases flag for v1.32.0+ (#12351, @ErikJiang)
• Fix: A timeout occurs when running the offline deployment script using Podman. (#11962, @DearJey)
• Fix: When running ./manage-offline-container-images.sh register with using Podman, getting the image_id fails and the script is interrupted. (#11961, @DearJey)
• Fix: kubeadm secondary nodes use file discovery validation failed (#12132, @tico88612)
• Fixed a looping timeout bug when deleting an entire cluster (#12300, @chadswen)
• Fixed cilium_enable_bgp_control_plane config (#12430, @XuhuiSun95)
• Fixed packages installation on Alma/Rocky Linux when behind a proxy (#12264, @root-expert)
• Fixes a syntax error that made the '_bgp_config' an 'AnsibleUnsafeText' instead of a 'dict', which caused the "Calico | Process BGP Configuration" step to fail (#12258, @mathgaming)
• Make APT updates its package cache before dist-upgrade (#12465, @guoard)
• Nodelocaldns capabilities only use NET_ADMIN, not privileged (#12398, @tico88612)
• [reset] When flush_iptables: true, set IPv4/IPv6 default policies (INPUT/FORWARD/OUTPUT) to ACCEPT before flushing and delete user-defined chains to ensure a clean, non-locking reset. (#12552, @sasantk)

Other (Cleanup or Flake)

• Change the EOL Debian backports apt package to archive.debian.org (#12434, @tico88612)
• Dnsmasq directories are no longer cleaned up (#12380, @wangsifei99)
• Fix netcheck etcd image tag align with the etcd current version (#12402, @wangsifei99)
• Remove Ubuntu 20.04 support (#12301, @tico88612)
• When using timer based kubeadm certs renewal, only renew if certificates are near expiration (#12194, @panpan0000)

Components

• kubernetes 1.33.5
• etcd 3.5.22
• docker 28.3
• containerd 2.1.4
• cri-o 1.33.4
• cni-plugins 1.8.0
• calico 3.30.3
• cilium 1.18.2
• flannel 0.27.3
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.2.2
• kube-vip 0.8.0
• cert-manager 1.15.3
• coredns 1.12.0
• ingress-nginx 1.13.3
• argocd 2.14.5
• helm 3.18.4
• metallb 0.13.9
• registry 2.8.1
• aws-ebs-csi-plugin 0.5.0
• azure-csi-plugin 1.10.0
• cinder-csi-plugin 1.30.0
• gcp-pd-csi-plugin 1.9.2
• local-path-provisioner 0.0.32
• local-volume-provisioner 2.5.0
• node-feature-discovery 0.16.4

Changes by Kind

Bug or Regression

• Add argocd_install_checksum: str, to define the checksum of argocd_install_url (#12467, @k8s-infra-cherrypick-robot)
• Add missing addresses in kube-apiserver certificate SAN. (#12505, @k8s-infra-cherrypick-robot)
• Fix Cilium installation issues (caused by templating syntax errors) when certain non-default features (encryption, etc.) are enabled (#12283, @k8s-infra-cherrypick-robot)
• Fix Hubble-Relay peer discovery in clusters using non-default cluster name by properly configuring clusterDomain in Cilium Helm values (#12374, @k8s-infra-cherrypick-robot)
• Fix cilium installation role to render cilium_config_extra_vars into helm values (#12338, @k8s-infra-cherrypick-robot)
• Fix invalid PodSecurity admission configuration when kube_pod_security_use_default: false (#12478, @k8s-infra-cherrypick-robot)
• Fix the Cilium cluster, which is upgraded from 2.27 to 2.28 will break
  Fix helm release re-use message when installing repeatedly (#12324, @k8s-infra-cherrypick-robot)
• Fix the issue of etcd node addition failure caused by incorrect ETCD_INITIAL_CLUSTER configuration. (#12352, @k8s-infra-cherrypick-robot)
• Fix(kubeadm): Conditionally add --skip-phases flag for v1.32.0+ (#12354, @k8s-infra-cherrypick-robot)
• Fix: When running ./manage-offline-container-images.sh register with using Podman, getting the image_id fails and the script is interrupted. (#12316, @k8s-infra-cherrypick-robot)
• Fixed cilium_enable_bgp_control_plane config (#12432, @k8s-infra-cherrypick-robot)
• Fixes a syntax error that made the '_bgp_config' an 'AnsibleUnsafeText' instead of a 'dict', which caused the "Calico | Process BGP Configuration" step to fail (#12394, @k8s-infra-cherrypick-robot)

Components

• kubernetes 1.32.8
• etcd 3.5.22
• docker 28.0
• containerd 2.0.6
• cri-o 1.32.0
• cni-plugins 1.4.1
• calico 3.29.5
• cilium 1.17.7
• flannel 0.22.0
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.1.0
• weave 2.8.7
• kube-vip 0.8.0
• cert-manager 1.15.3
• coredns 1.11.3
• ingress-nginx 1.12.1
• argocd 2.14.5
• helm 3.16.4
• metallb 0.13.9
• registry 2.8.1
• aws-ebs-csi-plugin 0.5.0
• azure-csi-plugin 1.10.0
• cinder-csi-plugin 1.30.0
• gcp-pd-csi-plugin 1.9.2
• local-path-provisioner 0.0.24
• local-volume-provisioner 2.5.0
• node-feature-discovery 0.16.4

Changes by Kind

Feature

• The external_cloud_provider support manual option lets users install the cloud controller manager themselves. (#11884, @k8s-infra-cherrypick-robot)
• Upgrade ingress-nginx to version v1.12.1 to resolve critical vulnerabilities (CVE-2025-1974 and others) and webhook certgen to v1.5.2. (#12080, @tico88612)

Documentation

• Fix sample inventory for the reserved resource (#11922, @k8s-infra-cherrypick-robot)

Failing Test

• Fix CI by exclude the .ansible in .ansible-lint
  Remove ctr image pull workaround for nerdctl (#11956, @k8s-infra-cherrypick-robot)

Bug or Regression

• Add support for control plane reconfiguration on upgrades
  Add support for kubeadm-config v1beta4 UpgradeConfiguration.apply and UpgradeConfiguration.node
  Use kubeadm upgrade node during secondary control plane node upgrades (#12015, @chadswen)
• Fix coredns deployment with coredns_pod_disruption_budget: true or enable_nodelocaldns_secondary (#11957, @k8s-infra-cherrypick-robot)
• Fix: When running ./manage-offline-container-images.sh register with using Podman, getting the image_id fails and the script is interrupted. (#12314, @k8s-infra-cherrypick-robot)
• Install symlinks parroting as other control plane nodes etcd certificates (and key) on all control plane nodes, to make kubeadm works (#12192, @k8s-infra-cherrypick-robot)
• Make fallback_ip cacheable in facts (#12182, @guoard)
• [calico] Fix kubecontrollersconfigurations list permission (#12039, @k8s-infra-cherrypick-robot)

Components

• kubernetes v1.31.9
• etcd v3.5.21
• docker v26.1
• containerd v1.7.27
• cri-o v1.31.6
• cni-plugins v1.4.1
• calico v3.29.4
• cilium v1.15.9
• flannel v0.22.0
• kube-ovn v1.12.21
• kube-router v2.0.0
• multus v3.8
• weave v2.8.7
• kube-vip v0.8.0
• cert-manager v1.15.3
• coredns v1.11.3
• ingress-nginx v1.12.1
• krew v0.4.4
• argocd v2.11.0
• helm v3.16.4
• metallb v0.13.9
• registry v2.8.1
• cephfs-provisioner v2.1.0-k8s1.11
• rbd-provisioner v2.1.1-k8s1.11
• aws-ebs-csi-plugin v0.5.0
• azure-csi-plugin v1.10.0
• cinder-csi-plugin v1.30.0
• gcp-pd-csi-plugin v1.9.2
• local-path-provisioner v0.0.24
• local-volume-provisioner v2.5.0
• node-feature-discovery v0.16.4

Announcement

⚠️ This is the last version of RHEL 8 that we support, and it will be deprecated in the next release, see #11872 for a discussion and reasons why.
⚠️ We have removed the Weave CNI test in previous versions and will remove it in the next release because the project has been deprecated.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  Krew installation support is removed (#11824, @VannTen)
• Action required
  You should remove the leading 'v' of all explicit version of components deployed by kubespray (most notably kube_version) (#11890, @VannTen)
• Action required
  etcd_kubeadm_enabled (was deprecated) is removed. You should remove it from your inventory (#11901, @VannTen)
• gateway_api_experimental_channel is deprecated, please use gateway_api_channel and set experimental. (#11763, @tico88612)

Changes by Kind

Feature

• Add Kubernetes 1.32.x hash (#12161, @tmurakam) (#11885, @yankay) (#12003, @mzaian) (#12052, @0ekk)
• Add containerd 2.0.x hash (#11845, @mzaian) (#12011, @mzaian)
• Update runc binary to v1.2.4
  Set containerd_limit_open_file_num to 1048576 so it's configurable. (#11845, @mzaian)
• Update runc binary to v1.2.5 (#12011, @mzaian)
• Make nerdctl 2.0.3 default (#11913, @mzaian)
• Add deploy_coredns: bool (true by default), to let kubespray deploy or not coredns in kube-system (#12218, @ant31)
• Add option ubuntu_stop_unattended_upgrades to stop Ubuntu unattended upgrades (#12174, @0ekk)
• Add support for ranges: (start‑stop or single start) as an additional way to define Cilium LoadBalancer IP pools, alongside the existing cidrs: field. (#12140, @Kimcheolhui)
• Adds the script controb/offline/upload2artifactory.py for offline environments. (#11886, @bbaassssiiee)
• ArgoCD updated to version 2.14.5 to maintain compatibility with Kubernetes version 1.31. (#12041, @farshadasadpour)
• Automatically publish ingress-nginx service address if manual address is not specified and ingress-nginx is not using host network (#11879, @ThisIsQasim)
• Bump node-local-dns (k8s-dns-node-cache) image (#11981, @sathieu)
• Cilium CNI installation replaces Jinja template with Cilium CLI
  cilium_agent_custom_args and cilium_operator_custom_args are deprecated, please use cilium_agent_extra_args and cilium_operator_extra_args.
  cilium_identity_allocation_mode default change to crd.
  cilium_enable_host_legacy_routing default change to false.
  Add CIlium hubble export advanced flow log settings (cilium_hubble_export_file_max_backups, cilium_hubble_export_file_max_size_mb, cilium_hubble_export_dynamic_enabled and cilium_hubble_export_dynamic_config_content)
  Deprecated cilium_ipsec_node_encryption, replace it with cilium_encryption_node_encryption (#12101, @tico88612)
• Default etcd snapshot count to 10000 (#11997, @ErikJiang)
• Enable_dual_stack_networks deprecated, refact network stack with separate ipv4 and ipv6 (#11953, @borislitv)
• Ensure metrics port exists for nodelocaldns/nodelocaldns-second daemonsets (#11998, @Rickkwa)
• Fix cilium network plugin config issue deploying cilium 1.17 (#11986, @pedro-peter)
• For RHEL hosts, checking for subscription status timeout after rh_subscription_check_timeout (default to 3 minutes) (#12115, @VannTen)
• Gateway API can be brought forward before the CNI installation. (#12189, @tico88612)
• Improve ntp package conflict handling (#12212, @ErikJiang)
• Increase the control plane memory requirement to 2GB (#11864, @yankay)
• Network: Fix calico-kube-controller can't list the tiers resources (#12169, @cyclinder)
• Setting up a Docker image service for offline installation on a Mac (#11960, @diguage)
• Support containerd registry mirror certificate configuration (#11857, @KubeKyrie)
• Support kube-proxy nftables mode (#12060, @yankay)
• Terraform upcloud: Add possibility to setup cluster using nodes with no public IPs (#11696, @Xartos)
• Terraform: Added support for UpCloud routers and gateways (#11386, @Xartos)
• The external_cloud_provider support manual option lets users install the cloud controller manager themselves. (#11883, @tico88612)
• Tolerations of cilium-operator deployments can be defined using the cilium_operator_tolerations group_var (#12200, @felipe88alves)
• Update default crio capabilities to allow rancher to start (#11989, @jvkassi)
• Update CI test from AlmaLinux8 to AlmaLinux9 (#11889, @yankay)
• Update kube-vip to v0.8.9 (#11983, @sathieu)
• Upgrade OpenStack Cloud Controller Manager to v1.32.0 (#12121, @tico88612)
• Upgrade ingress-nginx to version v1.12.1 to resolve critical vulnerabilities (CVE-2025-1974 and others) and webhook certgen to v1.5.2. (#12075, @farshadasadpour)
• Upgrade kube-router to 2.1.1 (#12066, @VannTen)
• Upgrade load balancers image version to Nginx 1.27, Haproxy 3.1. (#11928, @guoard)
• Upgrade the default Docker version to 28.0 (#12070, @tico88612)
• Users can now configure hubble-export-file-max-backups and hubble-export-file-max-size-mb through the Kubespray inventory. (#12072, @ErmolenkoMaxim)
• [calico] Update default calico to v3.29.2 (#12012, @mzaian)
• [kubernetes/control-plane] Added support for structured AuthorizationConfiguration files. (#11852, @chadswen)

Documentation

• Fix documentation for offline usage by adding the 'v' prefix in download urls (#12166, @tmurakam)
• Fix path to facts.yml in node facts refresh section (#12177, @guoard)
• Fix sample inventory for the reserved resource (#11895, @anshuman-agarwala)
• No longer reserve outdated cephfs-provisioner installation and documentation (#12113, @tico88612)
• No longer reserve outdated rbd-provisioner installation and documentation (#12114, @tico88612)
• Our CRI-O default capabilities remove NET_RAW and SYS_CHROOT. (#12018, @tico88612)

Failing Test

• Add dns_autoscaler_affinity and remove in-place values. (#12165, @tico88612)
• Fix CI by exclude the .ansible in .ansible-lint
  Remove ctr image pull workaround for nerdctl (#11948, @yankay)

Bug or Regression

• Add support for control plane reconfiguration on upgrades
  Add support for kubeadm-config v1beta4 UpgradeConfiguration.apply and UpgradeConfiguration.node
  Use kubeadm upgrade node during secondary control plane node upgrades (#12015, @chadswen)
• Enable NRI by default on containerd (following containerd defaults) (#12152, @ShinyaIshitobi)
• File download.url's are masked unless the extra var unsafe_show_logs is true. (#11959, @bbaassssiiee)
• Fix a bug where kubeadm_certificate_key was not defined if control plane nodes were not in correct order (#11875, @Xartos)
• Fix a bug where custom TCP/UDP ports were not exposed by the ingress-nginx-controller container and service. (#11850, @commx)
• Fix broken calico Typha template when using both calico_ipam_host_local and typha_secure (#11917, @c-romeo)
• Fix broken dhclient hooks when using resolvconf (#11946, @kyrbrbik)
• Fix control plane pods deletion with proper shell quoting (#11943, @iptizer)
• Fix coredns deployment with coredns_pod_disruption_budget: true or enable_nodelocaldns_secondary (#11952, @RaulButuc)
• Fix hubble-ui deployment to not renders tls volume when the cilium_hubble_tls_generate option not configured. (#12143, @atobaum)
• Fix scale.yml problems with cached IP facts (#12020, @0ekk)
• Fix: Using the ./manage-offline-container-images.sh register command does not create a new container but registers the image in the existing container registry. (#11964, @DearJey)
• Fix: arm64 checksums for youki and kata-containers (#12173, @ErikJiang)
• Fix: missing 'v' prefix in offline image tags (#12086, @ErikJiang)
• Fix: prevent kubeadm to override coredns configuration/deployment on upgrade (#12028, @sathieu)
• Fixed an issue where the second and subsequent parameters in kubelet_cpu_manager_policy_options were ignored due to incorrect indentation. (#12123, @HoKim98)
• Fixed kube-vip to use kube-vip/kube-vip-iptables image instead of kube-vip/kube-vip when lb_fwdmethod or kube_vip_lb_fwdmethod is set to masquerade (#12145, @aviral-agarwal)
• Install symlinks parroting as other control plane nodes etcd certificates (and key) on all control plane nodes, to make kubeadm works (#12181, @VannTen)
• Kubelet-csr-approver moves to regular application installation (#12141, @tico88612)
• New Boolean default variable leave_etc_backup_files: true, set to false for uncluttered /etc directory on target nodes. (#11937, @bbaassssiiee)
• [calico] Fix kubecontrollersconfigurations list permission (#12035, @darkobas2)

Other (Cleanup or Flake)

• Binary checksums are no longer overridable from inventories or host facts (#12234, @VannTen)
• Calico-node pods no longer have a cpu limit by default (#11914, @VannTen)
• Enhance safety and validation mechanisms in the node removal process (#12085, @farshadasadpour)
• Heketi playbook (contrib) is removed. (#12091, @VannTen)
• Kubectl bash completion and alias available for Suse operation systems family (#11860, @noama-nv)
• Kubespray-defaults role is renamed to kubespray_defaults (#12202, @VannTen)
• Remove contrib/kvm-setup and contrib/mitogen. (#12093, @VannTen)
• Rename role bootstrap-os to bootstrap_os (#12203, @VannTen)
• Update KUBESPRAY_VERSION for v2.27.0 (#11854, @yankay)
• Update containerd.options key name (#12170, @flpanbin)
• Upgrade CI for openSuse 15.6 (#12074, @yankay)

Component versions

• kubernetes 1.32.5
• etcd 3.5.16
• docker 28.0
• containerd 2.0.5
• cri-o 1.32.0
• cni-plugins 1.4.1
• calico 3.29.3
• cilium 1.17.3
• flannel 0.22.0
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.1.0
• weave 2.8.7
• kube-vip 0.8.0
• cert-man...

Urgent Upgrade Notes

No, really, you MUST read this before you upgrade

• Action required
  Change kubeadm_patches format to use an array of inline patch instead of patch files.
  See the example for new format. (#11521, @VannTen)
• Action required
  Removes the generation of static tokens for every node in the cluster when kube_token_auth: true (#11567, @VannTen)
• Action required
  The kubelet_node_{config_extra_args,custom_flags} are removed. Use kubelet_{config_extra_args,custom_flags} in <your_inventory>/group_vars/kube_node.yml.
  The {kube,system}_master_{cpu,memory,ephemeral-storage,pid} are removed. Use the {kube,system}_{cpu,memory,ephemeral-storage,pid} variables in <your_inventory>/group_vars/kube_control_plane.yml. kubelet_custom_flags` can no longer be a string, an array is required. (#10643, @VannTen)
• Action required
  k8s_cluster group is now automatically defined, it can be removed from your inventory if you're not using it for group_vars (#11559, @VannTen)
• Action required
  kubeadm_ignore_preflight_errors is introduced to ignore specific preflight checks from kubeadm. The previous was effectively all, so some errors might surface during upgrade, in which cases, users should add the ones they choose to ignore to that variable. (#11710, @VannTen)

Container-Managers

• Containerd upgrade to 1.7.24 (#11833, @tico88612)
  Add hashes containerd 1.7.23 (#11642, @robertvolkmann)
  Add hashes containerd 1.7.22 default (#11554, @foobaar)
• Runc upgrade to v1.2.3 (#11833, @tico88612)
  Runc add version v1.1.14 (#11574, @janosbabik)

API Change

• If you use CRI-O and want to keep runc as your container default runtime when you upgrade cluster, you must set runc_enable: true and crio_default_runtime: "runc".
  Make CRI-O's default runtime configurable
  CRI-O v1.31 default runtime change to crun
  Crun upgrade to 1.17
  Skopeo upgrade to v1.16.1 (#11601, @tico88612)

Feature

• Make Kubernetes v1.31.4 default
  Add hashes for Kubernetes 1.31.4, 1.30.8 and 1.29.12 (#11828, @tico88612)
  Add hashes for Kubernetes 1.31.3, 1.30.7 and 1.29.11 (#11737, @tico88612)
  Add hashes for Kubernetes 1.31.2, 1.30.6 and 1.29.10 (#11662, @robertvolkmann)
  Add hashes for Kubernetes 1.31.1 and 1.31.0 (#11533, @philipsabri)
  Add hashes for kubernetes 1.29.8, 1.29.9, 1.30.5 (#11581, @DirkTheDaring)
• Add CI for openeuler 24.03
  Add CI Image for openeuler 24.03, 22.03 (#11689, @yankay)
• Add ResourceQuota AdmissionController plugin Configuration (#11814, @chadswen)
• Add a new CRI-O crio_root variable (#11692, @toliger)
• Add external Oracle cloud infrastructure cloud controller manager (#11378, @tico88612)
• Add optional support for Host Firewall and PolicyAuditMode features in Cilium (#11230, @ledroide)
• Add support Fedora 39/40 (#11573, @tico88612)
• Add support to use existing fips with terraform OpenStack (#11558, @anders-elastisys)
• Add the support of network isolation configuration in Multus. (#11605, @Sispheor)
• Added support for using ntpsec (#11665, @davidumea)
• Adds ingress_nginx_service_annotations variable to allow setting annotations for ingress-nginx controller service (#11544, @ThisIsQasim)
• Adds nodelocaldns_additional_configs variable (#11657, @0x4c6565)
• Allow disabling cilium hubble-ui using cilium_enable_hubble_ui variable (#10939, @pedro-peter)
• Allow to skip network configuration by setting kube_network_plugin value to none (#11844, @ant31)
• Configuration can now be supplied to ImagePolicyWebhook and PodNodeSelector admission plugins (#11471, @VannTen)
• Feat(calico): add support for numAllowedLocalASNumbers on bgppeers per node definition (#11570, @mirwan)
• Feat: Kubeadm config API support v1beta4 (#11674, @tico88612)
• Iproute is installed before gathering facts (needed for getting ansible_default_ipv4) (#11816, @0ekk)
• Partial Support of Cilium v1.16+ - kube-proxy replacement var changes
  Add optional support for configuring BGP Control Plane, IP Load Balancer Pools , Legacy BGP Peer Config v1 and BGP Config v2 features in Cilium (#11620, @logicsys)
• [cilium] Make cilium 1.15.9 default (#11593, @foobaar)
• Make cri-dockerd log level configurable (#11646, @mirwan)
• Remove support Fedora 37/38 (#11600, @tico88612)
• Reset operation: remove /var/log/containers and disable service auto-boot, make sure that multi-user.target.wants is deleted. (#11501, @leeonfu)
• Support Configuring EncryptionAlgorithm in Kubeadm v1beta4 (#11757, @ErikJiang)
• Update crictl to version v1.31.1 for Kubernetes 1.31
  Update crictl to version v1.30.1 for Kubernetes 1.30 (#11661, @robertvolkmann)
• Update multus to v4.1.0 (#11434, @ThisIsQasim)
• Upgrade CoreDNS version to v1.11.3 (#11653, @tico88612)
• Upgrade OpenStack Cloud Controller Manager to v1.31.1 (#11738, @tico88612)
• Upgrade pause container to 3.10 (#11695, @tico88612)
• [calico] Update default calico to v3.29.1 (#11798, @mzaian)
• [cert-manager] upgrade to v1.15.3 (#11668, @tico88612)
• [cri-o] Switch binaries to libexecdir
  Update youki version to 0.4.1 to fix ci. (#11584, @yankay)
• [etcd] Default version to 3.5.16 for 1.28, 1.29, 1.30, 1.31 (#11572, @janosbabik)
• [helm] Upgrade to v3.16.4, add 3.16.x checksum (#11832, @tico88612)
• [ingress-nginx] upgrade controller to version 1.12.0 (#11846, @mzaian)
• [need notice] update containerd max_container_log_line_size default value to 16384 (#11585, @KubeKyrie)
• [nerdctl] Default version to 1.7.7 (#11575, @janosbabik)

Documentation

• No longer support in-tree cloud provider, please delete or write external to the cloud_provider variable. (#11633, @tico88612)
• Remove inventory_builder scripts and contrib/dind (#11748, @VannTen)
• Update dns-stack.md reference in docs/ansible/vars.md (#11745, @emmanuel-ferdman)

Failing Test

• Fix CI: elastx cleanup security groups (#11411, @yankay)
• Update rockylinux9 base image to 9.5 (#11744, @yankay)

Bug or Regression

• Action required
  Running kubespray with --limit without cached facts is no longer supported. Improves the scaling for large clusters. (#11598, @VannTen)
• Always copy cert generation script to first etcd to pick up fixes on existing clusters (#11612, @VannTen)
• Fix Cilium agent permission can't read loadbalancerippools and secrets (#11466, @foobaar)
• Fix calico dual stack installation when using ip and ip6. (#11770, @VannTen)
• Fix collection usage for calico and other configuration depending on .sh and .conf files in Kubespray (#11707, @VannTen)
• Fix format of kubeadm-config v1beta4 (#11709, @VannTen)
• Fix kube-vip container securityContext (#11647, @KubeKyrie)
• Fix openEuler system packages installation (#11688, @VannTen)
• Fix pretty-printing (in kubectl) of nodelocaldns and coredns configmap when using dns_upstream_forward_extra_opts with an empty value option. (#11694, @VannTen)
• Fix spurious failure with 'localhost' when using scale.yml --limit <some nodes> (#11817, @VannTen)
• Fix task naming in bootstrap-os (#11714, @ErikJiang)
• Fix terraform.py on python >=3.12 (#11773, @enrico9034)
• Fix the check for cached data when using --limit (#11693, @VannTen)
• Fix the usage of --limit when using legacy groups (#11577, @VannTen)
• Fix usage of admission plugins configuration. (#11779, @VannTen)
• Fix using the default network manager in reset.yml (#11678, @KubeKyrie)
• Fix: cannot stop & remove all cri containers via remove_node.yml (#11631, @tico88612)
• Fixed: VSphere CSI and CPI drivers and are now retrieved from registry.k8s.io instead of gcr.io, as they have been deleted from the latter. Only a few recent versions are available in the new repository; if you have pinned vsphere_csi_controller, vsphere_csi_driver_image_tag or vsphere_syncer_image_tag to a version older than v3.1.2, please check if that version is available from the new repository. The same goes for external_vsphere_cloud_controller_image_tag which can no longer be latest, and should align with the running version of Kubernetes. It now defaults to v1.31.0. (#11564, @luringens)
• HA etcd cluster keeps quorum during upgrades. (#11677, @VannTen)
• Kubeadm images (kube-controller-manager,kube-scheduler,kube-apiserver,kube-proxy) are properly downloaded, including when using the download cache. (#11741, @VannTen)
• Make sure kubespray-defaults can be executed successfully by executing bootstrap-os first (#11441, @huangkevin404)
• Make upcloud csi_driver use the correct pull secret (#11597, @VannTen)
• Modifies Helm parameters wait and atomic to be set to false when using kube_network_plugin=cni to prevent deployment issues with kubelet-csr-approver. (#11704, @M-JavadHeydarpour)
• Remove invalid extraArgs entry and update template file reference (#11703, @agravgaard)
• Update calico-nopde template and remove flexvol-driver initContainer (#11634, @KubeKyrie)
• Use correct version for community.general collection (#11724, @VannTen)

Other (Cleanup or Flake)

• Cleanup older terminology, replace "master" with "control plane" (#11394, @bogd)
• Drop support for Kubernetes 1.28.x minimum version now is 1.29.x
  Drop support for CRI-O 1.28.x minimum version now is 1.29.x (#11609, @yankay)
• Fix roles/download/tasks/download_file.yml task name typo (#11684, @dmncmn)
• Optimize CA cert hash calculation with community.crypto (#11758, @ErikJiang)
• Remove pip install . support and rpm spec file (#11760, @VannTen)
• Replace deprecated unarchive.copy with unarchive.remote_src (#11207, @Payback159)
• Update KUBESPRAY_VERSION for v2.26.0 (#11511, @yankay)
• containerd_use_config_path is removed as kubespray now always use containerd config_path configuration. (#11755, @VannTen)