Deploy OpenShift on KVM / Libvirt with the Agent-Based Installer

A full OpenShift cluster on one KVM host no longer needs a bastion VM, a PXE server or HAProxy. The agent-based installer boots every node from a single ISO. The dnsmasq instance that libvirt runs for the network hands out the DHCP leases and DNS records. Keepalived inside the cluster holds the API and ingress addresses. What is left for you is one network definition, two YAML files and a virt-install loop.

Original content from computingforgeeks.com - post 99936

This guide deploys OpenShift Container Platform on KVM and libvirt with three control plane nodes and one worker. A second worker joins after install with oc adm node-image create. Then the image registry gets NFS storage, a sample app gets built and exposed, and workloads move off the control plane. It replaces our older PXE and HAProxy walkthrough. Every command below was run on a Rocky Linux 10.2 KVM host against OpenShift 4.22.15 in October 2026, and the errors at the end are the ones that run produced.

What the lab looks like

Everything lives on one NAT network, 192.168.100.0/24, which libvirt creates on the host. The cluster is called ocp4 under the base domain example.com, so the API answers at api.ocp4.example.com and apps land under *.apps.ocp4.example.com. Two addresses are virtual IPs, not machines. OpenShift moves them between nodes with keepalived.

Name	Role	IP address	MAC address	vCPU	RAM
kvm01	KVM host and libvirt gateway	192.168.100.1	n/a	16 threads	128 GB
api VIP	API and api-int	192.168.100.5	n/a	n/a	n/a
ingress VIP	*.apps wildcard	192.168.100.7	n/a	n/a	n/a
master01	Control plane (rendezvous host)	192.168.100.11	52:54:00:4f:00:11	6	20 GB
master02	Control plane	192.168.100.12	52:54:00:4f:00:12	6	20 GB
master03	Control plane	192.168.100.13	52:54:00:4f:00:13	6	20 GB
worker01	Compute	192.168.100.21	52:54:00:4f:00:21	4	16 GB
worker02	Compute, added after install	192.168.100.22	52:54:00:4f:00:22	4	16 GB

The Red Hat agent-based installer docs recommend 8 vCPUs, 16 GB of RAM and 120 GB of disk for every node in compact and HA clusters. Our host had 16 threads, so the five VMs share 26 vCPUs. That is overcommitted and below the recommendation, and the cluster still installed and ran builds. Treat it as a floor for a lab, not a sizing guide. The RAM is the hard limit: 92 GB of guest memory on a 128 GB host leaves room for the host itself and nothing else.

Before you start

The host needs hardware virtualization exposed to it. A bare-metal box works. So does a cloud VM with nested virtualization; ours was an AWS r8i.4xlarge (16 vCPU, 128 GB) launched with nested virtualization enabled, running the Rocky Linux 10 AMI. You also need:

• RHEL 10, Rocky Linux 10, AlmaLinux 10 or a current Fedora on the host, with sudo access
• About 600 GB of free disk for five thin-provisioned 120 GB qcow2 images
• A Red Hat account and the pull secret from console.redhat.com, saved as ~/.openshift/pull-secret on the host
• Outbound internet from the host, because the nodes pull release images from quay.io and registry.redhat.io

No subscription is needed for the install itself. The cluster runs as a 60-day self-support trial until you attach it to a subscription.

Install KVM and libvirt on the host

Check that the CPU flags reach the host first. Any number above zero is fine:

grep -cE 'vmx|svm' /proc/cpuinfo

Our host printed 32, two matching lines for each of its 16 threads:

32

Install the hypervisor stack. The nmstate package is only used by the installer when you give nodes static network configs, but it costs nothing to have it ready:

sudo dnf -y install qemu-kvm libvirt virt-install guestfs-tools libvirt-client nmstate tar jq bind-utils

Rocky 10 ships the modular libvirt daemons, so enable their sockets rather than the old monolithic libvirtd:

sudo systemctl enable --now virtqemud.socket virtnetworkd.socket virtstoraged.socket virtnodedevd.socket

Then let libvirt check the host:

sudo virt-host-validate qemu

Every line that matters passed. The two warnings are about PCI passthrough and confidential computing, neither of which OpenShift needs here:

  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device '/dev/kvm' exists                                 : PASS
  QEMU: Checking if device '/dev/kvm' is accessible                          : PASS
  QEMU: Checking if device '/dev/vhost-net' exists                           : PASS
  QEMU: Checking if device '/dev/net/tun' exists                             : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
  QEMU: Checking for secure guest support                                    : WARN (None of SEV, SEV-ES, SEV-SNP, TDX available)

Our KVM install guide for Rocky and AlmaLinux covers bridges, storage pools and virt-manager if you want more than the minimum.

Create the cluster network with DHCP and DNS

This one file replaces the bastion. It defines a NAT network with fixed DHCP leases for each node MAC, A records for the API and every node, and a dnsmasq wildcard for *.apps. Libvirt builds matching PTR records from the dns host entries, which OpenShift wants for node names. Open the file:

vim /tmp/ocp4-net.xml

Paste the network definition:

<network xmlns:dnsmasq='http://libvirt.org/schemas/network/dnsmasq/1.0'>
  <name>ocp4</name>
  <forward mode='nat'/>
  <bridge name='virbr-ocp4' stp='on' delay='0'/>
  <domain name='ocp4.example.com' localOnly='yes'/>
  <dns>
    <host ip='192.168.100.5'>
      <hostname>api.ocp4.example.com</hostname>
      <hostname>api-int.ocp4.example.com</hostname>
    </host>
    <host ip='192.168.100.11'>
      <hostname>master01.ocp4.example.com</hostname>
    </host>
    <host ip='192.168.100.12'>
      <hostname>master02.ocp4.example.com</hostname>
    </host>
    <host ip='192.168.100.13'>
      <hostname>master03.ocp4.example.com</hostname>
    </host>
    <host ip='192.168.100.21'>
      <hostname>worker01.ocp4.example.com</hostname>
    </host>
    <host ip='192.168.100.22'>
      <hostname>worker02.ocp4.example.com</hostname>
    </host>
  </dns>
  <ip address='192.168.100.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.100.100' end='192.168.100.199'/>
      <host mac='52:54:00:4f:00:11' name='master01' ip='192.168.100.11'/>
      <host mac='52:54:00:4f:00:12' name='master02' ip='192.168.100.12'/>
      <host mac='52:54:00:4f:00:13' name='master03' ip='192.168.100.13'/>
      <host mac='52:54:00:4f:00:21' name='worker01' ip='192.168.100.21'/>
      <host mac='52:54:00:4f:00:22' name='worker02' ip='192.168.100.22'/>
    </dhcp>
  </ip>
  <dnsmasq:options>
    <dnsmasq:option value='address=/apps.ocp4.example.com/192.168.100.7'/>
  </dnsmasq:options>
</network>

Define it, mark it to start at boot, and start it. libvirt keeps its own copy, so the file in /tmp is only the input:

sudo virsh net-define /tmp/ocp4-net.xml
sudo virsh net-autostart ocp4
sudo virsh net-start ocp4

Each command confirms on its own line:

Network ocp4 defined from /tmp/ocp4-net.xml

Network ocp4 marked as autostarted

Network ocp4 started

Query that dnsmasq directly at 192.168.100.1. The cluster domain goes in a variable because the next few checks all use it:

export CLUSTER_DOMAIN="ocp4.example.com"
dig +short @192.168.100.1 api.${CLUSTER_DOMAIN}
dig +short @192.168.100.1 test.apps.${CLUSTER_DOMAIN}
dig +short @192.168.100.1 master02.${CLUSTER_DOMAIN}
dig +short @192.168.100.1 -x 192.168.100.12

The API resolves to its VIP, any name under apps hits the ingress VIP, and the node record works both ways:

192.168.100.5
192.168.100.7
192.168.100.12
master02.ocp4.example.com.

Point the host at the cluster DNS

The installer runs on the host and has to resolve api.ocp4.example.com too. On RHEL-family hosts NetworkManager owns /etc/resolv.conf, so switch it to its dnsmasq plugin and forward only the cluster domain to libvirt. Everything else keeps using the upstream resolver. (Fedora uses systemd-resolved instead, and we did not test that variant.)

printf "[main]\ndns=dnsmasq\n" | sudo tee /etc/NetworkManager/conf.d/ocp4-dns.conf
echo "server=/ocp4.example.com/192.168.100.1" | sudo tee /etc/NetworkManager/dnsmasq.d/ocp4.conf
sudo systemctl reload NetworkManager

The host now answers cluster names through the local resolver:

getent hosts api.${CLUSTER_DOMAIN}

Same VIP as the direct query:

192.168.100.5   api.ocp4.example.com

Download the OpenShift installer and oc

The mirror keeps a stable alias that always points at the newest stable release, including the next minor once it ships. Read the version from it. To stay on one minor, swap stable for a channel such as stable-4.22, which is the channel we actually read in the lab:

OCP_VER=$(curl -s https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/release.txt | awk '/^Name:/{print $2}')
echo "${OCP_VER}"

This printed the release we installed:

4.22.15

Download the client and installer tarballs, then unpack only the binaries into /usr/local/bin:

cd /tmp
curl -sLO "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/${OCP_VER}/openshift-client-linux.tar.gz"
curl -sLO "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/${OCP_VER}/openshift-install-linux.tar.gz"
sudo tar -xzf openshift-client-linux.tar.gz -C /usr/local/bin oc kubectl
sudo tar -xzf openshift-install-linux.tar.gz -C /usr/local/bin openshift-install

The installer tarball is the big one at 448 MB, because it embeds the release metadata. Check both tools:

openshift-install version
oc version --client

The installer reports the release image it will deploy:

openshift-install 4.22.15
built from commit cdaf698d75d93e6ed9bf6203c988c5321a086758
release image quay.io/openshift-release-dev/ocp-release@sha256:fed788eac1c99388dd9b78dda4d6a73e39b70abb00ca918d2bb456a97187f0c1
release architecture amd64
Client Version: 4.22.15
Kustomize Version: v5.7.1

Write install-config.yaml and agent-config.yaml

The installer consumes these two files and deletes them when it builds the ISO. Keep the originals in one directory and copy them into a working directory each time. Create both directories and an SSH key for the core user on the nodes (skip the ssh-keygen line if you already have a key you want to use):

mkdir -p ~/ocp4-config ~/ocp4-install
ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519

Open the install config:

vim ~/ocp4-config/install-config.yaml

This is the file we ran, with the pull secret and SSH key cut out. Paste your pull secret JSON on one line between the quotes, and the contents of ~/.ssh/id_ed25519.pub as the SSH key:

apiVersion: v1
baseDomain: example.com
metadata:
  name: ocp4
compute:
- name: worker
  replicas: 1
controlPlane:
  name: master
  replicas: 3
networking:
  networkType: OVNKubernetes
  machineNetwork:
  - cidr: 192.168.100.0/24
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16
platform:
  baremetal:
    apiVIPs:
    - 192.168.100.5
    ingressVIPs:
    - 192.168.100.7
pullSecret: '{"auths": ...}'
sshKey: 'ssh-ed25519 AAAA... josphat@kvm01'

The platform: baremetal block is what removes the load balancer. With platform: none the installer expects you to provide HAProxy and full DNS yourself, which is the old bastion design. Here OpenShift runs keepalived and keeps the two VIPs alive on healthy nodes. The machineNetwork CIDR must contain both VIPs.

Now the agent config. It maps each MAC to a hostname and role and names the rendezvous host, the control plane node that runs the temporary assisted service during install:

vim ~/ocp4-config/agent-config.yaml

The file as executed:

apiVersion: v1beta1
kind: AgentConfig
metadata:
  name: ocp4
rendezvousIP: 192.168.100.11
hosts:
- hostname: master01
  role: master
  interfaces:
  - name: enp1s0
    macAddress: 52:54:00:4f:00:11
  rootDeviceHints:
    deviceName: /dev/vda
- hostname: master02
  role: master
  interfaces:
  - name: enp1s0
    macAddress: 52:54:00:4f:00:12
  rootDeviceHints:
    deviceName: /dev/vda
- hostname: master03
  role: master
  interfaces:
  - name: enp1s0
    macAddress: 52:54:00:4f:00:13
  rootDeviceHints:
    deviceName: /dev/vda
- hostname: worker01
  role: worker
  interfaces:
  - name: enp1s0
    macAddress: 52:54:00:4f:00:21
  rootDeviceHints:
    deviceName: /dev/vda

There is no networkConfig section because DHCP hands out the addresses. The rootDeviceHints pin the install to the virtio disk, so the CD-ROM device that holds the ISO can never be picked as the target.

Build the agent ISO

Copy the configs into the working directory and generate the image:

cp ~/ocp4-config/*.yaml ~/ocp4-install/
cd ~/ocp4-install
openshift-install agent create image --dir . --log-level=info

It pulled the RHCOS base ISO out of the release payload and finished in 55 seconds:

level=info msg=Configuration has 3 master replicas, 0 arbiter replicas, and 1 worker replicas
level=info msg=The rendezvous host IP (node0 IP) is 192.168.100.11
level=info msg=Extracting base ISO from release payload
level=info msg=Base ISO obtained from release and cached at [/home/josphat/.cache/agent/image_cache/coreos-x86_64.iso]
level=info msg=Consuming Install Config from target directory
level=info msg=Consuming Agent Config from target directory
level=info msg=Generated ISO at agent.x86_64.iso.

The directory now holds a 1.4 GB agent.x86_64.iso plus an auth folder with the kubeconfig and kubeadmin-password files. Back the auth folder up. It is the only admin login until you add an identity provider. Copy the ISO where qemu can read it:

sudo cp ~/ocp4-install/agent.x86_64.iso /var/lib/libvirt/images/

Create the cluster VMs

Attach the ISO as a regular CD-ROM disk and use --import. Don’t pass --cdrom. With --cdrom --noautoconsole, virt-install starts the install-phase VM with on_reboot=destroy and exits, so nothing restarts it when the guest reboots. Every node reboots once after writing RHCOS to disk, so the cluster stalls with hosts waiting for user action. We hit exactly that on the first pass (the error is at the end). The function below is the variant that fixed it:

create_node() {
  local name=$1 mac=$2 vcpus=$3 ram=$4 iso=$5
  sudo virt-install --name "$name" \
    --vcpus "$vcpus" --memory "$ram" --cpu host-passthrough \
    --osinfo rhel9.8 \
    --disk size=120,bus=virtio,cache=none,io=native,boot.order=1 \
    --disk path="$iso",device=cdrom,readonly=on,boot.order=2 \
    --network network=ocp4,mac="$mac" \
    --graphics vnc --import --noautoconsole
}

The disk boots first. It is empty on the first start, so the firmware falls through to the ISO. After RHCOS lands on /dev/vda, every later boot comes from disk. The rhel9.8 OS variant matches the RHCOS build inside the ISO, which file reports as rhcos-9.8. Create the four nodes:

ISO=/var/lib/libvirt/images/agent.x86_64.iso
create_node master01 52:54:00:4f:00:11 6 20480 "$ISO"
create_node master02 52:54:00:4f:00:12 6 20480 "$ISO"
create_node master03 52:54:00:4f:00:13 6 20480 "$ISO"
create_node worker01 52:54:00:4f:00:21 4 16384 "$ISO"

Confirm the reboot behaviour on any of them:

sudo virsh dumpxml master01 | grep on_reboot

It must print restart, not destroy. The full check with its output is in the worker02 section below, on the node we built with this function.

The fixed DHCP leases show up within a minute of the VMs booting the ISO:

sudo virsh net-dhcp-leases ocp4

Each MAC got the address from the network definition:

 Expiry Time           MAC address         Protocol   IP address          Hostname   Client ID or DUID
-----------------------------------------------------------------------------------------------------------
 2026-10-03 05:52:40   52:54:00:4f:00:11   ipv4       192.168.100.11/24   master01   01:52:54:00:4f:00:11
 2026-10-03 05:52:42   52:54:00:4f:00:12   ipv4       192.168.100.12/24   master02   01:52:54:00:4f:00:12
 2026-10-03 05:52:44   52:54:00:4f:00:13   ipv4       192.168.100.13/24   master03   01:52:54:00:4f:00:13
 2026-10-03 05:52:47   52:54:00:4f:00:21   ipv4       192.168.100.21/24   worker01   01:52:54:00:4f:00:21

Watch the installation

The installer talks to the assisted service on the rendezvous host and streams its progress. Run the bootstrap wait first:

cd ~/ocp4-install
openshift-install agent wait-for bootstrap-complete --dir . --log-level=info

The hosts first validate each other. Ours all failed NTP for about a minute, then passed on their own, and the install started:

level=warning msg=Host master01 validation: Host couldn't synchronize with any NTP server
level=info msg=Host master01 validation: Host NTP is synced
level=info msg=Cluster is ready for install
level=info msg=Cluster validation: All hosts in the cluster are ready to install.
level=info msg=Preparing cluster for installation

The non-rendezvous nodes write RHCOS to disk and reboot. The rendezvous host runs the bootstrap control plane and reboots last. The bootstrap wait gives up after 60 minutes. Ours did exactly that (bootstrap process timed out: context deadline exceeded) while the hosts were still progressing, so the timeout alone does not mean failure. Whether it exits cleanly or times out, move on to the install wait, which picks up wherever the cluster is:

openshift-install agent wait-for install-complete --dir . --log-level=info

It ends with the console URL and the kubeadmin login:

level=info msg=Bootstrap is complete
level=info msg=cluster bootstrap is complete
level=info msg=Waiting up to 40m0s (until 6:35AM UTC) for the cluster at https://api.ocp4.example.com:6443 to initialize...
level=info msg=Waiting up to 30m0s (until 6:37AM UTC) to ensure each cluster operator has finished progressing...
level=info msg=All cluster operators have completed progressing
level=info msg=Checking to see if there is a route at openshift-console/console...
level=info msg=Install complete!
level=info msg=To access the cluster as the system:admin user when using 'oc', run
level=info msg=    export KUBECONFIG=/home/josphat/ocp4-install/auth/kubeconfig
level=info msg=Access the OpenShift web-console here: https://console-openshift-console.apps.ocp4.example.com
level=info msg=Login to the console with user: "kubeadmin", and password: "REDACTED"

From creating the VMs to Install complete took about 75 minutes on this overcommitted host. Part of that went to the power-off problem from the first pass, so a clean run on a host with real cores should be quicker.

Verify the cluster

Point oc at the generated kubeconfig and list the nodes:

export KUBECONFIG=~/ocp4-install/auth/kubeconfig
oc get nodes -o wide

All four nodes are Ready, each on its fixed DHCP address:

NAME       STATUS   ROLES                         AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                                                KERNEL-VERSION                 CONTAINER-RUNTIME
master01   Ready    control-plane,master,worker   28m   v1.35.6   192.168.100.11   <none>        Red Hat Enterprise Linux CoreOS 9.8.20260915-0 (Plow)   5.14.0-687.48.1.el9_8.x86_64   cri-o://1.35.8-11.rhaos4.22.giteaf45bc.el9
master02   Ready    control-plane,master,worker   44m   v1.35.6   192.168.100.12   <none>        Red Hat Enterprise Linux CoreOS 9.8.20260915-0 (Plow)   5.14.0-687.48.1.el9_8.x86_64   cri-o://1.35.8-11.rhaos4.22.giteaf45bc.el9
master03   Ready    control-plane,master,worker   44m   v1.35.6   192.168.100.13   <none>        Red Hat Enterprise Linux CoreOS 9.8.20260915-0 (Plow)   5.14.0-687.48.1.el9_8.x86_64   cri-o://1.35.8-11.rhaos4.22.giteaf45bc.el9
worker01   Ready    worker                        28m   v1.35.6   192.168.100.21   <none>        Red Hat Enterprise Linux CoreOS 9.8.20260915-0 (Plow)   5.14.0-687.48.1.el9_8.x86_64   cri-o://1.35.8-11.rhaos4.22.giteaf45bc.el9

The control plane nodes also carry the worker role. Red Hat treats three masters with zero or one worker as a compact cluster, and the installer makes the masters schedulable so workloads and the two router replicas have room. A later section undoes that once a second worker joins. Check the cluster version and operators:

oc get clusterversion
oc get co

Every operator reports Available and nothing is Degraded:

NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.22.15   True        False         18m     Cluster version is 4.22.15
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.22.15   True        False         False      19m
baremetal                                  4.22.15   True        False         False      41m
cloud-controller-manager                   4.22.15   True        False         False      44m
cloud-credential                           4.22.15   True        False         False      85m
cluster-autoscaler                         4.22.15   True        False         False      42m
config-operator                            4.22.15   True        False         False      42m
console                                    4.22.15   True        False         False      25m
control-plane-machine-set                  4.22.15   True        False         False      41m
csi-snapshot-controller                    4.22.15   True        False         False      41m
dns                                        4.22.15   True        False         False      41m
etcd                                       4.22.15   True        False         False      39m
image-registry                             4.22.15   True        False         False      30m
ingress                                    4.22.15   True        False         False      30m
insights                                   4.22.15   True        False         False      42m
kube-apiserver                             4.22.15   True        False         False      36m
kube-controller-manager                    4.22.15   True        False         False      36m
kube-scheduler                             4.22.15   True        False         False      39m
kube-storage-version-migrator              4.22.15   True        False         False      42m
machine-api                                4.22.15   True        False         False      38m
machine-approver                           4.22.15   True        False         False      41m
machine-config                             4.22.15   True        False         False      41m
marketplace                                4.22.15   True        False         False      41m
monitoring                                 4.22.15   True        False         False      23m
network                                    4.22.15   True        False         False      42m
node-tuning                                4.22.15   True        False         False      27m
olm                                        4.22.15   True        False         False      41m
openshift-apiserver                        4.22.15   True        False         False      29m
openshift-controller-manager               4.22.15   True        False         False      30m
openshift-samples                          4.22.15   True        False         False      30m
operator-lifecycle-manager                 4.22.15   True        False         False      41m
operator-lifecycle-manager-catalog         4.22.15   True        False         False      42m
operator-lifecycle-manager-packageserver   4.22.15   True        False         False      31m
service-ca                                 4.22.15   True        False         False      42m
storage                                    4.22.15   True        False         False      41m

The core user on every node takes the SSH key from the install config, which is how you get onto a node when the API is down:

ssh [email protected] cat /etc/redhat-release

RHCOS reports its RHEL base:

Red Hat Enterprise Linux release 9.8 (Plow)

Open the web console from your workstation

The console lives on a private network inside the host, so your laptop can’t resolve or reach it. A SOCKS tunnel fixes both at once, because the browser hands DNS lookups to the far end where the host resolver knows the cluster domain. From your workstation:

ssh -N -D 1080 josphat@kvm01

Leave that running and start a browser that uses the tunnel. Chrome and Chromium send DNS through a SOCKS5 proxy by default. The separate --user-data-dir matters, because Chrome ignores --proxy-server when another Chrome window is already open:

google-chrome --user-data-dir=/tmp/ocp-chrome --proxy-server="socks5://127.0.0.1:1080" https://console-openshift-console.apps.ocp4.example.com

In Firefox, set a manual SOCKS v5 proxy on 127.0.0.1:1080 and tick “Proxy DNS when using SOCKS v5”. Accept the self-signed certificate warnings for the console and the OAuth server, then log in as kubeadmin with the password from ~/ocp4-install/auth/kubeadmin-password.

The Overview page confirms the BareMetal infrastructure provider and a healthy control plane. The one warning, AlertmanagerReceiversNotConfigured, only means nobody gets alert notifications yet:

Compute, then Nodes, lists all five machines with their real memory and CPU use. Only worker01 belongs to a MachineSet. The day-2 worker we add next shows up with none, which is expected for nodes added from an ISO:

Administration, then Cluster Settings, shows the installed release, its update channel and the 60-day self-support trial counter:

Add a worker node after install

Adding a node after day one needs no access to the original install directory. oc adm node-image create reads the configuration stored in the cluster and builds a node ISO for the MAC you give it. It needs rsync on the host and the pull secret:

sudo dnf -y install rsync
mkdir -p ~/add-worker02 && cd ~/add-worker02
oc adm node-image create --mac-address=52:54:00:4f:00:22 --hostname=worker02 --root-device-hint=deviceName:/dev/vda -a ~/.openshift/pull-secret

It took 1 minute 50 seconds and wrote node.x86_64.iso:

2026-10-03T06:26:19Z [node-image create] Launching command
2026-10-03T06:27:20Z [node-image create] Gathering additional information from the target cluster
2026-10-03T06:27:20Z [node-image create] Creating internal configuration manifests
2026-10-03T06:27:20Z [node-image create] Rendering ISO ignition
2026-10-03T06:27:20Z [node-image create] Retrieving the base ISO image
2026-10-03T06:27:20Z [node-image create]   Extracting base image from release payload
2026-10-03T06:27:45Z [node-image create]   Verifying base image version
2026-10-03T06:27:55Z [node-image create] Creating agent artifacts for the final image
2026-10-03T06:27:55Z [node-image create]   Extracting required artifacts from release payload
2026-10-03T06:28:05Z [node-image create]   Preparing artifacts
2026-10-03T06:28:05Z [node-image create] Assembling ISO image
2026-10-03T06:28:05Z [node-image create] Saving ISO image to /home/josphat/add-worker02
2026-10-03T06:28:10Z [node-image create] Command successfully completed

The worker02 MAC and IP are already in the network definition, so DHCP and DNS are ready for it. Boot the new VM from the node ISO with the same function:

sudo cp ~/add-worker02/node.x86_64.iso /var/lib/libvirt/images/
create_node worker02 52:54:00:4f:00:22 4 16384 /var/lib/libvirt/images/node.x86_64.iso

Check its reboot setting before it gets that far:

sudo virsh dumpxml worker02 | grep -E "on_reboot|boot order"

The disk carries boot order 1, the CD-ROM order 2, and a guest reboot restarts the VM. Our first four nodes used --cdrom, and the one still in its install phase showed destroy here, which is the error covered at the end:

  <on_reboot>restart</on_reboot>
      <boot order='1'/>
      <boot order='2'/>

Then follow it from the cluster side:

oc adm node-image monitor --ip-addresses 192.168.100.22 -a ~/.openshift/pull-secret

The monitor walks through validation, the disk write and one reboot. Then it stops at two certificate signing requests that need a human:

2026-10-03T06:31:08Z [node-image monitor] Node 192.168.100.22: Host worker02: updated status from known to installing (Installation is in progress)
2026-10-03T06:32:23Z [node-image monitor] Node 192.168.100.22: Host: worker02, reached installation stage Writing image to disk: 100%
2026-10-03T06:32:28Z [node-image monitor] Node 192.168.100.22: Host: worker02, reached installation stage Rebooting
2026-10-03T06:35:21Z [node-image monitor] Node 192.168.100.22: First CSR Pending approval
2026-10-03T06:35:21Z [node-image monitor] Node 192.168.100.22: CSR csr-7q75r with signerName kubernetes.io/kube-apiserver-client-kubelet and username system:serviceaccount:openshift-machine-config-operator:node-bootstrapper is Pending and awaiting approval

Approve every pending CSR. Run it once for the client CSR, then again a few minutes later when the kubelet serving CSR appears:

oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

The first run approved the client CSR:

certificatesigningrequest.certificates.k8s.io/csr-7q75r approved

After the second approval the monitor exits. The whole join took about 10 minutes. List the nodes:

oc get nodes

worker02 is Ready alongside the original four:

NAME       STATUS   ROLES                         AGE     VERSION
master01   Ready    control-plane,master,worker   46m     v1.35.6
master02   Ready    control-plane,master,worker   62m     v1.35.6
master03   Ready    control-plane,master,worker   62m     v1.35.6
worker01   Ready    worker                        46m     v1.35.6
worker02   Ready    worker                        6m26s   v1.35.6

Move workloads off the control plane

With two workers, the masters no longer need to run application pods. Flip the scheduler setting:

oc patch schedulers.config.openshift.io cluster --type merge -p '{"spec":{"mastersSchedulable":false}}'

Give it a few seconds and list the nodes again:

oc get nodes

The worker role is gone from the masters:

NAME       STATUS   ROLES                  AGE     VERSION
master01   Ready    control-plane,master   47m     v1.35.6
master02   Ready    control-plane,master   63m     v1.35.6
master03   Ready    control-plane,master   63m     v1.35.6
worker01   Ready    worker                 47m     v1.35.6
worker02   Ready    worker                 6m57s   v1.35.6

Pods already on the masters stay where they are. That matters most for the routers, because the ingress VIP follows the router pods. Ours still sat on master02 and master03. Restart the router deployment so the scheduler places them on workers:

oc rollout restart deployment/router-default -n openshift-ingress
oc rollout status deployment/router-default -n openshift-ingress --timeout=300s
oc get pods -n openshift-ingress -o wide

Both new router pods landed on the workers:

NAME                              READY   STATUS        RESTARTS      AGE   IP               NODE       NOMINATED NODE   READINESS GATES
router-default-6dc5cfbf65-qgs8l   0/1     Terminating   4 (51m ago)   62m   192.168.100.13   master03   <none>           <none>
router-default-99d9f7d6d-bh8v7    1/1     Running       0             35s   192.168.100.21   worker01   <none>           <none>
router-default-99d9f7d6d-vjh82    1/1     Running       0             70s   192.168.100.22   worker02   <none>           <none>

Thirty seconds later, keepalived had moved 192.168.100.7 to worker02, and both the console and a test route still answered HTTP 200 through the VIP.

Give the image registry NFS storage

On bare metal the internal image registry installs in the Removed state, because there is no default storage class. Builds need it. The KVM host is already on the cluster network at 192.168.100.1, so it can serve NFS. Our NFS server guide for Rocky and AlmaLinux covers the server side in depth. The short version for this lab:

sudo dnf -y install nfs-utils
sudo mkdir -p /srv/nfs/ocp-registry
sudo chown root:root /srv/nfs/ocp-registry
sudo chmod 0770 /srv/nfs/ocp-registry
echo "/srv/nfs/ocp-registry 192.168.100.0/24(rw,sync,no_subtree_check,no_root_squash)" | sudo tee /etc/exports.d/ocp-registry.exports
sudo systemctl enable --now nfs-server

The ownership is not arbitrary. The registry pod runs as a random UID with group 0, so the directory must be writable by the root group. We first used nobody:nobody, and every push failed with a 500 error (see the end of the guide).

If firewalld runs on the host, the libvirt bridge sits in the libvirt zone. That zone only allows DHCP, DNS, SSH and TFTP from guests. Our test from master01 was refused until NFS was added:

sudo firewall-cmd --zone=libvirt --add-service={nfs,rpc-bind,mountd} --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --zone=libvirt --list-services

NFS now shows in the zone:

dhcp dhcpv6 dns mountd nfs rpc-bind ssh tftp

Create a PersistentVolume that points at the export:

vim ~/registry-pv.yaml

Paste the volume definition:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: image-registry-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 192.168.100.1
    path: /srv/nfs/ocp-registry

Apply it, then set the registry to Managed with two replicas. An empty claim tells the operator to create its own PVC named image-registry-storage, which binds to the PV above:

oc apply -f ~/registry-pv.yaml
oc patch configs.imageregistry.operator.openshift.io cluster --type merge -p '{"spec":{"managementState":"Managed","replicas":2,"storage":{"pvc":{"claim":""}}}}'

Give the operator a minute, then check the claim and the registry pods:

oc get pvc -n openshift-image-registry
oc get pods -n openshift-image-registry -l docker-registry=default -o wide

The claim is bound and both registry pods are running:

NAME                     STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
image-registry-storage   Bound    image-registry-pv   100Gi      RWX                           <unset>                 60s
NAME                              READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
image-registry-5bf9d78cff-674h5   1/1     Running   0          59s   10.128.0.100   master02   <none>           <none>
image-registry-5bf9d78cff-fkk94   1/1     Running   0          59s   10.129.0.105   master03   <none>           <none>

Build and expose a test app

A source-to-image build exercises the whole stack at once: a Git clone from the internet, a build pod, a push to the internal registry on NFS, a deployment and a route through the ingress VIP:

oc new-project demo
oc new-app httpd~https://github.com/sclorg/httpd-ex.git --name=web
oc expose service/web

Once the build completes, the deployment rolls out on a worker:

oc get build -n demo
oc get pods -n demo -o wide

Build web-3 is the one that ran after the registry permission fix:

NAME    TYPE     FROM          STATUS                               STARTED          DURATION
web-1   Source   Git@658aff0   Failed (PushImageToRegistryFailed)   12 minutes ago   37s
web-2   Source   Git@658aff0   Failed (GenericBuildFailed)          6 minutes ago    20s
web-3   Source   Git@658aff0   Complete                             14 seconds ago   14s
NAME                   READY   STATUS      RESTARTS   AGE     IP            NODE       NOMINATED NODE   READINESS GATES
web-3-build            0/1     Completed   0          17s     10.130.0.15   worker01   <none>           <none>
web-7d9967f78c-q6gzs   1/1     Running     0          5s      10.128.2.8    worker02   <none>           <none>

The route is reachable from the host:

export CLUSTER_DOMAIN="ocp4.example.com"
curl -s http://web-demo.apps.${CLUSTER_DOMAIN}/ | grep -o "<title>[^<]*"

That title comes from the sample app itself:

<title>Welcome to OpenShift

The Topology view in the console shows the same deployment, with the route link on the top right of the node:

For a lighter OpenShift on a laptop, OpenShift Local (CRC) runs a single node with far less RAM. The community build of this same stack is covered in our OKD on Fedora CoreOS guide. To reach the internal registry from outside the cluster, expose it with a route.

Errors we hit and how we fixed them

Every error below came from this lab run. None were staged.

“Host timed out when pulling the configuration files”

master02 and master03 wrote RHCOS to disk, rebooted, and the installer then reported:

level=warning msg=Host master02: updated status from installing-in-progress to installing-pending-user-action (Host timed out when pulling the configuration files. Verify in the host console that the host boots from the OpenShift installation disk (vda, /dev/disk/by-path/pci-0000:04:00.0) and has network access to the cluster API. The installation will resume after the host successfully boots and can access the cluster API)
level=info msg=cluster has stopped installing... working to recover installation

The VMs were not booting the wrong disk. They were off. We had created them with virt-install --cdrom, which starts the install-phase VM with on_reboot=destroy. virsh dumpxml showed it on the node that had not rebooted yet:

  <on_reboot>destroy</on_reboot>

Starting the stopped VMs with sudo virsh start master02 let the install recover by itself. The rendezvous host and the worker then needed the same nudge when they rebooted. The permanent fix is the --import plus CD-ROM disk form used in this guide, which sets on_reboot=restart from the start. worker02 was created that way and came back from its reboot without help.

“Host couldn’t synchronize with any NTP server”

All four hosts reported this right after booting the ISO, and every host also flagged “No connectivity to the majority of hosts in the cluster”. Both cleared within about a minute once the nodes found each other and chronyd synced. If it persists, the nodes cannot reach UDP 123 on the internet. Point them at an internal server with additionalNTPSources in agent-config.yaml, or after install with a chrony MachineConfig.

“received unexpected HTTP status: 500 Internal Server Error” on build push

The first S2I build failed at the push step:

error: build error: Failed to push image: trying to reuse blob sha256:1587de7096c53a0b9fd09d99828507d9d9217e2b7edf597bc375022a17d2b1b9 at destination: checking whether a blob sha256:1587de7096c53a0b9fd09d99828507d9d9217e2b7edf597bc375022a17d2b1b9 exists in image-registry.openshift-image-registry.svc:5000/demo/web: received unexpected HTTP status: 500 Internal Server Error

The registry log had the real cause, permission denied on the NFS path:

oc logs -n openshift-image-registry deploy/image-registry | grep -i denied

One of the matching lines, trimmed to the error detail:

err.detail="filesystem: stat /registry/docker/registry/v2/blobs/sha256/22/224633db4e3d3cb4ff79a3a57fa245728ba7db41f2fc2e8c9cf051adc6e13e15/data: permission denied"

The registry process runs as uid=1000320000 gid=0(root), and the export directory was nobody:nobody with mode 0770, so the pod could not even list it. Changing the owner to root:root with 0770 fixed it without opening the share to everyone. A build that started in the same minute as the change still failed with the same error on a pull, so retry the build after fixing the permissions.

No route to host after restarting the libvirt network

Restarting the network with virsh net-destroy ocp4 and net-start (to put it into the firewalld zone, for example) detaches the tap device of every running VM from the bridge. The nodes keep running but drop off the network:

ssh: connect to host 192.168.100.11 port 22: No route to host

Reattach each VM interface to the bridge, or restart the VMs. In our run the first ssh after the reattach still failed; it worked after the next firewall-cmd --reload a few seconds later, so give it a moment (and reload firewalld if it runs) before you conclude the reattach did not work:

for d in master01 master02 master03 worker01 worker02; do
  for v in $(sudo virsh domiflist "$d" | awk '/ocp4/{print $1}'); do
    sudo ip link set "$v" master virbr-ocp4
  done
done

Our cluster went back to all nodes Ready and the test route answered again. The cleaner habit is to make firewalld and NFS changes before you create the VMs, so the network never needs a restart while the cluster is running.