Customize Qcow2 and Raw VM Images with virt-customize

virt-customize edits a VM disk image while the VM is switched off. It boots a small appliance of its own, mounts the guest’s filesystems and installs packages, sets passwords, injects SSH keys and runs commands inside the image. On an Ubuntu host it fails on your first run with a one-line supermin error, and the cause is a single permission: Ubuntu ships the host kernel in /boot readable by root only.

Original content from computingforgeeks.com - post 23781

This guide fixes that failure and the slow fallback behind it, then runs each customization you are likely to need and the gotchas that cost us a boot. It ends with a golden image built from one commands file and the SSH login that proves it works, plus SELinux on Rocky Linux, raw versus qcow2 and virt-sysprep. Everything below ran in October 2026 with virt-customize 1.54 from Ubuntu 26.04, against Ubuntu 24.04 and Rocky Linux 10 guest images.

Install virt-customize on Ubuntu

virt-customize ships in the libguestfs-tools package. On Ubuntu 26.04 that package pulls in QEMU, supermin and the whole virt-* toolkit, plus the libvirt client libraries but not the libvirt daemon. You don’t need the daemon either, because libguestfs already defaults to its direct backend here, so the old export LIBGUESTFS_BACKEND=direct line from older guides changes nothing. If the same host will also run the VMs you build, set up KVM and libvirt on Ubuntu 26.04 separately.

Your build host needs less than you would think, but it does need hardware virtualization and some disk:

• An x86_64 host with /dev/kvm (nested KVM works, ours was a 2 vCPU, 4 GB VM)
• About 370 MB for the packages
• 1.7 GB per user for the appliance cache in /var/tmp
• Space for images: our working directory peaked at 7.9 GB
• Outbound internet for --install and --update

A 4 GB VM is a floor for following along. Image work is mostly disk I/O and appliance boots, so faster storage helps more than extra cores. Install the package.

sudo apt-get update
sudo apt-get install -y libguestfs-tools

It’s a big dependency tree. On our host apt added 186 packages, including qemu-system-x86, supermin, guestfish and guestmount:

2 upgraded, 186 newly installed, 0 to remove and 158 not upgraded.
Need to get 85.6 MB of archives.
After this operation, 367 MB of additional disk space will be used.

Check which versions you got, because the options you’ll read about online follow upstream releases:

virt-customize --version; guestfish --version

Ubuntu 26.04 reports these two.

virt-customize 1.54.0
guestfish 1.58.1

That is one stable series behind upstream, which lists libguestfs 1.60 and guestfs-tools 1.56 as current stable on download.libguestfs.org. Nothing in this guide needs the newer series. When an online example uses an option your build rejects, check virt-customize --help on your own host before you blame the command.

Download a cloud image and verify it

Start from the vendor’s official cloud image and always customize a copy, so a failed run never costs you the download. We used the Ubuntu 24.04 (noble) image and the Rocky Linux 10 GenericCloud image, and fetching both with their checksum files took just over three minutes:

mkdir -p ~/images && cd ~/images && wget -q https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img https://cloud-images.ubuntu.com/noble/current/SHA256SUMS https://dl.rockylinux.org/pub/rocky/10/images/x86_64/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2 https://dl.rockylinux.org/pub/rocky/10/images/x86_64/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2.CHECKSUM

Verify both images before anything else. A golden image built on a corrupted or tampered download carries the problem into every VM you clone from it:

cd ~/images && sha256sum -c SHA256SUMS --ignore-missing && cat Rocky-10-GenericCloud-Base.latest.x86_64.qcow2.CHECKSUM && sha256sum -c Rocky-10-GenericCloud-Base.latest.x86_64.qcow2.CHECKSUM 2>&1 | tail -2

Each image must print OK, and anything else means the file is incomplete or not the one the vendor published, so download it again:

noble-server-cloudimg-amd64.img: OK
# Rocky-10-GenericCloud-Base.latest.x86_64.qcow2: 544997376 bytes
SHA256 (Rocky-10-GenericCloud-Base.latest.x86_64.qcow2) = 9fc9e9ff16888bb68ac39b0392e25c9c92684d50c85f1cce6ab549363bbc4b48
Rocky-10-GenericCloud-Base.latest.x86_64.qcow2: OK

Despite the .img name, the Ubuntu file is a qcow2 image. Its virtual size is small, which matters later when packages fill the root filesystem:

qemu-img info noble-server-cloudimg-amd64.img

Look for the virtual size, 3.5 GiB, against 597 MiB on disk:

image: noble-server-cloudimg-amd64.img
file format: qcow2
virtual size: 3.5 GiB (3758096384 bytes)
disk size: 597 MiB
cluster_size: 65536

Make a working copy. Every single-option example below runs against ubuntu-test.qcow2, and the download stays pristine for the golden build later. Our qemu-img guide covers creating, resizing and converting images in more depth.

cd ~/images && cp noble-server-cloudimg-amd64.img ubuntu-test.qcow2

Fix the first-run errors on Ubuntu hosts

Two things break a first run as a normal user on Ubuntu. The host kernel is unreadable, which is fatal, and your user isn’t in the kvm group, which only makes every run five times slower. Fix both before you spend time on an image.

virt-customize: error: libguestfs error: /usr/bin/supermin exited with error status 1

Our first run, a simple hostname change, died in half a second with this:

cd ~/images && time virt-customize -a ubuntu-test.qcow2 --hostname test01

You get advice to enable debugging and nothing about the cause:

[   0.0] Examining the guest ...
virt-customize: error: libguestfs error: /usr/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

Run libguestfs-test-tool instead. It builds the same appliance with full debug output and needs no image, so it is the fastest way to see what supermin tripped on:

libguestfs-test-tool

Its last lines name the file supermin couldn’t read.

supermin builds the appliance from the host’s own kernel and packages, so it copies the newest /boot/vmlinuz-* that has modules installed into /var/tmp/.guestfs-1000. Ubuntu installs that file as mode 0600, owned by root, so the copy fails for anyone else. On a host with one kernel, make it world-readable with dpkg-statoverride, which records the mode in dpkg so a reinstall of the same kernel package doesn’t quietly undo it:

sudo dpkg-statoverride --update --add root root 0644 /boot/vmlinuz-$(uname -r) && ls -l /boot/vmlinuz-$(uname -r) && dpkg-statoverride --list | grep vmlinuz

You should now see -rw-r--r-- on the file and the override in dpkg’s list.

-rw-r--r-- 1 root root 17283464 Jun 18 17:19 /boot/vmlinuz-7.0.0-27-generic
root root 644 /boot/vmlinuz-7.0.0-27-generic

The override covers that one file, and supermin doesn’t follow the running kernel. We installed a second kernel package without rebooting, and the next cold run picked /boot/vmlinuz-7.0.0-38-generic and failed with the same Permission denied, because a new kernel package lands as 0600 again. Re-running the one-liner doesn’t help, since $(uname -r) still names the old kernel, which already has an override:

dpkg-statoverride: error: an override for '/boot/vmlinuz-7.0.0-27-generic' already exists; aborting

Cover every installed kernel that has no override yet. The loop skips files that already have one, so it is safe to run after every kernel install, and it exits 0 when there’s nothing to do:

for k in /boot/vmlinuz-*; do dpkg-statoverride --list "$k" >/dev/null || sudo dpkg-statoverride --update --add root root 0644 "$k"; done

The loop itself prints nothing. Afterwards dpkg-statoverride --list | grep vmlinuz lists an override for both kernels, and both files read -rw-r--r--.

root root 644 /boot/vmlinuz-7.0.0-27-generic
root root 644 /boot/vmlinuz-7.0.0-38-generic

With that in place, libguestfs-test-tool built the appliance from the new kernel and finished with TEST FINISHED OK. A second run of the loop changed nothing. The other way out is sudo virt-customize, which worked first time (18.4 s on a cold cache), but it runs QEMU and the appliance as root and keeps a second 1.7 GB cache in /var/tmp/.guestfs-0. We’d use sudo for a one-off and the override on any host that builds images regularly.

libguestfs: warning: current user is not a member of the KVM group

With the kernel readable, the same run succeeded but took 1 minute 17 seconds and printed this warning:

libguestfs: warning: current user is not a member of the KVM group (group ID 991). This user cannot access /dev/kvm, so libguestfs may run very slowly. It is recommended that you 'chmod 0666 /dev/kvm' or add the current user to the KVM group (you might need to log out and log in again).

/dev/kvm is crw-rw---- root kvm, so without the group libguestfs falls back to software emulation (TCG). Skip the chmod 0666 it suggests, because that opens KVM to every account on the host. Add your user to the group instead.

sudo usermod -aG kvm $USER && getent group kvm

Your user should appear at the end of the line.

kvm:x:991:josphat

Group changes only apply to new logins, so log out and back in, then confirm id lists 991(kvm). These are the wall times for the same one-option run under each condition:

Condition	Wall time	Spent examining the guest
Not in kvm group (TCG)	1m17.6s	63.5 s
KVM, cold appliance cache	16.8 s	15.4 s
KVM, cached appliance	13.9 s	12.5 s
sudo, cold root cache	18.4 s	17.0 s

Look at the last column. Almost the whole run is appliance boot, and the hostname change itself takes about a second, so ten separate virt-customize calls cost ten boots. That is why you should put every option for one image into a single run, and later into a commands file. The first run builds the appliance into /var/tmp/.guestfs-1000 (kernel, initrd and a root image, 1.7 GB), and later runs reuse it. A last check that the whole stack is healthy:

libguestfs-test-tool

After both fixes it finished in 11.5 s with the direct backend, a 1280 MB appliance and KVM acceleration:

guestfs_get_backend: direct
guestfs_get_memsize: 1280
guestfs_get_smp: 1
...
    -machine q35,accel=kvm:tcg,graphics=off \
...
===== TEST FINISHED OK =====

Set the hostname, timezone and root password

Identity settings take one run. Start from a fresh copy if you’ve already experimented on ubuntu-test.qcow2, so your output matches ours.

cd ~/images && rm -f ubuntu-test.qcow2 && cp noble-server-cloudimg-amd64.img ubuntu-test.qcow2

If you’re not sure what a set of options will do, add --dry-run first: it runs every step against the image and then throws the changes away. Our dry run took 13.9 s and the image’s md5sum (0a01cf41ac7b) was identical before and after:

virt-customize -a ubuntu-test.qcow2 --dry-run --hostname dry01 --timezone Africa/Nairobi

Now the real run. The password here is a lab example, so pick your own. Whatever you type also lands in your shell history:

cd ~/images && time virt-customize -a ubuntu-test.qcow2 --root-password password:LabOnly-Example-2026 --hostname web01 --timezone Africa/Nairobi

Each option prints its own step, and the random seed warning shows up on every Ubuntu guest. It’s harmless and does not appear for Rocky:

[   0.0] Examining the guest ...
[  10.4] Setting a random seed
virt-customize: warning: random seed could not be set for this type of guest
[  10.4] Setting the machine ID in /etc/machine-id
[  10.4] Setting the hostname: web01
[  11.5] Setting the timezone: Africa/Nairobi
[  11.6] Setting passwords
[  12.5] SELinux relabelling
[  12.6] Finishing off

real	0m12.615s
user	0m6.841s
sys	0m1.546s

Notice the order. The random seed and machine ID are set before anything else, and passwords are always applied last. Everything else, hostname and timezone included, runs in the order you give it: in a second test with --root-password first, then --touch, --timezone and --hostname, the touch ran first and the password last. The machine ID line only appears when /etc/machine-id is empty, as it is in a fresh download. Read the result back with the read-only tools, which open the image without changing it:

cd ~/images && virt-cat -a ubuntu-test.qcow2 /etc/hostname && virt-ls -l -a ubuntu-test.qcow2 /etc/localtime && virt-cat -a ubuntu-test.qcow2 /etc/shadow | grep '^root:' | cut -c1-20

You should see the new hostname, /etc/localtime resolving to the Nairobi zone file, and the root hash starts with $y$, which means yescrypt, the guest’s own default:

web01
-rw-r--r-- 1 0 0 265 Jul 17 12:49 /sysroot/usr/share/zoneinfo/Africa/Nairobi
root:$y$j9T$uYZJps3A

Install, update and remove packages

--install runs the guest’s own package manager inside the appliance, and the appliance has network access by default, so you don’t need to set anything up. Three small packages took 38.5 s.

cd ~/images && time virt-customize -a ubuntu-test.qcow2 --install qemu-guest-agent,htop,tree

Most of that is apt-get update, which virt-customize runs before every install.

[   0.0] Examining the guest ...
[  13.0] Setting a random seed
virt-customize: warning: random seed could not be set for this type of guest
[  13.0] Installing packages: qemu-guest-agent htop tree
[  38.0] SELinux relabelling
[  38.5] Finishing off

real	0m38.526s

We saw exactly what it runs when an install failed with --no-network: export DEBIAN_FRONTEND=noninteractive, then apt-get update and install with -q -y -o Dpkg::Options::=--force-confnew. That last flag matters, because it means a package’s new config file silently replaces one you edited earlier in the same build. Install packages first and edit configs after. --update runs a full upgrade the same way and took 43.7 s on the noble image, pulling newer systemd, openssh-server, sudo and tzdata builds:

cd ~/images && time virt-customize -a ubuntu-test.qcow2 --update

systemctl enable qemu-guest-agent does nothing

Old guides add --run-command 'systemctl enable qemu-guest-agent' after the install. On Ubuntu 24.04 it runs without an error and changes nothing, along with the --write and --append-line options we ran in the same pass:

cd ~/images && time virt-customize -a ubuntu-test.qcow2 --run-command 'systemctl enable qemu-guest-agent' --write /etc/motd:'Built with virt-customize on kvm01' --append-line /etc/hosts:'10.0.1.50 web01.example.com web01'

No symlink appeared in multi-user.target.wants, and systemctl is-enabled inside the image returned static. The unit file explains it:

virt-cat -a ubuntu-test.qcow2 /usr/lib/systemd/system/qemu-guest-agent.service

Its [Install] section is empty, and the unit binds to the virtio-serial device.

[Unit]
Description=QEMU Guest Agent
BindsTo=dev-virtio\x2dports-org.qemu.guest_agent.0.device
After=dev-virtio\x2dports-org.qemu.guest_agent.0.device

[Service]
ExecStart=-/usr/sbin/qemu-ga
Restart=always
RestartSec=0

[Install]

udev starts the agent when the VM has an org.qemu.guest_agent.0 channel, and in our boot test it was active without any enable step. So installing the package is enough. Give the VM the channel and leave systemctl out. Note that --write does not add a trailing newline, which is fine for /etc/motd but worth knowing for files a parser reads.

AppArmor parser error after –uninstall snapd

Dropping snapd from a server template saves about 130 MB, and --uninstall looks like the obvious tool. It ran in 15.9 s, but on Debian and Ubuntu it removes the package without purging it:

cd ~/images && time virt-customize -a ubuntu-test.qcow2 --uninstall snapd

Afterwards dpkg showed snapd as deinstall ok config-files, and the leftover config includes an AppArmor profile that points at files snapd no longer has. When we booted an image built this way, apparmor.service failed and the system came up degraded:

apparmor.systemd[568]: AppArmor parser error for /etc/apparmor.d in profile /etc/apparmor.d/usr.lib.snapd.snap-confine.real at line 13: Could not open '/var/lib/snapd/apparmor/snap-confine'
apparmor.systemd[419]: Error: At least one profile failed to load

Purge it with --run-command 'apt-get purge -y snapd' followed by apt-get autoremove -y --purge. The golden image file further down does exactly that, and its VM boots to running.

Copy files into the image

--copy-in copies a local file or directory into an existing guest directory, and --upload copies one file to a new path. Create a small payload to push in.

mkdir -p ~/images/payload/app && echo 'export HISTTIMEFORMAT="%F %T "' > ~/images/payload/history.sh && echo 'APP_ENV=lab' > ~/images/payload/app/app.env && echo 'echo app ok' > ~/images/payload/app/run.sh

Then create the target directory, copy the app directory, upload the profile script and make the app’s script executable, all in one run:

cd ~/images && virt-customize -a ubuntu-test.qcow2 --mkdir /opt/demo --copy-in payload/app:/opt/demo --upload payload/history.sh:/etc/profile.d/history-time.sh --chmod 0755:/opt/demo/app/run.sh

Before you move on, check the owners. virt-ls --uids prints numeric ownership:

cd ~/images && virt-ls --csv --uids -lR -a ubuntu-test.qcow2 /opt/demo; virt-ls --csv --uids -lR -a ubuntu-test.qcow2 /etc/profile.d | grep history

Both --copy-in and --upload kept the host user’s uid, gid and mode.

d,0755,4096,   0,   0,/opt/demo
d,0775,4096,1000,1000,/opt/demo/app
-,0664,12,1000,1000,/opt/demo/app/app.env
-,0755,12,1000,1000,/opt/demo/app/run.sh
-,0664,31,1000,1000,/etc/profile.d/history-time.sh

uid 1000 on the build host is you, but inside an Ubuntu guest it becomes the first normal user cloud-init creates. A file in /etc/profile.d owned by that user lets them change what every login shell runs, including root’s. Fix the owner and mode in the same build, right after the copy.

cd ~/images && virt-customize -a ubuntu-test.qcow2 --chown 0:0:/etc/profile.d/history-time.sh --chmod 0644:/etc/profile.d/history-time.sh --run-command 'chown -R root:root /opt/demo/app'

Running the virt-ls check again shows everything as 0, 0, and the profile script as 0644. --chown takes one path per option, so for a whole tree a chown -R inside --run-command is shorter.

Add a user and inject an SSH key

--ssh-inject writes a public key into a user’s authorized_keys with the right modes, but only for a user that already exists in the image. That trips up anyone who tries it on a cloud image’s default account.

ssh-inject: the user ubuntu does not exist on the guest

Injecting a key for ubuntu fails, because cloud-init creates that account at first boot and it isn’t in the image yet:

cd ~/images && virt-customize -a ubuntu-test.qcow2 --ssh-inject ubuntu:file:/home/josphat/.ssh/id_ed25519.pub

virt-customize stops with exit code 1.

[  11.0] SSH key inject: ubuntu
virt-customize: error: ssh-inject: the user ubuntu does not exist on the guest

Create your own admin user first, then inject the key and add a sudoers drop-in. Replace josphat with your user and the key path with your own public key. If the build host has no key yet, create one with ssh-keygen -t ed25519 first:

cd ~/images && time virt-customize -a ubuntu-test.qcow2 --run-command 'useradd -m -s /bin/bash -G sudo josphat' --ssh-inject josphat:file:/home/josphat/.ssh/id_ed25519.pub --write /etc/sudoers.d/90-josphat:'josphat ALL=(ALL) NOPASSWD:ALL' --chmod 0440:/etc/sudoers.d/90-josphat

Check the result without booting.

cd ~/images && virt-ls --csv --uids -lR -a ubuntu-test.qcow2 /home/josphat/.ssh && virt-cat -a ubuntu-test.qcow2 /home/josphat/.ssh/authorized_keys | cut -d' ' -f1,3

A correct injection gives .ssh mode 0700 and authorized_keys mode 0600, both owned by the new user:

d,0700,4096,1000,1000,/home/josphat/.ssh
-,0600,93,1000,1000,/home/josphat/.ssh/authorized_keys
ssh-ed25519 josphat@lab

If you deploy through cloud-init anyway, the key can go in user-data instead, as in our cloud-init and virt-install walkthrough. Baking it in with virt-customize is the better fit when the image boots somewhere with no datasource at all.

Run a command on first boot

Some things cannot happen offline, such as anything that needs the VM’s real network, its final hostname or a running service. --firstboot-command queues a command to run once when a VM made from the image first boots:

cd ~/images && virt-customize -a ubuntu-test.qcow2 --firstboot-command 'date > /var/log/firstboot-done.txt'

It installs /usr/lib/virt-sysprep/firstboot.sh, a numbered script under /usr/lib/virt-sysprep/scripts/ and a guestfs-firstboot.service unit enabled in multi-user.target.wants. On our test VM the journal showed Running /usr/lib/virt-sysprep/scripts/5000-0001-date----var-log-firstboot-done-txt, and the file held the boot time. Use it for per-VM work that has to wait for a real boot, and keep anything every boot needs in a normal systemd unit.

Make room in the root filesystem

The noble image’s root filesystem is 2.4G with about 515M free, so a few large packages fill it. When that happens, virt-customize can’t fix it from the inside, because the tools that grow a filesystem need free space too.

fallocate failed: No space left on device

We filled the disk on purpose with an 800M file. The command failed, but it left a 647 MB partial file behind and the root filesystem at 100%:

cd ~/images && virt-customize -a ubuntu-test.qcow2 --run-command 'fallocate -l 800M /var/tmp/filler'

Before the real error, virt-customize printed about 385 lines of apt output from earlier runs on this image:

[  11.1] Running: fallocate -l 800M /var/tmp/filler
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Get:2 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
...
Removing snapd (2.76.3+ubuntu24.04) ...
Processing triggers for dbus (1.14.10-4ubuntu4.1) ...
Processing triggers for man-db (2.12.0-4build2) ...
fallocate: fallocate failed: No space left on device
virt-customize: error: fallocate -l 800M /var/tmp/filler: command exited with an error

That replay comes from /tmp/builder.log inside the image. virt-customize appends every command’s output to it, and it had grown to 391 lines and 30 KB. When a later --run-command fails, it dumps the whole file, so read failures from the bottom up. The same file also tells anyone who gets your image exactly how you built it, which is why the golden build uses --no-logfile to scrub it.

Growing the virtual disk alone is not enough here, because growpart needs a temporary directory on the full filesystem. Our first attempt failed with mkdir: cannot create directory '/tmp/growpart.395': No space left on device and FAILED: failed to make temp dir. Options apply in command-line order, so resize the disk, then delete what filled it before you grow the partition:

cd ~/images && qemu-img resize ubuntu-test.qcow2 +5G

Now free the space and grow partition 1 and its ext4 filesystem in one pass.

cd ~/images && time virt-customize -a ubuntu-test.qcow2 --delete /var/tmp/filler --delete /tmp/builder.log --run-command 'growpart /dev/sda 1 && resize2fs /dev/sda1'; virt-df -h -a ubuntu-test.qcow2 | head -2

In 11.1 s the root filesystem went from 2.4G to 7.2G with 5.4G free.

[  10.4] Deleting: /var/tmp/filler
[  10.4] Deleting: /tmp/builder.log
[  10.4] Running: growpart /dev/sda 1 && resize2fs /dev/sda1
[  11.0] SELinux relabelling
[  11.0] Finishing off

real	0m11.088s
user	0m5.051s
sys	0m1.064s
Filesystem                                Size       Used  Available  Use%
ubuntu-test.qcow2:/dev/sda1               7.2G       1.7G       5.4G   25%

virt-resize is the other route. virt-resize --expand /dev/sda1 into a new 10G qcow2 worked in 46.8 s. On this GPT cloud image, though, it copied partitions 14, 15 and 16 first and reported Expanding /dev/sda1 (now /dev/sda4), and the output was 1.76 GiB on disk against 1.3 GiB. We didn’t boot that copy. For cloud images we’d use the growpart route, because it keeps the partition layout the image’s bootloader and fstab expect.

Customize Rocky Linux 10 with SELinux

On an SELinux guest every file virt-customize writes needs the right label, or a confined service cannot read it after boot. The Rocky Linux 10 GenericCloud image ships with SELINUX=enforcing. The same options work as on Ubuntu, with dnf doing the installs:

cd ~/images && cp Rocky-10-GenericCloud-Base.latest.x86_64.qcow2 rocky-test.qcow2 && time virt-customize -a rocky-test.qcow2 --hostname rocky01 --timezone Africa/Nairobi --install vim-enhanced,tmux,qemu-guest-agent --selinux-relabel

Rocky finished in 28.7 s and printed a SELinux relabelling step.

[   0.0] Examining the guest ...
[  12.1] Setting a random seed
[  12.1] Setting the machine ID in /etc/machine-id
[  12.1] Setting the hostname: rocky01
[  12.1] Setting the timezone: Africa/Nairobi
[  12.2] Installing packages: vim-enhanced tmux qemu-guest-agent
[  28.1] SELinux relabelling
[  28.6] Finishing off

The --selinux-relabel flag from older guides no longer does anything. The help text calls it a compatibility option, and relabelling is now automatic, with --no-selinux-relabel to turn it off:

  --no-selinux-relabel                Do not relabel files with correct SELinux labels
--
  --selinux-relabel                   Compatibility option doing nothing

On an Ubuntu build host there’s a catch. The man page says virt-customize relabels immediately, and if that fails it touches /.autorelabel so the guest relabels on next boot. Ubuntu’s appliance has no setfiles, so it always takes the fallback. Ask the appliance directly.

guestfish -a /dev/null run : feature-available selinuxrelabel : feature-available xfs : feature-available ldm

You want the first line, the answer for selinuxrelabel.

false
true
true

Reading the labels confirms it. Files virt-customize wrote have no SELinux label at all, an untouched file keeps its label, and the relabel flag is waiting at the root:

cd ~/images && for f in /etc/hostname /usr/bin/tmux /etc/passwd; do echo "== $f"; guestfish --ro -a rocky-test.qcow2 -i lgetxattrs $f | tr -d "\000"; done; echo "== /.autorelabel exists:"; guestfish --ro -a rocky-test.qcow2 -i exists /.autorelabel

Only /etc/passwd carries a label.

== /etc/hostname
== /usr/bin/tmux
== /etc/passwd
[0] = {
  attrname: security.selinux
  attrval: system_u:object_r:passwd_file_t:s0\x00
}
== /.autorelabel exists:
true

The image still works. On first boot selinux-autorelabel ran fixfiles -T 0 restore and rebooted, so the VM reached its login prompt after 59.5 s and two boots instead of 28.3 s and one. For a handful of VMs that is a one-time cost you can accept. It hurts when every clone pays it, because the flag ships inside the template.

We also tested an unofficial workaround: install policycoreutils on the host, add it to the appliance package list and clear the cache so supermin rebuilds the appliance with setfiles:

sudo apt-get install -y policycoreutils
echo policycoreutils | sudo tee /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-selinux
rm -rf /var/tmp/.guestfs-$(id -u)

After that, feature-available selinuxrelabel returned true. The same Rocky build took 36.8 s, of which relabelling was about 2.9 s, and the files got hostname_etc_t and screen_exec_t labels with no /.autorelabel. The VM booted to a login prompt in 28.3 s. It adds a file to a directory the libguestfs package manages, so it is not a supported setup, and we only consider it worth the risk on a host that builds RHEL-family templates often. We reverted it after testing:

sudo rm -f /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-selinux
sudo apt-get purge -y policycoreutils
sudo apt-get autoremove -y --purge
rm -rf /var/tmp/.guestfs-$(id -u)

Clearing the cache again matters, because the cached appliance still contains setfiles until supermin rebuilds it.

One more trap comes from the old version of this article, which ran --run-command 'systemctl disable firewalld'. Rocky 10 GenericCloud has no firewalld, so systemctl printed Failed to disable unit: Unit firewalld.service does not exist into /tmp/builder.log. virt-customize still exited 0. A clean exit does not prove a --run-command did what you meant, so check the result with virt-cat or virt-ls. --edit /etc/selinux/config:'s/^SELINUX=.*/SELINUX=permissive/' does work if you really need permissive, but on a template we would keep enforcing and fix the labels instead.

Build a golden image from a commands file

Once you know which options you need, put them in a file. --commands-from-file takes the long option names without the leading dashes, one per line, and runs the lot in a single appliance boot. The file goes into version control, and the build becomes one command anyone can repeat. This is the file we built and boot-tested, verbatim:

vim ~/images/golden-ubuntu.cmds

Change josphat to your admin user everywhere, including the copy-in path under /home/josphat/images, point ssh-inject at your own public key and pick your own root password:

# Golden Ubuntu 24.04 template: virt-customize --commands-from-file
hostname ubuntu-golden
timezone Africa/Nairobi
root-password password:LabOnly-Example-2026

# Packages: refresh, add tools, purge snapd (uninstall leaves its AppArmor profile behind)
update
install qemu-guest-agent,htop,tree,curl,vim
run-command apt-get purge -y snapd
run-command apt-get autoremove -y --purge

# DHCP on any en* NIC when no cloud-init datasource is present
write /etc/netplan/60-dhcp-all.yaml:network: {version: 2, ethernets: {all-en: {match: {name: "en*"}, dhcp4: true}}}
chmod 0600:/etc/netplan/60-dhcp-all.yaml

# Admin user with the build host's SSH key and passwordless sudo
run-command useradd -m -s /bin/bash -G sudo josphat
ssh-inject josphat:file:/home/josphat/.ssh/id_ed25519.pub
append-line /etc/sudoers.d/90-josphat:josphat ALL=(ALL) NOPASSWD:ALL
chmod 0440:/etc/sudoers.d/90-josphat

# Files and settings
write /etc/motd:Golden image built with virt-customize
append-line /etc/hosts:10.0.1.50 web01.example.com web01
mkdir /opt/demo
copy-in /home/josphat/images/payload/app:/opt/demo
run-command chown -R root:root /opt/demo/app

# Runs once, on the first boot of every VM made from this image
firstboot-command date > /var/log/firstboot-done.txt

The netplan line is written in YAML flow style because write takes a single line, and it gets mode 0600 so only root can read it. Build from a fresh copy of the download, never from an image you’ve already experimented on:

cd ~/images && rm -f ubuntu-golden.qcow2 && cp noble-server-cloudimg-amd64.img ubuntu-golden.qcow2 && time virt-customize -a ubuntu-golden.qcow2 --commands-from-file golden-ubuntu.cmds --no-logfile

Our build took 1 minute 15 seconds.

Read the timestamps to see where the time goes: the package update took 46.6 s of the 75, and everything after the installs finished in about 10 seconds. Setting passwords ran near the end even though it is the fourth line of the file, and --no-logfile added the Scrubbing the log file step. The golden image came out at 1.25 GiB on disk against 597 MiB for the download.

The golden image boots but SSH never answers

Our first version of this file used uninstall snapd and had no netplan file. That image booted to a login prompt on the serial console, but SSH never came up, and the console showed [FAILED] Failed to start systemd-networkd-w… Wait for Network to be Configured. after a two-minute wait. Inside, cloud-init status --long reported disabled-by-generator, /etc/netplan was empty and ens3 was off unmanaged.

The Ubuntu cloud image has no network config of its own. cloud-init writes one from a datasource, and a plain QEMU boot has none, so cloud-init disables itself and the NIC is never configured. That’s why the final file ships /etc/netplan/60-dhcp-all.yaml. On a platform that does provide a datasource, such as OpenStack or a Proxmox template with a cloud-init drive, cloud-init writes its own netplan. We didn’t test the two files together, so drop the netplan lines if every VM from this image gets a datasource.

Test the image over SSH

A customization is not proven until a VM boots from the image and you can log in with the key you injected. This is the script we used. It boots the image with -snapshot, so nothing is written back to the golden file, forwards host port 2222 to the guest’s SSH and adds the guest agent channel:

vim ~/boot-test-qemu.sh

Paste the script. It polls SSH every second for up to 300 tries and prints how long the login took:

#!/bin/bash
# usage: boot-test-qemu.sh IMAGE USER PORT
IMG=$1; U=$2; P=${3:-2222}
cd ~/images
rm -f /tmp/console-$P.log
s=$(date +%s.%N)
qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -smp 2 \
  -drive file=$IMG,if=virtio,format=qcow2 -snapshot \
  -nic user,model=virtio-net-pci,hostfwd=tcp:127.0.0.1:$P-:22 \
  -device virtio-serial -chardev socket,path=/tmp/qga-$P.sock,server=on,wait=off,id=qga0 \
  -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 \
  -display none -serial file:/tmp/console-$P.log -daemonize -pidfile /tmp/vm-$P.pid
echo "qemu rc=$? pid=$(cat /tmp/vm-$P.pid)"
for i in $(seq 1 300); do
  if ssh -q -o BatchMode=yes -o ConnectTimeout=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p $P $U@127.0.0.1 true 2>/dev/null; then
    e=$(date +%s.%N); echo "SSH OK after $(python3 -c "print(round($e-$s,1))") s"; exit 0
  fi
  sleep 1
done
echo "SSH NOT reachable after 300+ s"; exit 1

Run it against the golden image as the user you created. If it prints SSH NOT reachable, read /tmp/console-2222.log, which is where we found the network and AppArmor failures above:

bash ~/boot-test-qemu.sh ubuntu-golden.qcow2 josphat 2222

Ours printed SSH OK after 16.3 s, and later boots of its machine-ID-truncated and sparsified copies took 16.5 and 17.6 seconds. Then run one SSH command that checks everything the commands file promised:

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -p 2222 [email protected] 'hostnamectl hostname; timedatectl show -p Timezone --value; date; cat /var/log/firstboot-done.txt; cat /etc/motd; echo; dpkg -l qemu-guest-agent htop tree curl vim | tail -5; dpkg -l snapd 2>&1 | tail -1; sudo -n true && echo "sudo: passwordless OK"; systemctl is-system-running; systemctl is-active apparmor qemu-guest-agent ssh; networkctl --no-legend; ls -l /opt/demo/app; tail -1 /etc/hosts; cloud-init status; df -h / | tail -1; uname -r; grep PRETTY /etc/os-release'

Every line maps back to a line in the commands file.

You should see the hostname and timezone you set. The firstboot file holds a time 13 seconds before the date call, so it ran on this boot. All five packages are installed and snapd is un (purged). Passwordless sudo works, and the system state is running with AppArmor, the guest agent and SSH active. ens3 is routable configured thanks to the netplan file, /opt/demo/app is root-owned and cloud-init is disabled, as expected with no datasource. Root is at 75% of 2.4G, so grow the disk before you add anything large.

Failed to get “write” lock

Don’t run virt-customize on an image a VM has open. With the test VM still running, a write attempt failed with only libguestfs error: guestfs_launch failed., and the real reason showed up with LIBGUESTFS_DEBUG=1:

cd ~/images && LIBGUESTFS_DEBUG=1 virt-customize -a ubuntu-golden.qcow2 --touch /etc/lock-test 2>&1 | grep -i -E "lock|qemu-system-x86_64:|error" | head -8

QEMU’s image locking refused the write.

qemu-system-x86_64: -drive file=/home/josphat/images/ubuntu-golden.qcow2,cache=writeback,discard=unmap,id=hd0,if=none: Failed to get "write" lock
virt-customize: error: libguestfs error: guestfs_launch failed, see earlier error messages

That lock protects you: the man page warns that running virt-customize on a live VM can corrupt its disk. Read-only tools such as virt-cat still worked on the in-use image. Shut the test VM down before the next step, or every write and a second boot on port 2222 will fail:

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -p 2222 [email protected] 'sudo poweroff'

QEMU exits a few seconds later, and pgrep -fc '[q]emu-system-x86_64' printing 0 confirmed it on our host, which ran no other QEMU. If SSH is already gone, the script wrote QEMU’s PID to /tmp/vm-2222.pid, so kill $(cat /tmp/vm-2222.pid) stops it; we did not need that route in the lab.

Remove machine-specific data before cloning

A template needs a last pass that removes anything unique to one machine, because every VM cloned from it inherits it. Two steps cover it, virt-sysprep for the general cleanup and then a blank machine ID.

virt-sysprep does the broader cleanup. It comes from the same package, and the default run took 8.7 s on a copy of the golden image. Know what it removes before you run it on an image with a baked-in key:

• /home/*/.ssh, including the key from --ssh-inject (ssh-userdir)
• All SSH host keys (ssh-hostkeys)
• Log files, DHCP leases, bash history and temp files
• The machine ID, which its closing customize step then rewrites with a new random value

Deleting the host keys is what you want, since every clone should generate its own, and the golden image generated fresh keys on its first boot. Deleting the user’s key is usually not. Exclude that one operation and keep the rest:

cd ~/images && cp ubuntu-golden.qcow2 ubuntu-sysprep.qcow2 && virt-sysprep -a ubuntu-sysprep.qcow2 --operations defaults,-ssh-userdir | grep -c Performing; virt-ls -l -a ubuntu-sysprep.qcow2 /home/josphat/.ssh

Sysprep ran 37 operations and left authorized_keys in place.

37
total 12
drwx------ 2 1000 1000 4096 Oct 15 07:57 .
drwxr-x--- 3 1000 1000 4096 Oct 15 07:57 ..
-rw------- 1 1000 1000   93 Oct 15 07:57 authorized_keys

Now the machine ID. virt-customize writes a fresh /etc/machine-id into the cloud image, where the download had an empty one, and sysprep’s closing step writes another random one rather than blanking it. Every clone would share that ID, and systemd-networkd derives the DHCP client ID from it, so truncate it as the very last step, after sysprep:

virt-customize -a ubuntu-sysprep.qcow2 --truncate /etc/machine-id

We boot-tested the truncate on a copy of the golden image without sysprep: virt-cat /etc/machine-id | wc -c returned 0, the booted VM generated its own ID (38fbecb6...) and reached running with SSH up in 16.5 s. We didn’t boot an image that went through sysprep and then the truncate, so boot-test that combination before you clone from it.

Read an image without booting it

The read-only tools from the same package answer most questions about an image in seconds, and they work on an image a VM is using. virt-inspector identifies the OS and lists every installed package:

cd ~/images && virt-inspector -a ubuntu-golden.qcow2 > /tmp/golden-inspect.xml && grep -E "<(name|distro|product_name|major_version|minor_version|hostname|package_format|package_management|osinfo)>" /tmp/golden-inspect.xml | head -12 && grep -c "<application>" /tmp/golden-inspect.xml

virt-inspector took 5.4 s and found 665 packages.

    <name>linux</name>
    <distro>ubuntu</distro>
    <product_name>Ubuntu 24.04.5 LTS</product_name>
    <major_version>24</major_version>
    <minor_version>4</minor_version>
    <package_format>deb</package_format>
    <package_management>apt</package_management>
    <hostname>ubuntu-golden</hostname>
    <osinfo>ubuntu24.04</osinfo>
        <name>adduser</name>
        <name>apparmor</name>
        <name>apport</name>
665

You have already seen the others, because we used them to check each step:

• virt-df -h for free space per filesystem
• virt-ls -l, or --csv --uids -lR for numeric owners
• virt-cat to print a file
• virt-filesystems --long -h --all for partitions and labels

For interactive edits, guestfish opens a shell on the image, and you can also mount a VM disk on the host with guestmount.

Use raw images instead of qcow2

virt-customize treats raw and qcow2 the same, and it auto-detected the format in our tests. Keep templates in qcow2 and convert to raw only when the target platform asks for it, because raw costs more disk. Converting the golden image took 6.9 s:

cd ~/images && qemu-img convert -p -f qcow2 -O raw ubuntu-golden.qcow2 ubuntu-golden.raw

Identical content takes very different space depending on format and whether it was sparsified.

Image	Format	Size on disk
Ubuntu 24.04 cloud image (download)	qcow2, compressed	597 MiB
Golden image after customization	qcow2	1.25 GiB
Golden image converted	raw (3.5G apparent, sparse)	2.3G allocated
Golden image after virt-sparsify –compress	qcow2, compressed	660 MiB

The raw file reports 3.5G with ls but only 2.3G with du, because it’s sparse, while the same content as qcow2 takes 1.25 GiB. Two option-parsing quirks hit us on the raw file.

–format parameter must appear before -a parameter

virt-customize rejects -a ubuntu-golden.raw --format raw with virt-customize: error: --format parameter must appear before -a parameter, because the format applies to the disks that follow it. Put it first.

cd ~/images && virt-customize --format raw -a ubuntu-golden.raw --hostname raw01 --append-line /etc/motd:' (raw copy)'

virt-cat has the opposite problem. Its --format takes an optional argument, so virt-cat --format raw -a ... reads raw as a path and fails with libguestfs: error: download: download_stub: path must start with a / character. Write it with an equals sign.

virt-cat --format=raw -a ubuntu-golden.raw /etc/hostname

virt-sparsify: There may not be enough free space on /tmp

virt-sparsify --compress shrinks a customized qcow2 back toward the download size, which matters when you copy the template to many hosts. On Ubuntu 26.04 it stopped straight away, because /tmp is a tmpfs sized at half the RAM (2.0G on our 4 GB host) and the tool wanted up to 3.5G. Run non-interactively, its Press RETURN to continue prompt then failed with virt-sparsify: error: exception: End_of_file. Point TMPDIR at a disk-backed directory.

cd ~/images && time TMPDIR=/var/tmp virt-sparsify --compress ubuntu-golden.qcow2 ubuntu-golden-small.qcow2

virt-sparsify zeroed the free space in each filesystem and copied the result in 75.5 s.

[   0.0] Create overlay file in /var/tmp to protect source disk
[   0.1] Examine source disk
[   4.3] Fill free space in /dev/sda1 with zero
[   6.1] Fill free space in /dev/sda15 with zero
[   6.6] Fill free space in /dev/sda16 with zero
[   8.2] Copy to destination and make sparse
[  75.4] Sparsify operation completed with no errors.
virt-sparsify: Before deleting the old disk, carefully check that the target disk boots and works correctly.

The 1.25 GiB golden image came out at 660 MiB, and it booted with SSH up in 17.6 s. Take that last line of output seriously and boot-test the sparsified copy before you delete the original.

Production hardening for a golden image pipeline

Everything above works by hand. A pipeline that builds templates every week needs a few habits so that the next image is as trustworthy as the one you tested:

• Keep golden-ubuntu.cmds in Git, so every image maps to a commit
• Remove root-password from the file, or keep the repository private
• Run sha256sum -c on the source image every build
• Build from a fresh copy of the download every time
• Rebuild for each kernel or security update instead of patching an old template
• Run virt-sysprep with -ssh-userdir, then truncate /etc/machine-id and boot-test the result
• Boot-test every build with the SSH check before publishing it

The root password line puts a plain-text password in a file you’re about to commit, and a key plus passwordless sudo already gives you a way in, so we would drop it. Recording the source image build date (the noble current image was the 20260926 build) tells you which vendor image a template came from when a CVE lands. Building from a fresh copy matters because an image you’ve customized before carries its own history: our test image had a 391-line builder.log and stray files from earlier runs that a clean build never has.

Rebuilding beats layering --update on last month’s template. The update step was 46 of the 75 seconds in our build, so a full rebuild costs little and keeps the template traceable to one file and one source image. The same rule applies on the build host: a new kernel package brings back a 0600 /boot/vmlinuz the moment it’s installed, before any reboot, so run the per-kernel dpkg-statoverride loop in the same job that updates the host.

Size the build host for the appliance, not the guests. Each user that runs virt-customize keeps a 1.7 GB appliance cache in /var/tmp/.guestfs-<uid>. Run your builds as one dedicated user. A cached appliance cut a small run from 16.8 to 13.9 seconds. Our working directory peaked at 7.9 GB for two source images and their copies. /tmp on Ubuntu 26.04 is a RAM-sized tmpfs that can be too small for virt-sparsify, so set TMPDIR=/var/tmp in the job’s environment. If the pipeline also builds RHEL-family templates on an Ubuntu host, either accept the extra relabel boot on every clone’s first start or add the setfiles workaround to the host’s configuration management, so a package update does not quietly remove it.