Containers

Fedora Hummingbird Hands-On: What It Is and How to Install It

Fedora Hummingbird has been covered as a distroless operating system, with no package manager and no shell. The image you can pull today has both. It also boots Fedora 44’s kernel, ships with its automatic update timer switched off, and is labelled experimental by its own build files. None of that makes it bad. It does mean you should know what you are installing before it touches a machine you care about.

Original content from computingforgeeks.com - post 173098

We pulled the Hummingbird bootc image, verified its signature, built a qcow2 disk with image-builder, booted it on Proxmox VE, then customized it, pushed it to a registry, switched the host to it, upgraded it and rolled it back. A stock Fedora bootc 45 image went through the same steps on identical VMs as the baseline. Every number below comes from those two VMs and the Fedora build machine that produced their images.

Tested October 2026 on the Hummingbird bootc-os image built 5 October 2026, SELinux enforcing, against Fedora bootc 45 as the control.

What Fedora Hummingbird actually is

Project Hummingbird started as a catalog of minimal, hardened container images: Python, Go, Node.js, PostgreSQL, nginx and dozens more, built from Fedora Rawhide sources by Red Hat’s Konflux pipeline. Each image is signed, ships an SBOM, and runs as a non-root user by default. Fedora Hummingbird takes the same factory and points it at a whole operating system, delivered as a bootable container (bootc) image.

Two OS images are public today, quay.io/hummingbird-community/bootc-os and quay.io/hummingbird-community/os-minimal. Both sit in the community organization, which the project reserves for images without full support commitments. The bootc-os README opens with “Experimental: This image is a proof of concept and is not suitable for production use.” We compared the two images package by package and found identical lists, so this guide uses bootc-os.

The Quay tag list explains the “rolling” part. The repository had over 3,000 tags when we looked, and new builds landed several times a day:

Quay.io tag list for hummingbird-community/bootc-os showing builds from two hours ago and the 43 High security scan result

The scanner built into Quay flags 43 High findings on the per-architecture manifests. We ran Grype against the same build in Step 8 and got a similar answer.

Hummingbird vs Fedora bootc: what we measured

Both VMs had 2 vCPUs, 2 GB of RAM, a 20 GB disk and the same virtual hardware, including a serial port. Boot time and memory are the range of three cold boots each, read 60 seconds after SSH came up:

MeasurementHummingbird bootc-osFedora bootc 45
Container image size849 MB1.99 GB
Installed packages265540
KernelFedora 44 buildFedora 45 build
qcow2 from image-builder521 MB1.2 GB
Disk used after first boot920 MB2.0 GB
RAM used after boot411 to 416 MiB503 to 507 MiB
Boot time (systemd-analyze)6.5 to 6.7 s6.7 to 7.2 s
Unique CVEs (Grype)35, 16 High98, 27 High
firewalldinstalled, activenot installed
Automatic update timerdisabledenabled
cloud-init, qemu-guest-agentneitherneither

The size and memory savings are real. Boot time is close to a tie. An early run showed Fedora booting two seconds faster, and the cause turned out to be our lab, not the OS: only the Hummingbird VM had a serial port, and both images put console=ttyS0 on the kernel command line, so the kernel writes its boot log to that slow serial line. Hummingbird’s own boot arguments add console=ttyS0,115200n8:

cat /proc/cmdline | grep -o "console=[^ ]*"

Both serial and screen consoles are configured:

console=tty1
console=ttyS0,115200n8
console=tty0
console=ttyS0

Give both VMs the same hardware before you compare boot times.

The image is also not distroless in the container sense. It carries bash, dnf, Python, Podman and an SSH server. We ran the package check inside the image:

sudo podman run --rm quay.io/hummingbird-community/bootc-os:latest bash -c "rpm -qa | wc -l; ls /usr/lib/modules; rpm -q kernel bootc podman systemd dnf5 bash openssh-server python3 selinux-policy-targeted"

The kernel has a Fedora 44 release tag, and everything else carries the hum1 suffix of Hummingbird’s own rebuilds:

265
7.2.8-200.fc44.x86_64
kernel-7.2.8-200.fc44.x86_64
bootc-1.16.13-1.hum1.x86_64
podman-6.1.3-1.hum1.x86_64
systemd-261.2-1.hum1.x86_64
dnf5-5.4.6.0-1.hum1.x86_64
bash-5.3.15-2.1.hum1.x86_64
openssh-server-10.5p1-1.hum1.x86_64
python3-3.14.8-1.hum1.x86_64
selinux-policy-targeted-45.20-1.hum1.noarch

The Fedora Magazine announcement says the OS will use the Always Ready Kernel (ARK). This build does not use it yet. The README for the image says why: Hummingbird currently builds the userspace, and Fedora supplies the kernel and lower-level stack.

Step 1: Pull the image and verify the signature

You need a Linux machine with Podman to build the disk image. We used a Fedora 45 Cloud VM called build-01. If Podman is not installed yet, our guide to installing Podman on Fedora covers it. Install jq as well, because the scan in the last section uses it:

sudo dnf -y install podman skopeo jq >/dev/null 2>&1; rpm -q podman skopeo

Pull the image as root, because image-builder reads rootful container storage:

time sudo podman pull quay.io/hummingbird-community/bootc-os:latest 2>&1 | tail -2

About 850 MB came down in just over a minute:

Writing manifest to image destination
27c5388a92e0c624bfa3a97f311f25f58586dacfb691d588f9488153517577df

real	1m5.048s
user	0m11.566s
sys	0m4.099s

Verify the signature before you build anything from it. Community images are signed with the Konflux key published in the Hummingbird repository, not with Red Hat’s product key. The project’s cosign image does the check:

sudo podman run --rm registry.access.redhat.com/hi/cosign:latest verify --key "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/ci/key.pub?ref_type=heads" --insecure-ignore-tlog quay.io/hummingbird-community/bootc-os:latest 2>&1 | cut -c1-200

The signature matched. The --insecure-ignore-tlog flag is required because Konflux does not publish to the Rekor transparency log:

WARNING: Skipping tlog verification is an insecure practice that lacks transparency and auditability verification for the signature.

Verification for quay.io/hummingbird-community/bootc-os:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

The Red Hat key from the main Hummingbird docs is the wrong one for this image. We tried it:

sudo podman run --rm registry.access.redhat.com/hi/cosign:latest verify --key https://security.access.redhat.com/data/63405576.txt --insecure-ignore-tlog quay.io/hummingbird-community/bootc-os:latest 2>&1 | grep -v WARNING | cut -c1-200

The check failed:

Error: no matching signatures: crypto/rsa: verification error
error during command execution: no matching signatures: crypto/rsa: verification error

That failure means “wrong key”, not “tampered image”. Do not skip verification because the first key you found did not work.

Step 2: Write a blueprint with a password, not just a key

The image has no default user and no cloud-init, so the account has to be baked into the disk at build time. image-builder reads it from a TOML blueprint. Create a working directory and a blueprint.toml inside it:

mkdir -p ~/hummingbird && cd ~/hummingbird
vi blueprint.toml

Add one user. Generate the password hash with openssl passwd -6 and paste your own public key:

[[customizations.user]]
name = "mercy"
groups = ["wheel"]
password = "$6$replace-with-your-hash"
key = "ssh-ed25519 AAAA... mercy@ws-lab01"

Do not skip the password. The inline README snippet sets only an SSH key (the separate blueprint-example.toml in the repo sets a password and no key). We built the key-only version first, and the login worked, but sudo could never succeed, because a wheel member with no password has nothing to authenticate with. We checked from the SSH session:

hostnamectl hostname; sudo -n true && echo "sudo ok" ; sudo -l -n 2>&1 | tail -3; cat /etc/sudoers.d/* 2>/dev/null; ls /etc/sudoers.d

Every sudo attempt asked for a password the account does not have:

localhost
sudo: a password is required
sudo: a password is required
ls: cannot open directory '/etc/sudoers.d': Permission denied

With no root password set and no cloud-init to fix it, that machine had no route to root. We rebuilt the disk with the hashed password above.

Step 3: Build a qcow2 disk with image-builder

Run image-builder against the local image. The output directory has to exist before Podman can mount it, and our first attempt failed with Error: lstat output: no such file or directory for that reason:

cd ~/hummingbird && mkdir -p output && sudo podman run --rm --privileged --pull=newer -v /var/lib/containers/storage:/var/lib/containers/storage -v ./blueprint.toml:/blueprint.toml:ro -v ./output:/output ghcr.io/osbuild/image-builder:latest build qcow2 --bootc-ref quay.io/hummingbird-community/bootc-os:latest --bootc-default-fs ext4 --blueprint /blueprint.toml --output-dir /output --output-name hummingbird 2>&1 | tail -1; ls -lh output/

The build took about two minutes and produced a 521 MB disk:

Image build successful: /output/hummingbird.qcow2
total 520M
-rw-r--r--. 1 root root 521M Oct  5 20:53 hummingbird.qcow2

The disk carries a BIOS boot partition and an EFI system partition. We booted it under SeaBIOS only. The root filesystem grows to fill the virtual disk on first boot.

Step 4: Boot it on Proxmox VE

Copy hummingbird.qcow2 to the Proxmox host, then create a VM and import the disk. Our storage was zfs-pool; replace it with yours, for example local-lvm:

qm create 302 --name hb-lab01 --memory 2048 --cores 2 --cpu host --machine q35 --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --serial0 socket --vga std --ostype l26
qm importdisk 302 hummingbird.qcow2 zfs-pool
qm set 302 --scsi0 zfs-pool:vm-302-disk-0,discard=on,ssd=1 --boot order=scsi0
qm resize 302 scsi0 20G
qm start 302

There is no guest agent in the image, so Proxmox cannot show you the VM’s address. Log in at the console with the blueprint user and run ip -4 addr, or look up the MAC address in your DHCP leases. The console greets you as Hummingbird OS, not Fedora:

Hummingbird OS 20251124 console login prompt showing a Fedora 44 kernel on Proxmox VE

The version string 20251124 is fixed and predates this build by ten months. It does not change with each rebuild, so use the image digest from bootc status to tell builds apart. Set a hostname while you are here; the minimal image does not even include the hostname command, so use sudo hostnamectl set-hostname hb-lab01.

Check what the host thinks it is running:

sudo bootc status

The host tracks the Quay image and records the exact digest it booted:

  image:
    image: quay.io/hummingbird-community/bootc-os:latest
    transport: registry
  bootOrder: default
status:
  staged: null
  booted:
    image:
      image:
        image: quay.io/hummingbird-community/bootc-os:latest
        transport: registry
      version: latest
      timestamp: 2026-10-05T18:12:41Z
      imageDigest: sha256:02834e9e80b281448ae3c9e94bd12d07da478c229ac952f6c624225a20a56697
      architecture: amd64

Step 5: Learn the rules of the running system

The root filesystem is a read-only composefs mount. Writes to /usr fail no matter who you are:

sudo touch /usr/bin/test-file 2>&1; mount | grep -E " on / | on /usr "

The mount options show ro on the composefs root:

touch: cannot touch '/usr/bin/test-file': Read-only file system

Only /etc and /var keep changes across reboots. That is also why dnf install on the running host refuses to work:

sudo dnf install -y htop 2>&1 | tail -3; sudo dnf install -y git 2>&1 | tail -4

Both attempts end the same way:

Error: this bootc system is configured to be read-only. Pass --transient to perform this operation in a transient overlay which will reset when the system reboots.

The --transient flag the error suggests does work. It installs into an overlay that disappears at the next reboot, which is fine for a debugging session and nothing else:

sudo dnf install -y --transient git 2>&1 | tail -3; git --version

Git installed as 70 packages including its dependencies:

[69/70] Installing perl-Git-0:2.56.0-1. 100% |  21.3 MiB/s |  65.4 KiB |  00m00s
[70/70] Installing perl-NDBM_File-0:1.1 100% | 147.5 KiB/s |  29.5 KiB |  00m00s
Complete!
git version 2.56.0

Security defaults are sensible. SELinux is enforcing, firewalld runs in the public zone, and only 16 services were running after boot:

getenforce; systemctl is-active firewalld sshd chronyd NetworkManager auditd; sudo firewall-cmd --get-default-zone; sudo firewall-cmd --list-services; systemctl list-units --type=service --state=running --no-legend | wc -l

The zone allows SSH, mDNS and DHCPv6 replies and nothing else:

Enforcing
active
active
active
active
active
public
dhcpv6-client mdns ssh
16

Stock Fedora bootc ships without firewalld at all, so on this point Hummingbird is the stricter of the two. The whole process list fits on one screen. This is htop on the console after Step 6 added it. htop counts used memory differently from free, so it shows 189 MB here against the 411 MiB in the table:

htop on the Hummingbird OS console showing 26 tasks and 189 MB of memory in use

Workloads are meant to run as containers. Rootless Podman works out of the box, and Hummingbird’s own application images run as UID 65532 rather than root:

podman run --rm quay.io/hummingbird/python:latest python3 -c "import sys,os; print(sys.version.split()[0], os.getuid())" 2>&1 | tail -2; podman image ls --format "{{.Repository}}:{{.Tag}} {{.Size}}"

The script printed the Python version and the unprivileged UID:

3.14.8 65532
quay.io/hummingbird/python:latest 115 MB

Step 6: Add packages without breaking the build

Everything you add goes into a Containerfile that starts FROM the Hummingbird image. The README’s own example installs htop. We saved it as a two-line Containerfile in ~/hb/c1 (mkdir -p ~/hb/c1, then vi ~/hb/c1/Containerfile) and built it:

cd ~/hb/c1 && cat Containerfile && sudo podman build -t localhost/hb-htop . 2>&1 | tail -6

It fails:

FROM quay.io/hummingbird-community/bootc-os:latest
RUN dnf install -y htop && dnf clean all
Repositories loaded.
Failed to resolve the transaction:
No match for argument: htop
You can try to add to command line:
  --skip-unavailable to skip unavailable packages
Error: building at STEP "RUN dnf install -y htop && dnf clean all": while running runtime: exit status 1

The image ships exactly one repository, Red Hat’s Hummingbird package repo:

sudo podman run --rm quay.io/hummingbird-community/bootc-os:latest bash -c "ls /etc/yum.repos.d/; grep -hE \"^\[|^baseurl|^metalink|^enabled\" /etc/yum.repos.d/*.repo"

There are no Fedora repositories in the image:

hummingbird.repo
[public-hummingbird-$basearch-rpms]
baseurl=https://packages.redhat.com/api/pulp-content/public-hummingbird/$arch/
enabled=1
[public-hummingbird-source-rpms]
baseurl=https://packages.redhat.com/api/pulp-content/public-hummingbird/source/
enabled=0

That repo had 3,535 package names on build day, a fraction of Fedora. We checked a handful of everyday tools against it:

sudo podman run --rm quay.io/hummingbird-community/bootc-os:latest bash -c "dnf -q repoquery --available 2>/dev/null | wc -l; for p in htop tmux git vim-enhanced cloud-init qemu-guest-agent rsync tcpdump; do printf \"%-18s \" \$p; dnf -q repoquery --available --latest-limit 1 \$p 2>/dev/null | head -1 | grep . || echo \"(not in repo)\"; done"

Five of the eight are missing:

htop               (not in repo)
tmux               (not in repo)
git                git-0:2.56.0-1.hum1.x86_64
vim-enhanced       vim-enhanced-2:9.2.1129-1.hum1.x86_64
cloud-init         cloud-init-0:26.2-2.hum1.noarch
qemu-guest-agent   (not in repo)
rsync              (not in repo)
tcpdump            (not in repo)

Worse, two of the packages that are listed cannot be installed:

sudo podman run --rm quay.io/hummingbird-community/bootc-os:latest bash -c "dnf install -y --assumeno vim-enhanced cloud-init 2>&1 | grep -E \"nothing provides\""

Their dependencies are missing from the repo:

  - nothing provides libgpm.so.2()(64bit) needed by vim-enhanced-2:9.2.1129-1.hum1.x86_64 from public-hummingbird-x86_64-rpms
  - nothing provides xfsprogs needed by cloud-init-26.2-2.hum1.noarch from public-hummingbird-x86_64-rpms
  - nothing provides python3dist(oauthlib) needed by cloud-init-26.2-2.hum1.noarch from public-hummingbird-x86_64-rpms
  - nothing provides python3dist(jsonpatch) needed by cloud-init-26.2-2.hum1.noarch from public-hummingbird-x86_64-rpms
  - nothing provides python3dist(configobj) needed by cloud-init-26.2-2.hum1.noarch from public-hummingbird-x86_64-rpms
  - nothing provides dhcpcd needed by cloud-init-26.2-2.hum1.noarch from public-hummingbird-x86_64-rpms
  - nothing provides /usr/bin/nc needed by cloud-init-26.2-2.hum1.noarch from public-hummingbird-x86_64-rpms

The build config for this image explicitly allows Fedora repositories, and that is the way out. Add the Fedora repos at a lower priority, so Hummingbird’s own builds still win wherever both have a package.

We used Fedora 44, the release the image’s kernel comes from. Fetch the Fedora 44 signing key into a custom directory:

mkdir -p ~/hummingbird/custom && cd ~/hummingbird/custom
curl -o RPM-GPG-KEY-fedora-44 https://src.fedoraproject.org/rpms/fedora-repos/raw/f44/f/RPM-GPG-KEY-fedora-44-primary

Then create fedora.repo next to it:

vi fedora.repo

Paste the two repositories. In dnf the lower number wins, and the image’s Hummingbird repo is set to priority=10, so priority=200 keeps Fedora as the fallback:

[fedora]
name=Fedora 44 - $basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-44&arch=$basearch
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-44
priority=200

[updates]
name=Fedora 44 - $basearch - Updates
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f44&arch=$basearch
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-44
priority=200

Now the Containerfile. Keep the additions to leaf tools, and finish with bootc container lint so a broken image fails here instead of on a host:

vi Containerfile

This is the exact file we built:

FROM quay.io/hummingbird-community/bootc-os:latest
COPY RPM-GPG-KEY-fedora-44 /etc/pki/rpm-gpg/
COPY fedora.repo /etc/yum.repos.d/
RUN dnf install -y htop tmux qemu-guest-agent && dnf clean all
RUN bootc container lint

You need a registry the host can pull from. For a lab, run the plain registry:2 container on the build machine:

sudo podman rm -f registry >/dev/null; sudo podman run -d --name registry -p 5000:5000 --restart=always docker.io/library/registry:2 | cut -c1-12

Podman prints the new container ID:

cb40ff2c939d

It listens on port 5000. Our builder had no firewalld, so nothing else was needed. Open 5000/tcp if yours runs a firewall. Confirm the container is up:

sudo podman ps --filter name=registry --format "{{.Names}} {{.Image}} {{.Ports}}"; rpm -q firewalld

The registry is published on port 5000:

registry docker.io/library/registry:2 0.0.0.0:5000->5000/tcp
package firewalld is not installed

We call the registry registry.lab. Point that name at the build machine in /etc/hosts on the builder (use 127.0.0.1 there) and on the Hummingbird host, for example echo "10.0.1.50 registry.lab" | sudo tee -a /etc/hosts with the real IP of the builder. Then build the image with a tag that names the registry:

cd ~/hummingbird/custom && sudo podman build --no-cache -t registry.lab:5000/hummingbird-custom:1 . 2>&1 | grep -E "^(STEP|COMMIT)|Checks passed|^ (htop|tmux|qemu-guest-agent) "

All three packages came from the Fedora repos, and the lint passed:

STEP 1/5: FROM quay.io/hummingbird-community/bootc-os:latest
STEP 2/5: COPY RPM-GPG-KEY-fedora-44 /etc/pki/rpm-gpg/
STEP 3/5: COPY fedora.repo /etc/yum.repos.d/
STEP 4/5: RUN dnf install -y htop tmux qemu-guest-agent && dnf clean all
 htop             x86_64 0:3.4.1-3.fc44    fedora                         464.3 KiB
 qemu-guest-agent x86_64 2:10.2.2-1.fc44   fedora                           1.0 MiB
 tmux             x86_64 0:3.7c-1.fc44     updates                          1.4 MiB
STEP 5/5: RUN bootc container lint
Checks passed: 11
COMMIT registry.lab:5000/hummingbird-custom:1

Mixing repositories is a trade. Packages from Fedora do not get Hummingbird’s CVE process, and a library from one side can drift from what the other side expects. Add as little as you can, and rebuild often.

Step 7: Push the image and switch the host to it

Push the image to your registry. Ours is plain HTTP, so TLS verification is off for the push:

sudo podman push --tls-verify=false registry.lab:5000/hummingbird-custom:1 2>&1 | tail -1

The push finished:

Writing manifest to image destination

On the Hummingbird host, mark the lab registry as insecure. Do this only for a registry on a network you control. A real deployment should use TLS:

sudo sh -c 'printf "[[registry]]\nlocation = \"registry.lab:5000\"\ninsecure = true\n" > /etc/containers/registries.conf.d/50-registry-lab.conf'
sudo cat /etc/containers/registries.conf.d/50-registry-lab.conf

The file is root-only, so read it back with sudo:

[[registry]]
location = "registry.lab:5000"
insecure = true

Check that the host can reach the registry by name before you switch:

curl -s http://registry.lab:5000/v2/_catalog

The pushed repository is listed:

{"repositories":["hummingbird-custom"]}

Switch the host to the custom image:

sudo bootc switch registry.lab:5000/hummingbird-custom:1 2>&1 | tail -6

Only 4 of the 36 layers had to be downloaded, 13.9 MB in total, because the rest were already on disk:

layers already present: 32; layers needed: 4 (13.9 MB)
Deploying...done (2 seconds)
Queued for next boot: registry.lab:5000/hummingbird-custom:1
  Version: latest
  Digest: sha256:a66291fbde21104d6ca47318d970a75405dee5626935b4aec5ffe9ff15746e1c

Nothing changes until the reboot. Reboot, then confirm the booted image and the new packages:

sudo bootc status --format humanreadable 2>/dev/null || sudo bootc status | grep -E "image:|timestamp"; rpm -q htop tmux qemu-guest-agent; systemctl is-active qemu-guest-agent

The custom image is booted, the original Quay image is kept as the rollback, and the guest agent is running:

● Booted image: registry.lab:5000/hummingbird-custom:1
        Digest: sha256:a66291fbde21104d6ca47318d970a75405dee5626935b4aec5ffe9ff15746e1c (amd64)
       Version: latest (2026-10-05T18:12:41Z)

  Rollback image: quay.io/hummingbird-community/bootc-os:latest
          Digest: sha256:02834e9e80b281448ae3c9e94bd12d07da478c229ac952f6c624225a20a56697 (amd64)
         Version: latest (2026-10-05T18:12:41Z)
htop-3.4.1-3.fc44.x86_64
tmux-3.7c-1.fc44.x86_64
qemu-guest-agent-10.2.2-1.fc44.x86_64
active

Enable the QEMU guest agent option on the VM in Proxmox (qm set 302 --agent enabled=1, then a full stop and start) and the VM’s IP address shows up in the Proxmox UI.

Step 8: Update, roll back, and turn updates on

To test an upgrade, we added rsync to the Containerfile, rebuilt, and pushed under the same tag. On the host, ask bootc whether anything changed:

sudo bootc upgrade --check 2>&1 | tail -6

The host saw the new digest. Only one layer differs:

Update available for: docker://registry.lab:5000/hummingbird-custom:1
  Version: latest
  Digest: sha256:1571efe1cf60bcc03fc0dd9b3c3de3a2235e01f85ed059379766865812c3870e
Total new layers: 36    Size: 454.8 MB
Removed layers:   1     Size: 13.8 MB
Added layers:     1     Size: 14.5 MB

Now stage the upgrade, and check the automatic update timer while you are there. Stock Fedora bootc enables bootc-fetch-apply-updates.timer, which downloads and applies new images on its own:

sudo bootc upgrade 2>&1 | tail -4; systemctl list-timers bootc-fetch-apply-updates.timer --no-pager 2>&1 | head -3; systemctl is-enabled bootc-fetch-apply-updates.timer

The upgrade is staged, but no timer is scheduled. The unit exists and is disabled, because Hummingbird’s presets do not turn it on:

  Digest: sha256:1571efe1cf60bcc03fc0dd9b3c3de3a2235e01f85ed059379766865812c3870e
Total new layers: 36    Size: 454.8 MB
Removed layers:   1     Size: 13.8 MB
Added layers:     1     Size: 14.5 MB
NEXT LEFT LAST PASSED UNIT ACTIVATES

0 timers listed.
disabled

This is the part to get right before Hummingbird runs anything real. A host you install and forget will never update. Either enable the timer with sudo systemctl enable --now bootc-fetch-apply-updates.timer, which reboots the host when an update lands, or run upgrades from your own automation. Pick one now, not later.

After a reboot, rsync was present and the previous image became the rollback:

rpm -q rsync; sudo bootc status --format humanreadable | grep -E "image:|Digest"

Both deployments are listed with their digests:

rsync-3.5.1-1.fc44.x86_64
● Booted image: registry.lab:5000/hummingbird-custom:1
        Digest: sha256:1571efe1cf60bcc03fc0dd9b3c3de3a2235e01f85ed059379766865812c3870e (amd64)
  Rollback image: registry.lab:5000/hummingbird-custom:1
          Digest: sha256:a66291fbde21104d6ca47318d970a75405dee5626935b4aec5ffe9ff15746e1c (amd64)

Rolling back is one command plus a reboot:

sudo bootc rollback 2>&1 | tail -3

bootc queues the previous deployment for the next boot:

Next boot: rollback deployment

After the reboot, the same check shows rsync gone and the host back on the earlier digest:

rpm -q rsync; sudo bootc status --format humanreadable | grep -E "Booted|Digest" | head -2

The earlier digest is booted again:

package rsync is not installed
● Booted image: registry.lab:5000/hummingbird-custom:1
        Digest: sha256:a66291fbde21104d6ca47318d970a75405dee5626935b4aec5ffe9ff15746e1c (amd64)

Our changes in /etc, such as the hostname and the registry file, survived every switch, upgrade and rollback, because we made them before the first switch. A rollback returns /etc to the state of the deployment you roll back to, so edits made after an upgrade are lost. Also note that an enabled update timer can pull the newer image again and undo a rollback; disable the timer while you investigate.

How many CVEs Hummingbird really has

“Near-zero CVEs” is the headline claim, so we scanned both OS images with Grype using the same vulnerability database. Save each image to an OCI archive first, then download the database into a named volume:

sudo podman pull quay.io/fedora/fedora-bootc:45
mkdir -p ~/hb/scan
sudo podman save --format oci-archive -o ~/hb/scan/hb.tar quay.io/hummingbird-community/bootc-os:latest
sudo podman save --format oci-archive -o ~/hb/scan/fb.tar quay.io/fedora/fedora-bootc:45
sudo podman run --rm -v grype-db:/db -e GRYPE_DB_CACHE_DIR=/db docker.io/anchore/grype:latest db update 2>&1 | tail -1; ls -lh ~/hb/scan

On a first run the update downloads the database. Ours was already current, and both archives were in place:

No vulnerability database update available
total 1.4G
-rw-r--r--. 1 root root 1011M Oct  5 20:27 fb.tar
-rw-r--r--. 1 root root  420M Oct  5 20:27 hb.tar

Then scan both archives and count the matches by severity:

for t in hb fb; do echo "== $t"; sudo podman run --rm -v grype-db:/db -e GRYPE_DB_CACHE_DIR=/db -e GRYPE_DB_AUTO_UPDATE=false -v ~/hb/scan:/scan:z docker.io/anchore/grype:latest oci-archive:/scan/$t.tar -o json 2>/dev/null > /tmp/$t.json; sudo cat /tmp/$t.json | jq -r "\"matches: \(.matches | length)\", ([.matches[].vulnerability.severity] | group_by(.) | map(\"\(.[0]): \(length)\") | join(\", \"))"; done

hb is Hummingbird and fb is Fedora bootc 45:

== hb
matches: 61
High: 40, Low: 2, Medium: 19
== fb
matches: 952
High: 43, Low: 5, Medium: 904

The match counts repeat each CVE once per affected package. The scan saved its JSON to /tmp, so count unique IDs from there:

for t in hb fb; do printf "%s unique IDs: %s, High: %s\n" $t "$(jq -r ".matches[].vulnerability.id" /tmp/$t.json | sort -u | wc -l)" "$(jq -r ".matches[] | select(.vulnerability.severity==\"High\") | .vulnerability.id" /tmp/$t.json | sort -u | wc -l)"; done

Hummingbird had 35 and Fedora had 98:

hb unique IDs: 35, High: 16
fb unique IDs: 98, High: 27

Grype recognizes Hummingbird as its own distribution and matched it against Hummingbird’s advisory data, so these are not guesses from package names. Group the Hummingbird matches by package to see where they land:

jq -r ".matches[] | \"\(.vulnerability.severity) \(.artifact.name) \(.artifact.version)\"" /tmp/hb.json | sort | uniq -c | sort -rn | head -15

The High findings cluster in the tools the OS uses to update itself:

     10 Medium bootupd 0.3.2-1.hum1
      7 High rpm-plugin-selinux 6.1.0-1.hum1
      7 High rpm-libs 6.1.0-1.hum1
      7 High rpm 6.1.0-1.hum1
      3 High rpm-sequoia 1.10.1.1-1.4.hum1
      3 High google.golang.org/grpc v1.82.0

Hummingbird is cleaner than Fedora bootc, by about a factor of three. It is not zero, and most of its findings had no fixed version published on the day we scanned (56 of 61 matches). Fedora’s count looks worse partly because its :45 image was three days old, and 884 of its matches already had fixes in newer packages. Scan the exact image you deploy, on the day you deploy it.

Before Hummingbird goes near production

Run through this list on any Hummingbird host before it carries real work. Each item maps to something that went wrong or nearly went wrong in our lab.

1. Confirm the image is still marked community or experimental on Quay, and read the README for changes. The package set and update mechanism may change without notice. 2. Verify every image you deploy with cosign and the Konflux key from Step 1. A failure with Red Hat’s product key means you used the wrong key. 3. Pin hosts to a digest or to your own rebuilt tag, not to :latest on Quay. The upstream tag moves several times a day. 4. Give the blueprint user a password hash as well as an SSH key, or you will have a login with no way to sudo. 5. Decide how updates happen. The bootc update timer is disabled by default, so either enable it or schedule bootc upgrade yourself. 6. Keep added packages to a short list of leaf tools from Fedora, at a lower priority than the Hummingbird repo, and keep bootc container lint in the Containerfile. 7. Use TLS on the registry the hosts pull from. The insecure registry in Step 7 is for a lab network only. 8. Scan the final image with Grype or a similar tool before rollout, and again after each rebuild.

For the immutable desktop side of Fedora, compare our guide to managing packages on Fedora Silverblue. For containers on top of a host like this, Podman Quadlet with systemd is the natural next step. Fedora CoreOS remains the production-supported image-based option today, and our Fedora 45 release overview covers the rest of this cycle.

Keep reading

How to Upgrade Fedora 44 to Fedora 45 (GUI, CLI and Rollback) Fedora How to Upgrade Fedora 44 to Fedora 45 (GUI, CLI and Rollback) Enable RPM Fusion and Install Multimedia Codecs on Fedora 45 / 44 / 43 Fedora Enable RPM Fusion and Install Multimedia Codecs on Fedora 45 / 44 / 43 Best UI Applications for Managing Docker Containers Containers Best UI Applications for Managing Docker Containers CapRover Docker Install on Debian 13: Deploy Your First App Containers CapRover Docker Install on Debian 13: Deploy Your First App Best OpenShift Books for Admins and EX280 Candidates Books Best OpenShift Books for Admins and EX280 Candidates How To Upgrade OpenShift / OKD Cluster Minor Version Containers How To Upgrade OpenShift / OKD Cluster Minor Version

Leave a Comment

Press ESC to close