Check /sys/module/kvm_intel/parameters/nested before you edit a single modprobe file, because on a current kernel KVM nested virtualization is usually switched on already. Ours printed Y on a fresh Debian 13 host, and the kernel’s nested guest documentation says the parameter has defaulted to on for Intel and AMD since Linux 4.20. When a guest still cannot start its own VMs, the problem is almost always one layer up: the hypervisor is not handing the vmx CPU flag to that guest.
Below you get the check, the kvm_intel nested=1 fix for hosts that read N, a nested guest built with virt-install and host-passthrough, the GRUB boot loop --graphics none caused on the Debian cloud image, and measured nesting costs. Every command below was run in October 2026 on Debian 13 (QEMU 10.0, libvirt 11.3) on an Intel Core i7-6700, and the AMD lines are marked as untested.
Check if nested virtualization is enabled
Two things have to be true on the KVM host before a guest can run its own VMs: the CPU must expose hardware virtualization to the host, and the kvm_intel module must have its nested parameter on. Start with the parameter, since it is one file read and it answers the question people search for most:
cat /sys/module/kvm_intel/parameters/nestedOur host printed a single letter.
YA Y on Intel (or 1 on AMD, where kvm_amd declares nested as an integer; not tested here) means nesting is on, and you can skip straight to building the guest. An N or 0 means a distro or an admin turned it off, which the next section fixes. If the file does not exist at all, kvm_intel is not loaded, so check the module list and the device node QEMU opens:
lsmod | grep kvm; ls -l /dev/kvm 2>&1Both modules are loaded, and the zero at the end of the kvm_intel line means no VM was using it yet. That count matters later, because the module cannot be reloaded while it is above zero. If kvm_intel is missing from your list, the CPU check below usually explains why, because the module will not load on a CPU that does not expose VT-x.
kvm_intel 417792 0
kvm 1380352 1 kvm_intel
irqbypass 12288 1 kvm
crw-rw---- 1 root kvm 10, 232 Oct 20 10:17 /dev/kvmThen confirm the CPU side. lscpu names the virtualization extension, and counting the vmx flag in /proc/cpuinfo tells you how many vCPUs carry it (one line per logical CPU):
lscpu | grep -E "Model name|Virtualization|Hypervisor vendor|Virtualization type"; grep -cw vmx /proc/cpuinfo; grep -cw svm /proc/cpuinfoOur host shows VT-x on all 8 logical CPUs and no AMD svm flag. The Hypervisor vendor: KVM line matters too: it means this “host” is itself a virtual machine, which the next section explains.
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Virtualization: VT-x
Hypervisor vendor: KVM
Virtualization type: full
8
0AMD hosts use a different module and flag, but the logic is the same. We only had Intel hardware for this lab, so the AMD column below comes from the kernel documentation and was not run here:
| Item | Intel (tested) | AMD (not tested in our lab) |
|---|---|---|
| CPU flag | vmx | svm |
| Kernel module | kvm_intel | kvm_amd |
| Nested parameter | /sys/module/kvm_intel/parameters/nested | /sys/module/kvm_amd/parameters/nested |
| modprobe option | options kvm_intel nested=1 | options kvm_amd nested=1 |
If the vmx count is 0 on bare metal, VT-x is off in the firmware and no kernel setting will help. Turn it on in the BIOS or UEFI setup first.
Know which layer you are on
Nested virtualization talks about levels, and the numbers only make sense once you know where your own shell sits. L0 is the hypervisor on physical hardware, L1 is a VM running on it, and L2 is a VM that L1 starts. Our lab had one more level than most readers will, because the “KVM host” for this guide is itself a VM on a Proxmox node:
| Level | Machine | Role in this guide |
|---|---|---|
| L0 | pve01, Proxmox VE on an Intel i7-6700 | Physical hypervisor, kvm_intel nested=Y |
| L1 | deb01, Debian 13 VM | The KVM host every command runs on |
| L2 | l2, Debian 13 cloud image guest | The nested guest built with virt-install |
| L3 | CirrOS test guests | Only used to measure the cost of nesting twice |
If you run KVM on bare metal, your host is L0 and your guests are L1, so everything below still applies one level shallower. If your host is a Proxmox VM like ours, the L1 only sees vmx because its Proxmox CPU type is host. We read that setting on the Proxmox node. Run the same check there, replacing 4004 with your L1 VM’s ID, before you blame anything on the Debian side:
qm config 4004 | grep -E "^cpu"Our L1 VM (ID 4004) showed the setting that exposes VT-x to it. If yours shows a named model instead, set it with qm set <vmid> --cpu host on the node, then fully stop and start that VM, since a reboot from inside it keeps the old CPU. We did not run that change in this lab, because our VM was already on host.
cpu: hostThe same idea applies to any hypervisor at L0. On XCP-ng it’s a per-VM flag rather than a CPU type, which our guide on nested virtualization on an XCP-ng VM walks through.
Enable KVM nested virtualization in kvm_intel
You only need this step when the check above printed N. The parameter is read when the module loads, so you set it in a modprobe file and then reload kvm_intel. The file also keeps the setting across reboots, which a one-off modprobe kvm_intel nested=1 would not. Put the option in its own file so you never clobber an existing one, because tee without -a overwrites whatever is there. Our lab used kvm.conf; the name is your choice, and any *.conf file in that directory works. Write the option and reload the module in one go:
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf; sudo modprobe -r kvm_intel && sudo modprobe kvm_intel; cat /sys/module/kvm_intel/parameters/nestedThe tee echoes the line it wrote, and the parameter reads Y after the reload. The setting now survives reboots too, because modprobe reads every file in /etc/modprobe.d each time it loads the module.
options kvm_intel nested=1
YOn an AMD host the file would carry options kvm_amd nested=1 and you’d reload kvm_amd instead. We did not run that variant. The reload only works while nothing holds the module, so shut every VM down first, or reboot the host if you cannot.
modprobe: FATAL: Module kvm_intel is in use
This is what you get when a guest is still running. We left the nested guest up and tried to unload the module:
sudo virsh list; sudo modprobe -r kvm_intel; echo rc=$?The kernel refuses, because the running domain has the module open.
Id Name State
----------------------
10 l2 running
modprobe: FATAL: Module kvm_intel is in use.
rc=1lsmod at that moment showed a use count of 4 on kvm_intel, so one VM is enough to block the reload. Run sudo virsh shutdown on each domain in virsh list, wait for virsh list --all to show them all shut off, then repeat the reload. Nothing is lost by failing here, because the modprobe file is already written and takes effect on the next boot anyway.
Install the KVM stack on the host
A bare Debian 13 host has the kvm modules but none of the userspace to manage guests. We installed the minimum for a headless host that builds guests from cloud images:
- qemu-system-x86 (the emulator QEMU runs guests with)
- libvirt-daemon-system and libvirt-clients (libvirtd and virsh)
- virtinst (virt-install and virt-xml)
- qemu-utils (qemu-img for disk overlays)
- cloud-image-utils (cloud image helpers)
- cpu-checker (the kvm-ok test)
Install them, add your user to the libvirt and kvm groups, and confirm the versions and that libvirtd is up. The group change only applies to new logins, which is why the guide keeps using sudo with virsh:
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq qemu-system-x86 libvirt-daemon-system virtinst libvirt-clients qemu-utils cloud-image-utils cpu-checker >/dev/null 2>&1; echo rc=$?; virsh --version; qemu-system-x86_64 --version | head -1; virt-install --version; sudo usermod -aG libvirt,kvm "$USER"; systemctl is-active libvirtdA zero return code, the three version strings and active mean the stack is ready. The redirect to /dev/null hides apt’s error text, so if you get a non-zero rc, rerun the install without -qq and the redirect to see what failed.
rc=0
11.3.0
QEMU emulator version 10.0.13 (Debian 1:10.0.13+ds-0+deb13u1)
5.0.0
activelibvirt ships its own host checker, and it’s worth one run because it tests /dev/kvm access the same way libvirtd will:
sudo virt-host-validate qemuEvery line that matters for nesting passed. The two warnings are about other features, so you can ignore them here:
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking if device '/dev/kvm' exists : PASS
QEMU: Checking if device '/dev/kvm' is accessible : PASS
QEMU: Checking if device '/dev/vhost-net' exists : PASS
QEMU: Checking if device '/dev/net/tun' exists : PASS
QEMU: Checking for cgroup 'cpu' controller support : PASS
QEMU: Checking for cgroup 'cpuacct' controller support : PASS
QEMU: Checking for cgroup 'cpuset' controller support : PASS
QEMU: Checking for cgroup 'memory' controller support : PASS
QEMU: Checking for cgroup 'devices' controller support : PASS
QEMU: Checking for cgroup 'blkio' controller support : PASS
QEMU: Checking for device assignment IOMMU support : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
QEMU: Checking for secure guest support : WARN (Unknown if this platform has Secure Guest support)The IOMMU warning only affects PCI passthrough, and our L1 VM exposes no DMAR table, so it is expected. The secure guest line is about confidential-computing guests such as AMD SEV. A FAIL on either /dev/kvm line is the one to act on: it means the vmx flag or the module from the first section is missing. The broader host setup, including bridges and virt-manager, is in our KVM on Debian with virt-manager and Cockpit guide, and Ubuntu readers can follow the KVM and QEMU install for Ubuntu.
Create a nested guest with host-passthrough
The guest needs a network, a disk and a login before you can test KVM inside it. Debian doesn’t start libvirt’s default NAT network after install, so start it and mark it to come up on boot:
sudo virsh net-start default 2>&1; sudo virsh net-autostart default; sudo virsh net-listThe network is active and marked for autostart. If you skip net-autostart, libvirt leaves the network down after the next host reboot and your guests come up with no DHCP address, so do not leave it out.
Network default started
Network default marked as autostarted
Name State Autostart Persistent
--------------------------------------------
default active yes yesSet the guest name once so the rest of the commands work as written. We called ours l2 after its level; use any name you like:
export VM_NAME="l2"A cloud image is the fastest way to a working guest, because it boots straight to a login with no installer. Download the Debian 13 generic cloud image into libvirt’s image directory:
cd /tmp && wget -q https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2 && sudo mv debian-13-genericcloud-amd64.qcow2 /var/lib/libvirt/images/Rather than writing to the downloaded file, create a 20 GB qcow2 overlay on top of it. The base image stays clean, so you can delete the overlay and start again in seconds:
sudo qemu-img create -f qcow2 -F qcow2 -b /var/lib/libvirt/images/debian-13-genericcloud-amd64.qcow2 "/var/lib/libvirt/images/${VM_NAME}.qcow2" 20Gqemu-img confirms the backing file and the virtual size. The overlay starts at a few hundred KiB (196 KiB on ours) and only grows as the guest writes, so the 20 GB figure costs you nothing until it is used. Leave the base image alone, because every overlay you create later can share it.
Formatting '/var/lib/libvirt/images/l2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=21474836480 backing_file=/var/lib/libvirt/images/debian-13-genericcloud-amd64.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16The cloud image has no password login, so the guest gets your SSH public key through cloud-init. Create a key if you do not have one yet:
test -f ~/.ssh/id_ed25519 || ssh-keygen -q -t ed25519 -N "" -f ~/.ssh/id_ed25519; ls ~/.ssh/id_ed25519.pubNow create the guest. We left the CPU mode out on purpose, because virt-install already picks host-passthrough on a KVM host, which hands the host CPU (and its vmx flag) to the guest unchanged. The --video vga flag next to --graphics none is not optional with this image, for the reason in the next subsection:
sudo virt-install --name "${VM_NAME}" --memory 3072 --vcpus 2 --import \
--disk "/var/lib/libvirt/images/${VM_NAME}.qcow2,bus=virtio" \
--osinfo debian13 --network network=default \
--cloud-init root-ssh-key="${HOME}/.ssh/id_ed25519.pub",clouduser-ssh-key="${HOME}/.ssh/id_ed25519.pub" \
--graphics none --video vga --noautoconsole 2>&1 | tail -2virt-install returns while cloud-init is still working inside the guest, so you will not have an IP address straight away.
Domain is still running. Installation may be in progress.
You can reconnect to the console to complete the installation process.Poll libvirt for the guest’s DHCP address, then keep it in a variable for the SSH checks later:
for i in $(seq 1 30); do A=$(sudo virsh domifaddr "${VM_NAME}" 2>/dev/null | awk "/ipv4/{print \$4}"); [ -n "$A" ] && break; sleep 4; done; echo "${VM_NAME} ip: $A after $((i*4))s"; sudo virsh list --allOurs had a lease 20 seconds after creation. If the loop gives up after 120 seconds with an empty address, the guest is not booting, so check its console before you touch any CPU setting. The subsection below shows the most likely reason.
l2 ip: 192.168.122.80/24 after 20s
Id Name State
----------------------
7 l2 runningStrip the prefix length into GUEST_IP. The cloud image’s login user is debian, and both values carry through to the next sections:
export GUEST_IP="${A%/*}"; echo "${GUEST_IP}"On an earlier attempt, before we added --video vga, the domain showed as shut off seconds after virt-install returned. If virsh list --all shows that for you, run sudo virsh start "${VM_NAME}" and poll again. Exports only last for the current shell. If you reconnect, set VM_NAME again and recover the address in one line:
GUEST_IP=$(sudo virsh domifaddr "${VM_NAME}" | awk '/ipv4/{print $4}' | cut -d/ -f1)It prints nothing, but every later ssh debian@"${GUEST_IP}" command picks the value up from your shell.
Guest stuck looping on “Booting `Debian GNU/Linux'”
Our first guest never got an IP. virsh domifaddr stayed empty, virsh net-dhcp-leases default listed nothing, and the overlay disk stayed at 196 KiB, which means the guest never wrote a byte. The serial console showed why: GRUB printed its boot line over and over and never reached “Loading Linux”:
sudo timeout 10 script -qc "virsh console ${VM_NAME} --force" /dev/null </dev/null | grep -c "Booting"Four GRUB boot attempts in ten seconds, and each one restarted the VM before the kernel loaded.
4This looks like a nesting failure, but it isn’t one. We got the same loop with host-model as with host-passthrough, and on both the i440fx and q35 machine types. The cause is the missing video device: --graphics none makes virt-install leave the <video> device out of the domain XML (libvirt just runs what is defined), and the Debian cloud image’s GRUB resets the machine without one. We proved it with plain QEMU at the same level, counting GRUB’s boot line over 20 seconds with -nodefaults (no VGA) and then with -vga std added. Stop the domain first, because QEMU will not open a disk image that the running guest still holds locked:
sudo virsh destroy "${VM_NAME}"; sudo timeout 20 qemu-system-x86_64 -nodefaults -machine pc -accel kvm -cpu host -m 3072 -smp 2 -display none -serial stdio -snapshot -drive file="/var/lib/libvirt/images/${VM_NAME}.qcow2",if=virtio 2>&1 | tr -d "\r" | sed "s/\x1b\[[0-9;]*[A-Za-z]//g" | grep -cE "Booting .Debian"; echo "^ grub Booting count in 20s with -nodefaults (no VGA)"; sudo timeout 20 qemu-system-x86_64 -nodefaults -vga std -machine pc -accel kvm -cpu host -m 3072 -smp 2 -display none -serial stdio -snapshot -drive file="/var/lib/libvirt/images/${VM_NAME}.qcow2",if=virtio 2>&1 | tr -d "\r" | sed "s/\x1b\[[0-9;]*[A-Za-z]//g" | grep -cE "Booting .Debian"; echo "^ same with -vga std"Ten GRUB boot attempts without a VGA device, one clean boot with it.
10
^ grub Booting count in 20s with -nodefaults (no VGA)
1
^ same with -vga stdSo keep --video vga whenever you pair --graphics none with this image. If a guest is already defined and looping, sudo virsh destroy and sudo virsh undefine it, recreate the overlay, and run virt-install again with the flag.
Pick a CPU mode for nested KVM
The CPU mode in the domain XML decides whether the guest sees vmx, so it is the second thing to check after the host parameter. Read what virt-install chose:
sudo virsh dumpxml "${VM_NAME}" | grep "<cpu"virt-install wrote host-passthrough without being asked. That is the mode you want here, because it hands the guest the host CPU’s full flag set, vmx included, so there is nothing to edit in the XML.
<cpu mode='host-passthrough' check='none' migratable='on'/>Guests defined years ago, or copied from another host, often carry a named CPU model instead, and that is where nesting quietly breaks. We tried each mode on the same guest, and this is what happened to vmx inside it:
| CPU mode set with virt-xml | XML libvirt wrote | vmx in guest |
|---|---|---|
host-passthrough | <cpu mode='host-passthrough' check='none' migratable='on'/> | 4 lines, /dev/kvm present |
host-model | <cpu mode='host-model' check='partial'/> | Not checked (set only while chasing the boot loop) |
Skylake-Client-IBRS | mode='custom' match='exact' plus four require features | 0 lines, no /dev/kvm |
Skylake-Client-IBRS,+vmx | Same, plus <feature policy='force' name='vmx'/> | 4 lines, /dev/kvm present |
Our pick is host-passthrough for any guest that will run its own VMs. It is the default, it worked first time, and a nested guest is rarely one you’ll live-migrate to a host with a different CPU. Use a named model plus +vmx only when the guest has to move between unlike hosts, because you then keep the stable model and still pass the flag. The libvirt CPU model documentation covers the remaining options.
Every change below was made with the guest fully shut down. The new XML only applies when libvirt starts a fresh QEMU process, and a reboot from inside the guest keeps the old one. This loop waits until the domain has actually stopped:
sudo virsh shutdown "${VM_NAME}"; while sudo virsh domstate "${VM_NAME}" | grep -q running; do sleep 2; doneTo reproduce the broken case, switch the guest to a plain named model, start it, and count vmx inside. We used Skylake-Client-IBRS because it matches our i7-6700; pick one your host supports from sudo virsh domcapabilities or virsh cpu-models x86_64:
sudo virt-xml "${VM_NAME}" --edit --cpu Skylake-Client-IBRS; sudo virsh dumpxml "${VM_NAME}" | sed -n '/<cpu/,/<\/cpu>/p'; sudo virsh start "${VM_NAME}"libvirt writes an exact custom model with the speculative-execution features the host needs, and nothing about vmx. An old guest definition, or one created with a specific --cpu model, usually looks like this.
Domain 'l2' defined successfully.
<cpu mode='custom' match='exact' check='none'>
<model fallback='allow'>Skylake-Client-IBRS</model>
<feature policy='require' name='ibpb'/>
<feature policy='require' name='md-clear'/>
<feature policy='require' name='spec-ctrl'/>
<feature policy='require' name='ssbd'/>
</cpu>
Domain 'l2' startedNotice there is no vmx line anywhere in that CPU definition, and the guest pays for it the moment you look for KVM inside it.
ls: cannot access ‘/dev/kvm’: No such file or directory
Once the guest is back up, ask it for the flag and the device node. You want a non-zero count and a character device, the same as on the host.
ssh debian@"${GUEST_IP}" 'grep -cw vmx /proc/cpuinfo; ls -l /dev/kvm 2>&1'Zero vmx lines and no /dev/kvm. QEMU inside this guest has no KVM to use, so anything it starts falls back to TCG emulation.
0
ls: cannot access '/dev/kvm': No such file or directoryThe host parameter was Y the whole time, which is why this error sends people in the wrong direction. Shut the guest down with the loop above and add the flag back to the named model:
sudo virt-xml "${VM_NAME}" --edit --cpu Skylake-Client-IBRS,+vmx; sudo virsh dumpxml "${VM_NAME}" | sed -n '/<cpu/,/<\/cpu>/p'; sudo virsh start "${VM_NAME}"The XML gains a force policy for vmx, and after the start the same SSH check printed 4 vmx lines and the /dev/kvm node again:
<feature policy='force' name='vmx'/>To go back to the default, shut down once more and set host-passthrough. virt-manager exposes the same choice on the VM’s CPU details page, but we did not test the GUI path for this guide:
sudo virt-xml "${VM_NAME}" --edit --cpu host-passthrough; sudo virsh start "${VM_NAME}"For the rest of the domain lifecycle (autostart, snapshots, console), the virsh commands cheat sheet has the commands in one place.
Verify KVM inside the nested guest
A working nested guest passes the same checks as a bare-metal KVM host. Install QEMU and cpu-checker inside the guest, then ask kvm-ok whether acceleration is available and read the guest’s own nested parameter:
ssh debian@"${GUEST_IP}" 'sudo apt-get update -qq; sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq qemu-system-x86 cpu-checker >/dev/null 2>&1; echo rc=$?; qemu-system-x86_64 --version | head -1; sudo kvm-ok; cat /sys/module/kvm_intel/parameters/nested'Our L2 guest reported KVM acceleration and its own nested parameter set to Y. kvm-ok checks both the flag and the device node, so a pass here means the guest can start accelerated VMs of its own.
rc=0
QEMU emulator version 10.0.13 (Debian 1:10.0.13+ds-0+deb13u1)
INFO: /dev/kvm exists
KVM acceleration can be used
YA healthy nested guest shows all of the following, and our l2 passed every item:
- one vmx line per vCPU (4 lines for 2 vCPUs here)
- a
/dev/kvmdevice owned by the kvm group - lscpu showing VT-x and Hypervisor vendor KVM
- kvm-ok printing “KVM acceleration can be used”
- kvm_intel loaded with nested=Y
The terminal capture below puts the host side and the guest side next to each other, which is the quickest way to see that both layers agree:
Note that the guest got nested=Y without any modprobe file of its own, which is the kernel default at work again. If kvm-ok says /dev/kvm is missing, go back to the CPU mode section, since the guest XML is the usual cause. For guests that should come up fully configured on first boot, our guide to cloud-init with virt-install covers user-data beyond SSH keys.
Nested guest performance against the L1 host
One level of nesting costs very little for CPU-bound work, and a second level made booting slower than switching KVM off entirely. That’s the short reading of the numbers below, all measured on the same i7-6700. Boot time was taken with a small script that starts a CirrOS guest and stops the clock when the guest kernel hands off to /init. Fetch the image first, along with bc, which the script uses for the arithmetic:
sudo apt-get install -y bc
cd /tmp && wget -q https://download.cirros-cloud.net/0.6.3/cirros-0.6.3-x86_64-disk.img #https://download.cirros-cloud.net/Create the timing script in /tmp, next to the image it boots.
sudo vim /tmp/bootcirros.shThis is the exact script we ran at both levels. It uses -cpu max so the same command works whether or not the level you run it on has KVM, which lets you swap kvm for tcg as the only change.
#!/bin/bash
# bootcirros.sh <accel>: seconds from qemu start until the guest kernel hands off to /init
ACC=$1; S=$(date +%s.%N)
timeout 600 qemu-system-x86_64 -accel $ACC -cpu max -m 256 -nographic -snapshot -drive file=/tmp/cirros-0.6.3-x86_64-disk.img,if=virtio </dev/null 2>&1 | while IFS= read -r l; do case "$l" in *"Run /init as init process"*) echo "accel=$ACC kernel reached /init after $(echo "$(date +%s.%N) - $S" | bc) s"; pkill -f "[q]emu-system-x86_64 -accel $ACC -cpu max"; break;; esac; doneMake it executable and run it three times on the host. Copy the image and script into the guest with scp and repeat there with kvm, then with tcg for pure emulation:
sudo chmod +x /tmp/bootcirros.sh; for r in 1 2 3; do sudo /tmp/bootcirros.sh kvm; doneThe host run came back in about six seconds each time. That is your baseline: a guest started on deb01, which makes it L2 counted from the hardware, the same depth as l2 itself.
accel=kvm kernel reached /init after 5.933989385 s
accel=kvm kernel reached /init after 6.153719051 s
accel=kvm kernel reached /init after 6.944468994 sTo time the same boot one level deeper, copy the image and the script into the nested guest, which already has QEMU from the verify step:
scp -q /tmp/cirros-0.6.3-x86_64-disk.img /tmp/bootcirros.sh debian@"${GUEST_IP}":/tmp/Then run it inside the guest with KVM and with TCG. Expect the KVM runs to take about a minute each there, which is the point of the test:
ssh debian@"${GUEST_IP}" 'sudo apt-get install -y -qq bc >/dev/null; for r in 1 2 3; do sudo /tmp/bootcirros.sh kvm; done; for r in 1 2 3; do sudo /tmp/bootcirros.sh tcg; done'The CPU and syscall tests used openssl speed -seconds 5 -bytes 16384 sha256 and a Python loop of 2,000,000 getppid() calls, run once on deb01 and once inside l2. All the results side by side:
| Test | deb01 (L1 host) | l2 (nested guest) | Reading |
|---|---|---|---|
| CirrOS boot to /init, KVM | 5.93, 6.15, 6.94 s (guest at L2) | 61.2, 68.4, 80.5 s (guest at L3) | About 10x slower one level deeper |
| CirrOS boot to /init, TCG | not run | 9.73, 10.49, 12.14 s (guest at L3) | Emulation beat nested KVM |
| openssl sha256, 16 KiB blocks | 394716.77k | 389981.88k | About 1% apart |
| 2M getppid() syscalls | 2.65 s | 2.98 s | About 12% slower |
| dd 1 GiB, oflag=direct | 312 MB/s | 584 MB/s | Not comparable (see below) |
The sha256 result is the one that matters for most people: a nested guest doing plain computation runs within about 1% of its host, because user-space instructions execute directly on the CPU at every level. Syscalls came out about 12% slower, and we did not profile where that gap comes from. The boot test is where nesting hurts, because booting a kernel is heavy on the privileged operations that cause VM exits, and at L3 each exit is handled through three hypervisor layers instead of two, with every layer’s exits multiplying the ones below it. KVM ended up six to seven times slower than TCG emulation doing the same boot inside l2.
The dd row is in the table so nobody quotes it. The guest wrote faster than the host because oflag=direct inside l2 only bypasses the guest’s own cache. Its writes still land in a qcow2 overlay with deb01’s page cache underneath, so the number measures caching, not the cost of nesting. Benchmark disk on a raw device with caching off before you draw any conclusion from it.
So keep nesting to one level below the host you control. A KVM host on bare metal that runs guests which run their own VMs (CI runners, OpenShift Local, a Proxmox lab in a VM) works well, and CPU-heavy work inside those guests is close to native. If the box under you runs Proxmox, as ours did, give the L1 VM cpu: host and stop at the guests it starts; our Proxmox VE post-install steps cover the rest of that node’s setup. Once a workload needs a third level, a TCG guest or a second physical host will serve you better than nested KVM.
