Containers

k0s vs k3s vs MicroK8s vs kubeadm: Lightweight Kubernetes Benchmarked

I had a 2 GB mini PC and a simple question: which small Kubernetes actually fits on it? k0s, k3s and MicroK8s all call themselves lightweight, and the docs disagree by a factor of four on how much RAM you need (MicroK8s says 540 MB, k3s says 2 GB). So I stopped reading spec sheets and ran all of them on identical VMs until they broke.

Original content from computingforgeeks.com - post 102297

This k0s vs k3s vs MicroK8s comparison adds plain kubeadm as the fourth candidate, because “k3s vs k8s” is the question most people really mean. Every distribution got the same Ubuntu VM and the same five tests: install time, cold boot to working DNS, idle memory and CPU, disk use, and the smallest RAM size it survives. I also logged what each one installs by default (CNI, ingress, LoadBalancer, storage, datastore), because that’s what you live with after the benchmark is over.

Ran all four on fresh Ubuntu 26.04 VMs in October 2026: k0s v1.36.4+k0s.1, k3s v1.36.5+k3s1, MicroK8s v1.36.2 and kubeadm v1.36.5, so every cluster is on the same Kubernetes minor.

1. k0s vs k3s vs MicroK8s vs kubeadm: the numbers

Boot times are medians of three cold boots. Memory figures average two idle samples. “Kubernetes memory” is the whole VM’s used memory minus what the same Ubuntu VM uses with nothing installed (441 MB), so it’s what the distribution itself costs you. The RAM floor is the smallest VM size where the node came back Ready, ran a new pod and was still healthy 60 seconds later, on every repeat.

Metrick0sk3sMicroK8skubeadm
Install to working cluster (1 run)263 s72 s166 s240 s
Cold boot to node Ready (median)36.4 s39.8 s55.9 s45.2 s
Cold boot to CoreDNS ready (median)76.0 s48.3 s91.2 s51.2 s
Idle memory, whole VM (bare Ubuntu: 441 MB)899 MB1,068 MB968 MB902 MB
Kubernetes memory (VM minus bare OS)~458 MB~627 MB~527 MB~461 MB
Idle CPU, 2 vCPU (two 5 min samples)3.2% / 3.5%3.3% / 4.1%10.6% / 9.6%7.4% / 6.1%
Disk added to the root filesystem1,231 MB1,159 MB1,758 MB1,380 MB
RAM floor (VM size)1,280 MB1,536 MB1,792 MB1,152 MB to run, 2 GB to install
Documented minimum1 GB (k0s only, OS extra)2 GB (server, workloads extra)540 MB2 GB

Three results surprised me. k3s, the one everybody picks for small boxes, uses the most memory of the four. Most of that goes on the extras it ships and the others don’t: Traefik, a LoadBalancer controller and a storage provisioner. Plain kubeadm idles at the same memory as k0s and runs on the least RAM of the lot once it’s installed. And MicroK8s, with the lowest documented minimum, has the highest real floor: it fell over at 1.5 GB, nearly three times its own published number.

Raw speed goes to k3s. It is the only one that went from a bare VM to a working cluster in about a minute, and it had DNS answering soonest after a reboot.

2. How I tested

Five VMs on one Proxmox VE node, all cloned from the same Ubuntu 26.04 cloud image: 2 vCPUs (host CPU type), 4 GB of RAM with the balloon device pinned so it never shrinks, a 20 GB virtio-scsi disk on ZFS, UEFI firmware and no swap. Each got the same prep (full upgrade, reboot, apt timers disabled so background updates couldn’t skew the idle numbers). One VM stayed bare as the baseline. The other four got one distribution each, as a single node that runs both the control plane and workloads.

Each number was taken the same way on every VM. Install time is wall clock from the first install command until the node is Ready and a CoreDNS pod reports ready, with the installs run one at a time on the same uplink so they didn’t fight for bandwidth. Cold boot is qm start until the node shows a Ready heartbeat newer than the boot and a CoreDNS container started after the boot is ready. (My first version of that script read the stale Ready status left over from before the shutdown and said k0s was up in 31 seconds. The API server keeps reporting the old status until the node-monitor grace period runs out, so check timestamps, not just the word “True”.) Idle memory comes from the virtio balloon stats, cross-checked against /proc/meminfo inside each guest, where the two agreed within about 30 MB. CPU is the sar average over 5 minutes, taken twice.

For the RAM floor I stepped each VM down (2048, 1792, 1536, 1280, 1152, 1024 MB) and cold booted it at each size. A pass meant the node came back Ready within 10 minutes, a fresh nginx pod (image already cached) reached Running within 3 minutes, and both were still healthy 60 seconds later. I repeated every boundary at least twice. One detail for reading the table: a VM sized at 1,536 MB shows about 1,439 MB to the guest, because the kernel and firmware take their cut.

3. What each distribution installs by default

This table matters more than the benchmark for most people. I created a LoadBalancer Service and a PVC with no StorageClass on each fresh cluster, waited 90 seconds and wrote down what happened.

Componentk0sk3sMicroK8skubeadm
PackagingSingle 252 MB binarySingle 76 MB binary + install scriptClassic snapdeb packages + a container runtime you install
Control planeSeparate processes run by the k0s supervisorOne k3s-server processOne kubelite processStatic pods
Datastore (single node)kine + SQLitekine + SQLitek8s-dqliteetcd
CNIkube-routerFlannel (built in)CalicoNone (I added Flannel)
Ingress controllerNoneTraefikNone (addon)None
LoadBalancer ServiceStays <pending>ServiceLB gives the node IP on a free port (80 and 443 stay <pending>, Traefik holds them)Stays <pending> (MetalLB addon)Stays <pending>
PVC with no StorageClassPendingBound (local-path)Pending (hostpath-storage addon)Pending
metrics-server (kubectl top)YesYesNo (addon)No
Container runtimeBundled containerdBundled containerdBundled containerdcontainerd from the Ubuntu archive

That’s where most of k3s’s extra 170 MB of RAM goes. It is the only one where an Ingress, a PVC and a LoadBalancer Service on a free port work on a fresh node without installing anything else. A new k3s node lists seven pods, two of them finished Helm install jobs:

k3s default pods on a fresh single-node cluster: CoreDNS, Traefik, ServiceLB, local-path provisioner and metrics-server

Compare that with k0s, which gives you a CNI, kube-proxy, CoreDNS and metrics-server and stops there:

sudo k0s kubectl get pods -A -o wide

Four pods, all in kube-system (the two on the node’s own address are host-network pods):

NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
kube-system   coredns-54f859f7dc-xd796          1/1     Running   0          22m   10.244.0.3      homelab-k0s   <none>           <none>
kube-system   kube-proxy-77wz9                  1/1     Running   0          22m   10.0.1.51       homelab-k0s   <none>           <none>
kube-system   kube-router-7lbvj                 1/1     Running   0          22m   10.0.1.51       homelab-k0s   <none>           <none>
kube-system   metrics-server-57d754cb45-dvk2k   1/1     Running   0          22m   10.244.0.2      homelab-k0s   <none>           <none>

MicroK8s is even leaner with three pods (Calico’s two plus CoreDNS), and everything else is a microk8s enable away. kubeadm gives you the control plane as static pods and nothing more. Until you pick a CNI the node stays NotReady.

4. k3s vs k8s: what kubeadm costs you and what it doesn’t

k3s is upstream Kubernetes packaged into one binary, with SQLite in place of etcd and a set of defaults bolted on. It passes the same conformance tests, so your manifests, Helm charts and kubectl habits carry over unchanged. What changes is the work around them. On my VMs, k3s was a working cluster in 72 seconds from one command:

curl -sfL https://get.k3s.io | sh -

The installer picks the stable channel, and right now that’s one minor behind the newest upstream release:

[INFO]  Finding release for channel stable
[INFO]  Using v1.36.5+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.36.5%2Bk3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.36.5%2Bk3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s

kubeadm took 240 seconds and about twenty commands: kernel modules, a sysctl, containerd with the systemd cgroup driver, the pkgs.k8s.io repo pinned to a minor release, the three packages, then init, a CNI and the taint removal. The last part looks like this, and the kubeadm install guide for Ubuntu has every step before it:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube; sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

Once it’s running, kubeadm is not the heavyweight people assume. It idled at about 460 MB of Kubernetes memory, the same as k0s and 170 MB less than k3s, and it kept running on a 1,152 MB VM where k3s and k0s did not. The catch is the install gate. kubeadm’s preflight check refuses to create a cluster below 1,700 MB of visible RAM, and on a 1,536 MB VM it stops cold:

kubeadm init preflight fails with ERROR Mem the system RAM is less than the minimum 1700 MB on a 1536 MB VM

You can pass --ignore-preflight-errors=Mem, or install on a 2 GB VM and shrink it later. The real kubeadm costs are elsewhere: etcd and the static pods kept 1 to 3% iowait on an idle node (k3s and k0s sat near zero), idle CPU was nearly double that of k3s, and a fresh cluster has no metrics, ingress, LoadBalancer or storage. If you’re learning for a certification or want a cluster that looks exactly like production, kubeadm is worth it. For a homelab node that has to be useful today, k3s gets there in a fraction of the effort.

5. k0s vs k3s: same weight class, different philosophy

Both are single binaries, both use kine with SQLite on a single node (/var/lib/k0s/db/state.db and /var/lib/rancher/k3s/server/db/state.db), both bundle containerd, and both idled at 3 to 4% CPU. The difference is what they assume you want.

k0s gives you a clean upstream control plane and nothing opinionated on top. On a single node it runs kube-apiserver, the controller manager, the scheduler and kine as separate processes under its own supervisor (264 MB of RSS for the API server alone). k3s compiles all of them into one k3s-server process, which sat at 627 MB of RSS. k0s was the lightest on memory (~458 MB) and survived down to 1,280 MB in three runs out of three. It wasn’t comfortable there, though. In one of those runs the node dropped to NotReady during the hold, and metrics-server kept failing its liveness probe, so I’d give it 1.5 GB in real use.

Install was where k0s lost. It took 263 seconds against 72 for k3s, even though the download and install were done in about 46. After the controller service started, the node needed another 106 seconds to go Ready, and CoreDNS did not report ready until almost two minutes after that. The same lag showed up on every cold boot: the node was Ready at 36 seconds and CoreDNS at 76. Install is three commands, and the k0s on Ubuntu walkthrough covers multi-node clusters with k0sctl:

curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo sh
sudo k0s install controller --single
sudo k0s start

k3s’s defaults also have a sharp edge. ServiceLB publishes a LoadBalancer by binding its port on the node, and on a single node Traefik already holds 80 and 443. So the first LoadBalancer Service you create on port 80 stays <pending> forever. Its svclb pod cannot schedule, and the event says exactly why:

k3s ServiceLB pod Pending with didn't have free ports for the requested pod ports, second LoadBalancer on port 8080 gets an address

The same deployment exposed on 8080 got the node’s address straight away and answered with HTTP 200. Either route web traffic through Traefik with an Ingress, or install k3s with --disable=traefik if you’d rather bring your own ingress controller. The k3s install guide covers the server flags and config file.

6. MicroK8s: check the channel before you install

The first thing MicroK8s did was install an older Kubernetes than I asked for. A plain snap install follows the snap’s default track, and that track is still 1.35 even though the 1.36 stable track has been out for months:

sudo snap install microk8s --classic

Note the track in brackets:

microk8s (1.35/stable) v1.35.6 from Canonical** installed

Always name the channel. I purged that install (snapd and the core22 base included) and timed a clean one instead, which reached a working cluster in 166 seconds:

sudo snap install microk8s --classic --channel=1.36/stable
sudo usermod -a -G microk8s $USER
sudo microk8s kubectl get nodes

The group change only applies to new logins, so use sudo until you log out and back in (or run su - $USER). Idle, MicroK8s was the busiest of the four at about 10% CPU on two cores, with five calico-node processes and the k8s-dqlite datastore doing steady writes. It was also the least predictable on cold boot: 45, 94 and 91 seconds to working DNS across three boots, a wider spread than any of the others. In its favor, dqlite gives you HA with three nodes and no external etcd, and the addon system (microk8s enable hostpath-storage, metallb, ingress) is the tidiest way to bolt features on. The MicroK8s install guide walks through the addons.

The 540 MB minimum on the getting-started page did not survive contact with my lab, and the next section shows how it failed.

7. How much RAM k0s, k3s and MicroK8s really need

None of the four crashed cleanly when RAM ran out. With no swap, the kernel throws away page cache to keep the processes alive, the API server slows to a crawl and the node stays technically up but useless. Every boundary I tested:

DistributionLast passing sizeFirst failing sizeWhat failure looked like
k0s1,280 MB (3 of 3)1,152 MB (0 of 2)Node never came back Ready in 10 min; API returned “TLS handshake timeout”; memory PSI full ~31%, load average 18 to 20
k3s1,536 MB (3 of 3)1,280 MB (1 of 2)CoreDNS, Traefik and metrics-server crash-looping (11 to 13 restarts); at 1,024 MB the kernel OOM-killed Traefik
MicroK8s1,792 MB (2 of 2)1,536 MB (0 of 2)API handler timeouts, “leaderelection lost”, kubelite exits and restarts; no OOM kill, about 600 MB still available
kubeadm1,152 MB (2 of 2)1,024 MB (0 of 2)Node Ready after 80 to 220 s, then API refused connections; new pod never ran

k3s at 1,024 MB is what thrashing looks like from inside. Only 16 MB free, a load average of 15 on two cores, and the “full” memory pressure line saying every task on the box was stalled waiting for memory about a third of the time. The kernel eventually killed Traefik, which runs as a BestEffort pod with oom_score_adj 1000, so it’s always first in line:

k3s on a 1024 MB VM: load average 15, memory PSI full above 30 percent and the kernel OOM killer kills Traefik

MicroK8s failed differently, and this one is worth understanding if you run it on small hardware. At 1,536 MB there was no OOM kill at all and 600 MB was still available. But the API server, backed by dqlite, answered slowly enough (90 “Handler timeout” errors in one boot) that the controller manager could not renew its leader lease. In MicroK8s the controller manager lives inside the same kubelite process as the API server and kubelet, so losing the lease takes the whole node down. systemd restarted it three times in ten minutes:

MicroK8s on a 1536 MB VM: kubelite logs leaderelection lost and systemd restarts the whole service

Read the published minimums carefully before comparing. The k0s 1 GB figure covers k0s alone with the operating system on top, and k3s’s 2 GB excludes workloads, while my floors are whole-VM sizes that include about 440 MB of Ubuntu. On that basis k0s lands close to its documentation. MicroK8s doesn’t. My practical numbers: 1.5 GB for k0s, 2 GB for k3s and kubeadm, and 2 GB as the absolute floor for MicroK8s, with 4 GB if you plan to run anything real on it. These are x86_64 VMs. ARM boards will land in different places, but the order should hold. For the operating system layer under the cluster, the best OS for Kubernetes benchmark ran the same RAM step-down on Talos, Flatcar and four others.

8. Which one to run

For a homelab node, an edge box or anything where you want Ingress, LoadBalancer and storage working the moment it boots, pick k3s. It installs fastest and has DNS back soonest after a reboot, and its extra 170 MB of RAM buys a usable cluster. Disable Traefik if you’re bringing your own ingress. When local-path stops being enough, the k3s and RKE2 storage guide covers Longhorn and NFS.

Take k0s when you need the lowest RAM you can both install and run on (kubeadm runs leaner once installed, but not below its 1,700 MB install gate), or when you want a vanilla control plane with nothing opinionated installed and a k0sctl workflow for adding nodes. It had the lowest memory and idle CPU of the four. kubeadm is the right call when the cluster has to match what you run in production or what a CKA lab expects. Give it 2 GB to get through preflight. MicroK8s fits Ubuntu shops that like snaps and want three-node HA without managing etcd, but size it at 4 GB and pin the channel.

Two outside options fill gaps these four don’t. Minikube is better for throwaway clusters on a laptop, and RKE2 is k3s’s hardened sibling if you need CIS defaults and etcd.

9. Removing each one cleanly

I uninstalled all four at the end and checked what was left. k3s cleaned up best. Its installer drops an uninstall script that removes the service, the binary, the data and the network interfaces:

/usr/local/bin/k3s-uninstall.sh

Afterwards the root filesystem was within 40 MB of the bare baseline. The only things left were /etc/rancher/node/password and an empty /var/lib/rancher.

k0s needs a stop and a reset:

sudo k0s stop
sudo k0s reset

The reset removes /var/lib/k0s and the systemd unit, then warns “To ensure a full reset, a node reboot is recommended.” It leaves /etc/k0s and the 252 MB binary in /usr/local/bin, so delete those by hand.

MicroK8s goes with one snap command. The core22 base snap and snapd stay installed, and the root filesystem was still about 520 MB above the bare baseline afterwards, so remove those too if nothing else on the box uses snaps:

sudo snap remove --purge microk8s

kubeadm leaves the most behind. kubeadm reset empties the etcd data directory and stops the kubelet, but it says outright that it “does not perform cleanup of CNI plugin configuration, network filtering rules and kubeconfig files”. The packages are held, so purging them needs --allow-change-held-packages:

sudo kubeadm reset -f
sudo apt-get purge -yq --allow-change-held-packages kubeadm kubelet kubectl
sudo rm -rf /etc/cni/net.d $HOME/.kube

Even after that, /etc/kubernetes with only empty subdirectories (pki, manifests), an empty /var/lib/etcd, containerd with its cached images (the root filesystem stayed 1.2 GB above baseline), and the flannel.1 and cni0 interfaces were still there until the next reboot. Purge containerd separately if you’re done with containers on that machine.

Keep reading

Best UI Applications for Managing Docker Containers Containers Best UI Applications for Managing Docker Containers Install Docker and Run Containers on Ubuntu 24.04|22.04 Containers Install Docker and Run Containers on Ubuntu 24.04|22.04 Install UniFi OS Server on Ubuntu 24.04 LTS Containers Install UniFi OS Server on Ubuntu 24.04 LTS CapRover Docker Install on Debian 13: Deploy Your First App Containers CapRover Docker Install on Debian 13: Deploy Your First App Best OpenShift Books for Admins and EX280 Candidates Books Best OpenShift Books for Admins and EX280 Candidates How To Install Docker CE on RHEL 7 Linux Containers How To Install Docker CE on RHEL 7 Linux

Leave a Comment

Press ESC to close