You have four disks and one question: build the array with mdadm, let LVM do the RAID, or hand everything to ZFS. We built all of them on the same disks in a homelab VM, loaded 4 GiB of data and pulled a disk. ZFS rebuilt in 16 seconds because it copied only the 2.04 GiB that disk held. mdadm and LVM RAID rebuilt the whole 10 GiB member, and mdadm did that even on an empty array.
This RAID vs LVM vs ZFS comparison covers mdadm RAID5 with XFS, LVM RAID5 from lvcreate --type raid5, an LVM thin pool on the mdadm array (LVM on RAID), and ZFS raidz1. Each got the same fio job, dataset, disk pull and snapshot test, so you can compare space, rebuild time, snapshot cost and commands. The disks are virtual on a shared host, so you should trust the speed rows less, and we say where.
Ran all four stacks on Ubuntu 26.04 with mdadm 4.5, LVM 2.03.31, OpenZFS 2.4.1 and fio 3.41 in October 2026; every number below came off that one box.
Which one should you use?
For a NAS or backup box holding files you care about, use ZFS. It rebuilt fastest in every round we ran, its snapshots cost nothing until data changes, and it checksums every block by default, where LVM only does that if you turn on dm-integrity and mdadm never does. Pick by what the box has to do, not by which tool sounds more serious:
- NAS, media library or backup target: ZFS raidz1 or raidz2
- Database or VM host on a RHEL family distro with no ZFS packages: mdadm RAID plus LVM
- A single disk or a cloud VM where the hypervisor already handles redundancy: plain LVM
- One box where you want RAID and volumes from a single tool: LVM RAID, if you keep a disk’s worth of free extents for repairs
- Boot drives: mdadm RAID1, because every installer and initramfs understands it
Already running mdadm and LVM on a RHEL family server? Keep it, and you lose nothing on the volume side. Skip ZFS when your distro does not ship it in the main archive or your support contract only covers in-tree kernel modules. On Ubuntu that is not a concern, because the kernel package already carries the ZFS module and zfsutils-linux is one apt install away. On Rocky or AlmaLinux it means a third-party repository and a module that has to match every kernel update, which is why the mdadm plus LVM stack is still the default there.
mdadm vs LVM vs ZFS: the measured results
Every row below comes from the lab logs. The fio columns are medians of three runs on each stack, and the range is in brackets so you can see how much the shared host moved them:
| Measured on 3 x 10 GiB disks + 1 spare | mdadm RAID5 + XFS | LVM RAID5 + XFS | LVM thin on mdadm RAID5 | ZFS raidz1 |
|---|---|---|---|---|
| Usable space (df) | 20 GiB | 20 GiB | 17.94 GiB pool, 15 GiB thin LV | 19.0 GiB (zpool list shows 29.5G raw) |
| Initial sync before full redundancy | 57 s, 195 s on a later run | 150 s, 68.6 s on a later run | inherits the md sync | none (pool ready in 2.6 s) |
| Seq write 1M, MiB/s | 793 (532 to 837) | 66 (38 to 74) | 146 (119 to 152) | 91 (81 to 92) |
| Seq read 1M, MiB/s | 5,185 (86 to 5,265) | 3,631 (2,325 to 3,677) | 622 (468 to 1,497) | 593 (162 to 1,208) |
| Random read 4k, IOPS | 61.5k (58.6k to 62.3k) | 56.0k (55.3k to 56.2k) | 44.2k (43.2k to 44.7k) | 3,993 with O_DIRECT, 33.9k through the ARC |
| Copy the 4 GiB dataset | 35 s | 1 min 30 s | 55 s | 1 min 5 s |
| Rebuild after pulling one disk | 62.6 s, 244 s on a later run (always the full 10 GiB) | 216.8 s, 215.6 s on a later run | same md rebuild as column 1 | 16.3 s, 78.9 s on a later run (2.04 GiB copied) |
| Snapshot create time | no snapshots | 2.16 s | 0.28 s | 0.62 s |
| Snapshot space up front | n/a | 2 GiB reserved on a non-redundant PV | 0 | 0 |
| Space after overwriting 1 GiB under a snapshot | n/a | 1.0 GiB of the 2 GiB reserve | 1.0 GiB of the pool | 1024M |
| 1 GiB overwrite, no snapshot vs snapshot held | n/a | 25.3 vs 15.2 MiB/s | 764 vs 107 MiB/s | 37.5 vs 37.5 MiB/s |
| Commands from blank disks to a mounted filesystem | 3 | 5 | 7 | 2 |
| Commands to replace the failed disk | 1 (mdadm –add) | 2 (lvconvert –repair, vgreduce) | 1, on the md layer | 1 (zpool replace) |
You should read the speed rows for ordering only. Even then, trust only the large gaps. These are virtual disks on a Proxmox host shared with other VMs, so the host page cache answered some reads and throttled some writes. The mdadm array proves it. We rebuilt it later in the session, and the same 1M sequential write fell from 793 MiB/s to 207 MiB/s, then to 23 to 28 MiB/s, with nothing changed inside the guest. A sequential read that swings from 86 to 5,265 MiB/s on one array is the host cache talking, not the disks, so we would not pick a stack on that column at all.
Two results survive that noise because they repeated in both rounds, and two more held up as back to back comparisons inside one stack:
- Repeated: ZFS resilvered three to four times faster than mdadm, at 4 GiB used of 20 GiB
- Repeated: O_DIRECT 4k reads on ZFS ran at a tenth to a fifteenth of the others
- Measured once: thin snapshots made overwrites seven times slower (764 down to 107 MiB/s)
- Measured once: ZFS wrote at the same 37.5 MiB/s with or without a snapshot
The two snapshot rows were each one pair of runs, a minute apart in the same host window, so read them as a ratio within that stack rather than against the other columns. The rebuild gap comes from what each tool knows. A resilver walks only allocated blocks, while mdadm and LVM RAID write every sector of a replacement disk because they have no idea which ones hold files. That advantage shrinks as the pool fills, and a nearly full raidz pool can resilver as slowly as an md member, or slower, because a resilver walks the block tree with random reads. The snapshot result works the same way in reverse: ZFS is copy on write anyway, so a snapshot only stops old blocks from being freed. The random read result is real for O_DIRECT but not for how a NAS normally reads, and Step 5 explains why we would not judge ZFS on it.
How the three tools differ
mdadm only does redundancy, LVM only does volumes unless you ask it for RAID, and ZFS does both plus the filesystem. That split decides how many layers you manage, and it explains most of the table above, since each extra layer is another set of commands and another place a rebuild can go wrong:
| Feature | mdadm | LVM | ZFS |
|---|---|---|---|
| Redundancy | RAID 0, 1, 4, 5, 6, 10 | raid1, raid4/5/6, raid10 through dm-raid (same md kernel code) | mirror, raidz1/2/3, draid |
| Volume management | No | Yes (PV, VG, LV) | Yes (pools and datasets) |
| Filesystem | Needs ext4 or XFS on top | Needs ext4 or XFS on top | Built in |
| Snapshots | No | Thick (fixed COW reserve) or thin | Yes, no reserve |
| Checksums on data | No | No by default; optional dm-integrity with –raidintegrity y (per block, scrubbed with lvchange –syncaction check, but not file or snapshot aware) | Yes, verified on every read and scrub |
| Rebuild copies | The whole member | The whole image | Allocated blocks only |
| Shrink | Rarely: shrink the filesystem, then the array | Yes with ext4, never with XFS | No for raidz vdevs |
| Kernel code | In tree (md) | In tree (device mapper) | Out of tree module |
Two rows trip people up. LVM is not “volumes only”: since the dm-raid target landed it builds real RAID5 and RAID6 with the same md personality mdadm uses, which is why its rebuild behaves like mdadm’s. And by default only ZFS can tell you a block came back wrong, because mdadm and plain LVM RAID trust whatever the disk returns. LVM can close part of that gap: lvcreate --type raid5 --raidintegrity y puts dm-integrity under each leg, so a bad block is detected and rewritten from the other legs. On our VM that added a 20 MiB imeta LV per 1 GiB leg, and it works per block, so it has no idea which files or snapshots a bad block belongs to. Whichever filesystem goes on top of mdadm or LVM, the ext4 vs XFS benchmarks cover that choice separately.
Prerequisites
The lab is small on purpose, so the rebuilds finish in minutes. What it ran on:
- Ubuntu 26.04 LTS VM, 2 vCPU, 4 GB RAM, boot disk on vda
- Four empty 10 GiB data disks on virtio-scsi (sda to sdd)
- Packages: mdadm, lvm2, zfsutils-linux, fio, xfsprogs, thin-provisioning-tools
- A user with sudo
That spec is a floor for following along, not a recommendation for a real NAS. Size your real array from the data you will store plus about 20% free space, because ZFS slows down as a pool fills. Your rebuild window matters as much: a 16 TB member needs about 22 hours even at mdadm’s default 200 MB/s ceiling (speed_limit_max). You also need RAM for the ZFS ARC, which on this 4 GB VM claimed a 3.0 GB ceiling by default. With large drives that rebuild window is the reason to go to raidz2 or RAID6, and our NAS hard drive picks list the CMR models worth buying by price per TB.
Ubuntu 24.04 ships the same three tools (the older OpenZFS 2.2 line in its archive), but we only ran this lab on 26.04, so the timings belong to the newer release.
Step 1: Prepare the disks and the test files
Install the tools first. On Ubuntu you get all of them from the main archive:
sudo apt update
sudo apt install -y mdadm lvm2 zfsutils-linux fio xfsprogs thin-provisioning-toolsEvery step below destroys whatever is on your data disks, so find them by their stable ID before you type anything. Kernel names like sdc can change after a reboot or a hot plug, and we watched exactly that risk play out when a pulled disk came back. List the IDs and check which disk holds the root filesystem:
ls -l /dev/disk/by-id/ | grep -v part
findmnt -no SOURCE /Our four data disks showed up as the QEMU scsi1 to scsi4 IDs, and the root filesystem lived on vda, so none of the commands that follow touch it:
lrwxrwxrwx 1 root root 9 Oct 16 14:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 -> ../../sda
lrwxrwxrwx 1 root root 9 Oct 16 14:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 -> ../../sdb
lrwxrwxrwx 1 root root 9 Oct 16 14:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi3 -> ../../sdc
lrwxrwxrwx 1 root root 9 Oct 16 14:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi4 -> ../../sdd
/dev/vda1Put the four IDs in shell variables so the rest of the guide pastes as written. Disks 1 to 3 form each array, and disk 4 stays empty as the replacement for the disk we pull. On real hardware the IDs carry the model and serial number, so swap in your own:
export DISK1=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
export DISK2=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2
export DISK3=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi3
export DISK4=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi4
for d in "${DISK1}" "${DISK2}" "${DISK3}" "${DISK4}"; do echo "$d -> $(readlink -f $d)"; doneEach line you see should resolve to an sd device, and none of them to vda. The variables only last for this shell, so export them again if you reconnect. Next, create the fio job file that every stack runs:
vim ~/storage-bench.fioThis is the exact file we ran. It writes and reads a 2 GiB file with 1M blocks, then runs 60 seconds of 4k random reads, all with direct=1 so the guest page cache stays out of it. The ${TESTDIR} line is expanded by fio from the environment:
[global]
ioengine=libaio
direct=1
size=2G
filename=${TESTDIR}/fio.dat
group_reporting=1
[seq-write-1m]
rw=write
bs=1M
iodepth=16
[seq-read-1m]
stonewall
rw=read
bs=1M
iodepth=16
[rand-read-4k]
stonewall
rw=randread
bs=4k
iodepth=32
runtime=60
time_based=1The rebuild test needs real data on each stack, otherwise ZFS has nothing to resilver and the comparison means nothing. We generated eight 512 MiB files of random data on the boot disk, so compression could not shrink them, and kept their checksums to verify each stack after the rebuild:
sudo mkdir -p /var/tmp/dataset
for i in 1 2 3 4 5 6 7 8; do sudo dd if=/dev/urandom of=/var/tmp/dataset/blob$i.bin bs=1M count=512 status=none; done
cd /var/tmp/dataset && sha256sum blob*.bin | sudo tee /var/tmp/dataset.sha256 && cdPlain dd is fine for this buffered copy, even though Ubuntu 26.04 now ships the Rust uutils version of it. Add oflag=direct and that same dd can fail, sometimes silently, which cost us one test run (the details are in the What broke section).
Step 2: Build mdadm RAID5 and pull a disk
mdadm builds the array in one command and is usable straight away, but it runs at reduced redundancy until the initial sync finishes. Create a three disk RAID5 from disks 1 to 3:
sudo mdadm --create /dev/md/data --level=5 --raid-devices=3 "${DISK1}" "${DISK2}" "${DISK3}" --run
cat /proc/mdstatThe array came up as md127 even though we named it /dev/md/data, which is normal for a named array: the name lives under /dev/md/ and the kernel picks the number. The --run flag skips the Continue creating array? question mdadm asks when a disk still carries an old partition table or superblock. Leave it off the first time on disks you are not sure about, so you see that warning. The [UU_] and the recovery line show the parity being built:
Personalities : [raid4] [raid5] [raid6]
md127 : active raid5 sdc[3] sdb[1] sda[0]
20953088 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
[>....................] recovery = 0.2% (29852/10476544) finish=5.8min speed=29852K/secWait for the sync with sudo mdadm --wait /dev/md/data, which blocks until it is done (57 seconds for us the first time). Then format, mount and benchmark it. Run fio as your own user once you own the mount point:
sudo mkfs.xfs /dev/md/data
sudo mkdir -p /mnt/md
sudo mount /dev/md/data /mnt/md
sudo chown "$USER": /mnt/md
TESTDIR=/mnt/md fio ~/storage-bench.fio
rm -f /mnt/md/fio.datEach job prints a summary line. These are from the first of our three runs on the md array, and the 5,185 MiB/s read is the host cache we warned about, not three 10 GiB virtual disks:
write: IOPS=793, BW=793MiB/s (832MB/s)(2048MiB/2581msec); 0 zone resets
read: IOPS=5184, BW=5185MiB/s (5437MB/s)(2048MiB/395msec)
read: IOPS=62.3k, BW=243MiB/s (255MB/s)(14.3GiB/60001msec)Copy your dataset in with sudo cp -r /var/tmp/dataset /mnt/md/. To simulate a dead disk without touching the hypervisor, we deleted disk 3 from the guest kernel, which is what the system sees when a drive drops off the bus. Check that readlink -f "${DISK3}" still prints /dev/sdc before running this, because the command takes the kernel name:
echo 1 | sudo tee /sys/block/sdc/device/delete
sudo dd if=/dev/urandom of=/mnt/md/poke.bin bs=1M count=64 status=none
sync
sudo mdadm --detail /dev/md/data | grep -E "State :|removed"The small write matters: until an I/O hits the missing disk, md still lists it as a member. After that write, mdadm reported the array degraded with slot 2 removed, and the data still read back with matching checksums from the two survivors:
State : clean, degraded
- 0 0 2 removedRebuilding onto your spare is one command. Time it with mdadm --wait, which returns the moment recovery finishes:
sudo mdadm /dev/md/data --add "${DISK4}"
time sudo mdadm --wait /dev/md/dataOur rebuild took 62.6 seconds and wrote the full 10 GiB member, though the array held only 4 GiB of files. mdadm does not know which blocks are in use. A new replacement disk on your half empty 16 TB array takes as long to rebuild as on a full one. The exception is re-adding the same member to an array with a write intent bitmap, which resyncs only the regions that changed while it was out. We checked that later by rebuilding an empty array, which took 244 seconds in a slower moment on the shared host. Before you trust an md array with real data, it also needs these:
- The array line saved in
/etc/mdadm/mdadm.conf - An initramfs rebuild with
update-initramfs -u - mdmonitor running with a mail address for alerts
We did not reboot this lab, so those steps are untested here, but our mdadm software RAID guide walks through each one with a reboot test.
Step 3: Build LVM RAID5 and repair it
LVM RAID puts all four disks in one volume group and lets the logical volume carry the RAID level, so the spare is just free space LVM can repair onto. The -i 2 means two data stripes, and the lvmraid man page sets three devices as the minimum for raid5. Listing disks 1 to 3 at the end keeps disk 4 free:
sudo pvcreate "${DISK1}" "${DISK2}" "${DISK3}" "${DISK4}"
sudo vgcreate vg_raid "${DISK1}" "${DISK2}" "${DISK3}" "${DISK4}"
sudo lvcreate --type raid5 -i 2 -l 100%PVS -n data vg_raid "${DISK1}" "${DISK2}" "${DISK3}"
sudo lvs -a -o lv_name,lv_size,segtype,sync_percent,devices vg_raidEach leg becomes a hidden rimage data LV plus a 4 MiB rmeta LV that holds the RAID superblock and write intent bitmap. Wait until Cpy%Sync reads 100.00 before you put data on it:
LV LSize Type Cpy%Sync Devices
data 19.98g raid5 100.00 data_rimage_0(0),data_rimage_1(0),data_rimage_2(0)
[data_rimage_0] 9.99g linear /dev/sda(1)
[data_rimage_1] 9.99g linear /dev/sdb(1)
[data_rimage_2] 9.99g linear /dev/sdc(1)
[data_rmeta_0] 4.00m linear /dev/sda(0)
[data_rmeta_1] 4.00m linear /dev/sdb(0)
[data_rmeta_2] 4.00m linear /dev/sdc(0) You format and mount it the same way as the md array (mkfs.xfs /dev/vg_raid/data, mount on /mnt/lvraid), run the fio job and copy the dataset in. Then test a classic thick snapshot, which needs a fixed copy on write reserve up front. We gave it 2 GiB:
time sudo lvcreate -s -L 2G -n data_snap vg_raid/data
sudo lvs -o lv_name,lv_size,data_percent,devices vg_raidThe snapshot took 2.16 seconds and, with the RAID legs filling disks 1 to 3, the only free extents were on disk 4. So the reserve landed on a single disk with no redundancy, and after we overwrote 1 GiB of the origin it was half full:
data 19.98g data_rimage_0(0),data_rimage_1(0),data_rimage_2(0)
data_snap 2.00g 50.24 /dev/sdd(0) Two things follow from that output. A thick snapshot that fills to 100% becomes invalid, so you size it to the change rate you expect while it exists. And the first write to each chunk of the origin also copies the old chunk into the reserve, which is why our overwrite speed fell from 25.3 to 15.2 MiB/s. Remove it with sudo lvremove vg_raid/data_snap as soon as the backup that needed it finishes.
The reserve also sits on disk 4, which is the disk LVM wants to rebuild onto, and if you lose disk 4 the snapshot goes with it. We tested the first trap on a fresh 20 GiB raid5 LV with a 2 GiB snapshot on disk 4: after pulling disk 3, the repair refused with Insufficient free space: 2559 extents needed, but only 2047 available. Removing the snapshot let the same lvconvert --repair run, and it resynced in 131 seconds. So remove snapshots before a repair, or keep a whole free disk outside the snapshot’s reach.
Now pull disk 3 the same way as in Step 2, write something so the kernel notices, and let LVM repair the LV onto the free disk. The vgreduce afterwards drops the missing PV from the volume group:
echo 1 | sudo tee /sys/block/sdc/device/delete
sudo dd if=/dev/urandom of=/mnt/lvraid/poke.bin bs=1M count=64 status=none
sync
sudo lvconvert -y --repair vg_raid/data "${DISK4}"
sudo lvs -o lv_name,sync_percent vg_raid/data
sudo vgreduce --removemissing vg_raidWhile degraded, lvs marks the LV partial and shows the lost leg as [unknown], and the data still verified. Without -y, lvconvert --repair stops to ask before it replaces the leg. The repair printed Faulty devices in vg_raid/data successfully replaced. Then it needed 216.8 seconds to resync, and 215.6 seconds on a second round. mdadm took 62.6 and 244 seconds in its two rounds, so on this host the rebuild time between mdadm and LVM RAID was a wash. That fits, since both run the same md personality and rebuild the whole image whatever data sits on it.
Step 4: Put LVM thin on mdadm RAID
LVM on RAID is the layout most RHEL family servers run: mdadm handles the disks, and LVM carves the array into volumes you can grow later. We used a thin pool on top because thin snapshots are the closest LVM gets to ZFS snapshots. On the mdadm array from Step 2 (unmounted first), build the volume group, a pool using 90% of it and a 15 GiB thin volume:
sudo pvcreate /dev/md/data
sudo vgcreate vg_md /dev/md/data
sudo lvcreate --type thin-pool -l 90%FREE -n tpool vg_md
sudo lvcreate -V 15G --thin -n tvol vg_md/tpool
sudo mkfs.xfs /dev/vg_md/tvol
sudo mkdir -p /mnt/thin
sudo mount /dev/vg_md/tvol /mnt/thinpvcreate wiped the old XFS signature from the md device without asking, so check the device name twice before you run it. The pool came out at 17.94 GiB with a 20 MiB metadata LV that was already 10.23% full before any data went in. We watched Meta% during every test, because a thin pool that runs out of metadata stops accepting writes.
The fio job ran slower here than on the bare md array, mostly on writes (146 against 793 MiB/s median). A fresh thin volume allocates each 64 KiB chunk the first time it is written. Each fio run deleted its file, so later runs could land on fresh chunks too. A second write into the same 1 GiB file ran at 764 MiB/s, which points at the first write, not the thin layer itself. We repeated it later on a single-disk thin pool, in a faster host window, and the overwrite was again the fastest of the three. The first write ran at 1,796 MiB/s and the overwrite at 3,657 MiB/s, then 2,138 MiB/s again after rm plus fstrim handed the chunks back. If you want thin volumes for a database, plan for that first-write penalty.
Thin snapshots need no reserve and you give them no size, since they share the pool:
time sudo lvcreate -s -n tvol_snap vg_md/tvol
sudo lvs -o lv_name,lv_size,data_percent,metadata_percent vg_mdCreation took 0.28 seconds and pool usage did not move. LVM also printed three warnings, and the last one matters, because the snapshot doubles the virtual size you have promised against a pool smaller than the volume group:
WARNING: You have not turned on protection against thin pools running out of space.
WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
WARNING: Sum of all thin volume sizes (30.00 GiB) exceeds the size of thin pool vg_md/tpool and the size of whole volume group (19.98 GiB).
Logical volume "tvol_snap" created.Overwriting 1 GiB of the origin while the snapshot existed moved the pool from 35.60% to 41.18%, which is 1.0 GiB, and that overwrite ran at 107 MiB/s against 764 MiB/s just before the snapshot. Every first write to a shared chunk has to allocate a new one, so thin snapshots are cheap to take and expensive to keep under a write heavy workload. Setting thin_pool_autoextend_threshold in /etc/lvm/lvm.conf only helps when dmeventd is monitoring the pool and the volume group has free extents, per the lvmthin man page. This VG had about 2 GiB left after the 90% pool, so autoextend could have bought one small step, not a rescue. The md rebuild under this stack is the same one Step 2 measured, and our older walkthrough on LVM on a RAID 1 device shows the mirrored version of this layout.
Step 5: Build ZFS raidz1 and resilver it
ZFS needs two commands to get from blank disks to a mounted, redundant filesystem: one for the pool and one for a dataset. You should use the by-id paths, because the pool records them and zpool status then tells you the exact drive to pull. Set ashift=12 for 4K sector drives, since it cannot be changed after the vdev exists:
sudo zpool create -o ashift=12 tank raidz1 "${DISK1}" "${DISK2}" "${DISK3}"
sudo zfs create tank/data
sudo zpool list tank
sudo zfs list tank/dataThe two sizes disagree, and both are right. zpool list counts raw space including parity, and zfs list shows what you can actually store:
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 29.5G 1.17M 29.5G - - 0% 0% 1.00x ONLINE -
NAME USED AVAIL REFER MOUNTPOINT
tank/data 128K 19.0G 128K /tank/dataPlan your capacity from the 19.0G figure. There was no initial sync to wait for, because raidz writes parity with the data and never builds it for empty space. The OpenZFS pool concepts page notes that raidz does not suffer from the RAID5 write hole. Compression is on (lz4) by default, but our random data stayed at a 1.00x ratio, so the comparison with the other stacks is fair.
The fio job on ZFS showed the one result here that looks bad on paper: 3,993 IOPS of 4k random reads with direct=1, against 61.5k on mdadm. We first suspected read amplification from the default 128K recordsize, but a smaller recordsize did not help in our runs. Because recordsize only applies to files written after it is set, we repeated the test on two fresh datasets with a new file each: 3,810 IOPS at 128K and 3,421 at 16K. The same test without direct=1, served from the ARC the way a NAS actually reads, gave 33.9k IOPS. So we treat the O_DIRECT number as a property of ZFS’s direct I/O path on this box, and we would not judge ZFS for file serving on it.
Snapshots on ZFS are instant and free until data changes. We took one, overwrote 1 GiB, and the snapshot then held exactly the old 1 GiB:
sudo zfs snapshot tank/data@before
sudo zfs list -t all -o name,used,refer,usedbysnapshots -r tank/dataCreation took 0.62 seconds, and the overwrite ran at 37.5 MiB/s both with and without the snapshot, because ZFS writes changed blocks to new space either way:
NAME USED REFER USEDSNAP
tank/data 6.00G 5.00G 1024M
tank/data@before 1024M 5.00G -Now the disk pull. After deleting disk 3 from the kernel and writing to the pool, ZFS marked the member REMOVED and the pool DEGRADED, and the dataset still verified. Replace it with disk 4 and wait for the resilver:
sudo zpool replace tank scsi-0QEMU_QEMU_HARDDISK_drive-scsi3 "${DISK4}"
time sudo zpool wait -t resilver tank
sudo zpool status tankThe resilver finished in 16.3 seconds, and the scan line explains why it beat both RAID rebuilds by so much:
ZFS resilvered 2.04G, which is the share of the 4 GiB dataset plus parity that lived on the lost disk, and nothing else. On the second round in the slower host window the same resilver took 78.9 seconds, still about three times faster than the mdadm rebuild in that window. Expect that gap to close as your pool fills, since a full pool has as much to copy as an md member and reads it in a less sequential order. The zpool replace -s sequential mode is not supported for raidz either.
A zpool scrub -w tank afterwards read and checksummed every block in 10.7 seconds with 0 errors. On RAID5 and mirrors an mdadm check can only say the copies disagree, not which one is right, and LVM gets block checksums only with dm-integrity enabled. For wider layouts, our notes on ZFS RAID levels on Proxmox VE and pool design with mirror, raidz and draid pick up from here.
What broke in the lab
Four things went wrong while we moved the same disks between stacks. Each one is the kind of message you meet when you reuse disks on a real box, so each entry below gives the cause and the fix that worked for us.
/proc/mdstat still showed [UUU] after the disk was gone
Right after we deleted disk 3, /proc/mdstat still listed sdc[3] and [3/3] [UUU], while mdadm --detail already said clean, degraded. md only fails a member when an I/O to it errors, and nothing had written yet. The first write produced md/raid:md127: Disk failure on sdc, disabling device. in dmesg and the status caught up. So do not trust a quiet mdstat as proof of health; run mdadm --detail and keep mdmonitor running so a failure sends an alert.
WARNING: outdated PV /dev/sdc seqno 7 has been removed in current VG vg_raid seqno 10
When the pulled disk came back after a SCSI rescan, it still carried the old LVM metadata, so every LVM command printed this warning plus See vgck --updatemetadata to clear outdated metadata. The returning disk had the same problem under mdadm, where mdadm --examine showed it still claiming Active device 2 with an old event count. Clear stale metadata with the owning tool before you reuse a disk:
sudo vgck --updatemetadata vg_raid
sudo mdadm --zero-superblock "${DISK3}"
sudo zpool labelclear -f "${DISK3}-part1"Run only the line that matches what your disk was used for, and check with readlink -f that you are pointing at a data disk. The vgck call printed WARNING: wiping header on outdated PV /dev/sdc and the warnings stopped.
Cannot use /dev/disk/by-id/…: device is partitioned
ZFS partitions whole disks it is given (a large partition 1 and a small partition 9), and zpool destroy plus labelclear leave that GPT behind. The next pvcreate refused all four disks with this message (we captured it again on a fresh pool), and mdadm only went ahead after warning partition table exists ... but will be lost or meaningless after creating array. Check the target is a data disk and not vda, then remove the partition table:
readlink -f "${DISK1}"
sudo wipefs -a "${DISK1}"Repeat it for each disk you are reusing, never on the disk your root filesystem lives on. wipefs -a erased the two GPT headers and the protective MBR, and pvcreate worked straight after.
dd: IO error: Input/output error
Our first snapshot overwrite used dd ... oflag=direct conv=notrunc, and on Ubuntu 26.04 it failed with this error repeatedly while the kernel log stayed clean. The dd there is uutils coreutils 0.10.0. Under strace a 4 MiB run showed all four writes failing with EINVAL, not an I/O error, yet dd still reported the bytes as copied and exited 0. The cause is most likely an unaligned buffer handed to O_DIRECT. It did not reproduce in three later traced runs on another volume, which all wrote 64 blocks cleanly. That makes it worse than a loud failure, because a script can think it wrote data it never wrote, so use gnudd or fio when you need direct I/O on 26.04.
Tear down each stack and reuse the disks
Each tool has to release the disks in its own order, top layer first, or the next step fails with the device busy. These are the sequences that left our disks clean for the next stack:
sudo umount /mnt/thin
sudo vgremove -y vg_md
sudo pvremove /dev/md/data
sudo mdadm --stop /dev/md/data
sudo mdadm --zero-superblock "${DISK1}" "${DISK2}" "${DISK3}" "${DISK4}"That clears the thin pool and the mdadm array. For LVM RAID, unmount and remove the volume group, then the PV labels:
sudo umount /mnt/lvraid
sudo vgremove -y vg_raid
sudo pvremove "${DISK1}" "${DISK2}" "${DISK3}" "${DISK4}"ZFS needs one more step than it looks like, because destroying the pool leaves the labels and the GPT on every member:
sudo zpool destroy tank
for d in "${DISK1}" "${DISK2}" "${DISK3}" "${DISK4}"; do sudo zpool labelclear -f "${d}-part1"; sudo wipefs -a "$d"; doneRun sudo udevadm settle, then confirm with sudo wipefs -n on each disk. If you get no output, no signature is left. Trust that over lsblk, which kept showing zfs_member from stale udev data on our VM until udev caught up. If you settle on ZFS for a NAS and want a web UI on top instead of the CLI, our TrueNAS SCALE storage server build runs the same pool layouts with alerts and scrub schedules already wired in.
