Unlike COW filesystems like BTRFS and ZFS, traditional Linux filesystems like ext4 and XFS have no way to detect slow data degradation (also known as bitrot), since they don’t perform data checksumming. When using those filesystems, however, we can store and verify integrity information at the block level, using dm-integrity. In this tutorial, we learn how to create dm-integrity devices with the integritysetup utility, and when creating luks containers, using cryptsetup.
In this tutorial you will learn:
- How to create dm-integrity devices with integrity setup on a single disk
- How to create dm-integrity devices on data-redundancy setups (a RAID1 array)
- How to implement dm-integrity functionalities when creating a LUKS container
| Category | Requirements, Conventions or Software Version Used |
|---|---|
| System | Distribution-agnostic |
| Software | integritysetup/cryptsetup |
| Other | Root privileges |
| Conventions | # – requires given linux-commands to be executed with root privileges either directly as a root user or by use of sudo command$ – requires given linux-commands to be executed as a regular non-privileged user |
Simulating bitrot on traditional Linux filesystems
As we already said, traditional Linux filesystems like ext4 and XFS, have no way to verify data-integrity, and are not able to spot data-corruptions. Let’s demonstrate this. As a first thing, we create a file with dd, allocating 512 MiB of space:
$ dd if=/dev/zero of=blkdevice bs=1M count=512
With the command above, we instructed dd to read 512 blocks of 1MiB each from the
/dev/zero pseudo-device, and write them on the “blkdevice” file, in the current working directory. Once we created the file, we can use it as a block device, and build a filesystem on it. For the sake of this example we will use ext4:
$ mkfs.ext4 blkdevice
We can mount the filesystem just as if we were dealing with any other “real” block device:
$ sudo mount blkdevice /mnt
At this point, we write random data to a file, letting it fill all the available space:
$ sudo dd if=/dev/urandom of=/mnt/testfile
We should get an output similar to the one below:
dd: writing to '/mnt/testfile': No space left on device
976521+0 records in
976520+0 records out
499978240 bytes (500 MB, 477 MiB) copied, 3.95061 s, 127 MB/s
Let’s now get the checksum of the file we just created:
$ sha256sum /mnt/testfile
In this case, we obtain the following result:
22eba8b04d389cd8f89ce055794823baa85c196e7ef81cc2465fb2a50e0f4f1e /mnt/testfile
We take note of it, and umount the filesystem:
$ sudo umount /mnt
Now, to simulate data corruption, or bitrot, we can write random data directly on the block device. In the example below, we specify that we want to write a single output block (count=1) of 1MiB (bs=1M), and we want to start writing after an offset of 15 output blocks of the same size, from the beginning of the target device (seek=15). We do this to ensure we cover the area of the filesystem which hosts the “testfile”. Finally, we use conv=notrunc to ensure our “block device” is not truncated:
$ dd if=/dev/urandom of=blkdevice seek=15 count=1 bs=1M conv=notrunc
At this point, we mount the filesystem again, and check if the checksum of “testfile” changed:
$ sudo mount blkdevice /mnt
$ sha256sum /mnt/testfile
Here is the result we get:
fb91581b5ed08eae9f3f8400d27bed0b4ad5a8fb4742f2fbabcc0ed409792160 /mnt/testfile
As expected, the checksum of the file is now different. The change happened silently: traditional filesystems don’t keep track of data checksums, therefore they have no way to notice the “bitrot”. Here is where dm-integrity can help.
Using dm-integrity on a single disk
Using dm-integrity on a single disk, is not that useful: while on a setup with multiple data copies, such as a RAID1, corrupted data can automatically be replaced with an intact copy, on a single disk setup, this is, of course, impossible. Being able to notice a corruption, however, can still be useful, perhaps as a sign of a possible failing disk.
Installing the integritysetup utility
To use dm-integrity, we need to format our device using the integritysetup utility. To install it on Fedora and other distributions of the Red Hat family, we can run:
$ sudo dnf install integritysetup
On OpenSuse (Leap/Tumbleweed), we can use the “zypper” to install the package with the same name:
$ sudo zypper in integritysetup
On Debian and Debian-based distributions, the utility is provided by the “cryptsetup-bin” package. We can install it by running:
$ sudo apt install cryptsetup-bin
On Archlinux the integritysetup utility is included in the “cryptsetup” package, available in the “core” repository:
$ sudo pacman -S cryptsetup
Creating a dm-integrity volume
To create a dm-integrity volume, we use the “integritysetup” utility to format a partition/block device. Following our previous examples, we operate on the existing “blkdevice” file, treating it as a block device. To format it, we use the following command:
$ sudo integritysetup format blkdevice
We will be asked to confirm we want to perform the operation, since it will ovewrite all data on the device:
WARNING! ======== This will overwrite data on blkdevice irrevocably. Are you sure? (Type 'yes' in capital letters): YES
Once we created the dm-integrity volume, we must open it, and provide a device-mapper name for it. For the sake of this tutorial, we use “testdevice”:
$ sudo integritysetup open blkdevice testdevice
At this point we can create a new filesystem on the mapped device (
/dev/mapper/testdevice):
$ sudo mkfs.ext4 /dev/mapper/testdevice
Let’s once again, mount the filesystem, and create the “testfile” file, filling it with random data:
$ sudo mount /dev/mapper/testdevice /mnt $ sudo dd if=/dev/urandom of=/mnt/testfile
Just as before, we keep track of the checksum of the file:
$ sha256sum /mnt/testdevice
3b0d6256f991d4a352ed9181621813ba6b909b5cab0547a9554cac400074c3a5 /mnt/testdevice
It’s time to unmount the filesystem, close the integritysetup device, and simulate the data corruption, again:
$ sudo umount /mnt $ sudo integritysetup close testdevice
$ dd if=/dev/urandom of=blkdevice seek=15 count=1 bs=1M conv=notrunc
At this point, we can re-mount the filesystem, and try to get the checksum of the file:
$ sudo integritysetup open blkdevice testdevice $ sudo mount /dev/mapper/testdevice /mnt $ sha256sum /mnt/testfile
As soon as we try to read the file, we get an error:
sha256sum: /mnt/testdevice: Input/output error
What happened? Since we are now using dm-integrity, the data corruption was spotted, and the filesystem returned an Input/output error. We can find further details about the incident by peeking at the kernel logs:
$ sudo dmesg | tail -5
We should be able to read something similar to:
[32942.115228] device-mapper: integrity: dm-8: Checksum failed at sector 0x5782
[32942.117906] device-mapper: integrity: dm-8: Checksum failed at sector 0x5758
[32942.119440] device-mapper: integrity: dm-8: Checksum failed at sector 0x5758
[32942.119532] device-mapper: integrity: dm-8: Checksum failed at sector 0x5758
[32942.119614] device-mapper: integrity: dm-8: Checksum failed at sector 0x5758
Using dm-integrity with data redundancy
As we already said, on a single disk setup, we have no way to fix corruptions, since we have just a single copy of the data. Things changes when using a data-redundancy setup such as RAID1. Let’s simulate it. First, we create two files: blkdevice1 and blkdevice2. We will use them as block devices:
$ dd if=/dev/zero of=blkdevice1 bs=1M count=512 $ dd if=/dev/zero of=blkdevice2 bs=1M count=512
On each of them, we create a dm-integrity volume. Below we use the -q option (short for --batch-mode) to skip the confirmation prompt:
$ sudo integritysetup -q format blkdevice1
$ sudo integritysetup -q format blkdevice2
Let’s open them:
$ sudo integritysetup open blkdevice1 testdevice1 $ sudo integritysetup open blkdevice2 testdevice2
We can now assemble the devices into a RAID1 with the mdadm utility:
$ sudo mdadm --create /dev/md0 --level 1 --raid-devices=2 /dev/mapper/testdevice1 /dev/mapper/testdevice2
We create a new array using the --create option, and provide the path of the device: /dev/md0. We specify the RAID level using the --level option, the number of active devices using --raid-devices, and, finally, we pass the path of the block devices we want to include into the array. Once the RAID is assembled, we can create a filesystem on the resulting device:
$ sudo mkfs.ext4 /dev/md0
Now, let’s mount the filesystem we just created, write a file filled with random data, and create the data corruption on the first of the two disks which are part of the array:
$ sudo mount /dev/md0 /mnt $ sudo dd if=/dev/urandom of=/mnt/testfile
$ dd if=/dev/urandom of=blkdevice1 seek=15 count=1 bs=1M conv=notrunc
We can now perform a data-scrub: by writing “test” to the
/sys/block/md0/md/sync_action file, we make the RAID read all blocks of data on all devices:
$ echo "check" | sudo tee /sys/block/md0/md/sync_action
As we saw before, integrity errors are reported in the kernel logs. Here is the relevant part:
[36735.559800] md: check of RAID array md0
[36735.579268] device-mapper: integrity: dm-8: Checksum failed at sector 0x5b00
[36735.579465] device-mapper: integrity: dm-8: Checksum failed at sector 0x5b80
[36735.579508] device-mapper: integrity: dm-8: Checksum failed at sector 0x5980
[36735.579522] device-mapper: integrity: dm-8: Checksum failed at sector 0x5a00
[36735.579533] device-mapper: integrity: dm-8: Checksum failed at sector 0x5c80
[36735.579544] device-mapper: integrity: dm-8: Checksum failed at sector 0x5900
[36735.579606] device-mapper: integrity: dm-8: Checksum failed at sector 0x5a80
[36735.579617] device-mapper: integrity: dm-8: Checksum failed at sector 0x5c00
[36735.581460] device-mapper: integrity: dm-8: Checksum failed at sector 0x5780
[36735.582377] device-mapper: integrity: dm-8: Checksum failed at sector 0x5e80
[36737.803123] md: md0: check done.
At this point, if we run the data-scrub again and check kernel logs, we should notice no error messages:
[37245.887325] md: check of RAID array md0
[37248.101670] md: md0: check done.
Thanks to dm-integrity the data corruption has been spotted and automatically repaired.
Using dm-integrity with LUKS
LUKS (Linux Unified Key Setup) is the standard for disk encryption on Linux. When formatting a raw block device or a partition as a LUKS container, we can specify we want to use dm-integrity without using integritysetup. All we have to do, is to invoke cryptsetup with the --integrity option, and pass the integrity algorithm we want to use, as argument:
$ sudo cryptsetup luksFormat --type luks2 --integrity hmac-sha256 blkdevice1
Conclusions
In this tutorial, we learned how to use dm-integrity to enable data checksumming and verification at the block level, when using traditional Linux filesystems. Being able to verify corrupted data on a storage media (bitrot), is really important, and is especially useful on multi-disks setup, where spotted data-corruptions can be automatically repaired. Enabling dm-integrity, however, is not cheap: among the other things, it causes write-amplifications, since, by default, it writes sector data and integrity tags into a journal. In this article, for obvious reasons, we didn’t mentioned all the possible options which can be used to fine-tune dm-integrity. To learn more about it, please take a look at the integritysetup manpage, and the relevant documentation.
