Given the increasing malware attacks against Linux-based IoT devices, there is growing interest in integrity protection schemes, as well as system update mechanisms that support over-the-air (OTA) field upgrades. At the recent Embedded Linux Conference, Patrick Ohly, a software engineer at Intel GmbH, Germany, who works on the Yocto Project and the IoT Reference OS Kit for Intel(r) architecture, surveyed both topics and explained how they interrelate.

With attacks on the rise, embedded developers need a combination of proactive hardening with integrity protection schemes and regular system updates, among other security precautions. “Integrity protection ensures that your device only runs software that has been verified to be unmodified,” said Ohly. “But you can’t count on catching everything in advance, and there will be new vulnerabilities and attack methods, so that’s why you need system updates.”

The bulk of Ohly’s talk explored the pros and cons of update and protection schemes available for Yocto/OpenEmbedded. Ohly is the maintainer of meta-integrity and meta-swupd — two layers that make the Linux Integrity Measurement Architecture (IMA) integrity protection scheme and the Clear Linux OS swupd update mechanism available on Yocto/OE. The experience has led him to see the benefits of swupd, as well as the problems with IMA.

Ohly explained the differences between block-based (swupdate, Mender.io) and file-based (swupd, OSTree) system update mechanisms. He then compared three integrity protection schemes: IMA with Extended Verification Module (EVM), or IMA/EVM, as well as whole-disk encryption with per-machine secret key, and dm-verity.

One’s choice of system update mechanism can affect the choice of integrity protection scheme. IMA/EVM works only with the file-based swupd and OSTree while dm-verity works only with the block-based Mender.io and swupdate. Whole disk encryption, meanwhile, works with all four. Although this talk is specific to Yocto, all but these components can work with other embedded Linux environments, with the possible exception of the more Yocto-specific swupdate.

Toward the end of the talk, Ohly detailed a recent project in which he took dm-verity and integrated it with whole-disk encryption into the IoT Refkit. “We are trying to use this to extend Yocto to additional use cases,” he said. Here, we only briefly reference the project, which involves integrating LUKS (Linux Unified Key Setup) and finessing QEMU to properly emulate TPM. If you’re interested, the discussion begins about 33 minutes into the 51-minute talk.

System update pros and cons

Before comparing the system update schemes available to Yocto/OE developers, Ohly noted that “they all have pros and cons, and in some cases, need further work.” The first decision is whether to with a block-based or file-based mechanism. Block-based schemes like swupdate and Mender.io have fixed partition sizes. This means “you can’t have an update stream that is supplied to different devices with different hard disks because they have to be partitioned the same way,” said Ohly. File-based mechanisms like swupd and OSTree “make it easier to support a variety of devices,” he added.

In addition, updating selected files instead of entire partitions is a lot faster. “Swupd in particular works really hard to do the minimal amount of work for updates,” said Ohly.

The two file-based mechanisms have further differences. “OSTree creates an alternative tree using hard-linking, and then during reboot it switches over to a new root,” said Ohly. “It’s a bit more atomic than swupd, but it still needs a writable root file system.”

Another point of comparison lies in partition layout. A single partition approach is the default mode for OSTree and swupd, and is supported by swupdate. Mender.io, by contrast, requires an A/B setup in which you have both a live partition and a standby partition. The A/B approach lets you override the standby partition, and after rebooting, switch to the other partition.

“Having a second partition is nice if you get file system corruption,” said Ohly. A/B partitioning is optional on swupdate, and is possible on OSTree and swupd, although it would require a lot more work, he added.

Other issues concern the integration with boot process. “If you have an A/B setup, how do you choose which partition to boot into?” said Ohly. “Can a hacker force us to boot into something we don’t want?” To address this, the mechanisms that currently support A/B setup rely on the U-Boot bootloader and set U-Boot variables, which must be stored by the hardware such that an attacker cannot tamper with them.

One’s choice of update mechanism can affect integration with an OTA update server. “If you have many devices you might want to update a subset, which depends on detecting which devices are currently checking for an update,” said Ohly.

Swupd and OSTree require that clients pull updates anonymously, so they would need additional telemetry to check update status. By comparison, Mender.io offers a dedicated update server. “This allows you to do real device management, so you know if a device has updated,” said Ohly. Swupdate, meanwhile, supports hawkBit, which does something similar.

Integrity protection pros and cons

Ohly went on to compare three integrity protection schemes: IMA/EVM, whole disk encryption, and dm-verity. In a previous project with the Yocto-based Ostro Project, Ohly used IVA/EVM, which like dm-verity is now baked into the Linux kernel. He had a lot of trouble making IVA/EVM more secure, however.

“I concluded that IVA/EVM is not yet ready for production,” said Ohly. When allowing to write files, there’s an increased risk that files become unreadable after a sudden power loss. IMA relies on hashing the modified file content after writing and verifies the checksum when opening a file again. This rehashing only happens on ‘close()’, which isn’t called by long-running processes that keep an sqlite database open. Even when rehashing happens, the checksum and the actual data are not necessarily written to disk together.”

Another problem with IMA/EVM is its lack of directory protection. “There are known attacks because IMA/EVM doesn’t protect the integrity of a directory,” said Ohly. “That allows an attacker to remove a configuration file and replace it with a link to something unprotected. There have been patches to address this, but they have not yet been merged.”

By comparison, the whole disk encryption with per-machine secret key technique “is a lot more mature” than IMA/EVM, said Ohly. With whole disk, when a device is turned off, a malicious hacker sees only “garbage” in the partition, he added. “If the intruder lacks access to the secret key and tries to modify the partition, once the device boots up again, some blocks will be completely modified, so there’s no way to do dedicated file editing.”

Whole disk encryption works great on laptops, but in the embedded realm, the scheme poses the tricky challenge of how to securely store the encryption key. “On a laptop, the user types in something, or plugs in a USB stick, but that doesn’t work in embedded,” said Ohly.

Ohly’s code for the IoT Ref Kit adds an extra layer of security for whole disk encryption by using a Trusted Platform Module (TPM) to store the secret key. “It’s not a perfect solution since someone still has the device with the key in their hands, but it’s a bit more secure than just dumping the key on the hard disk,” said Ohly. “When you combine that with the secure boot, you make it very hard for an attacker to get the key.”

Finally, Ohly examined dm-verity, which was originally designed for Chrome OS, and is supported by Android. Unlike the other schemes, dm-verity won’t let you write to and modify files. The mechanism verifies the integrity of each block in a read-only partition so that any modifications immediately lead to a read error.

“The advantage of dm-verity over whole disk encryption is that you get a really specific read error saying this block is unreadable,” said Ohly. One side benefit is its ability to detect if a USB stick is corrupted. “It’s a common problem,” said Ohly. “You flash something and it doesn’t work, and it turns out to be a bit flip on the USB stick.”

Finally, if you’re wondering why Ohly is wearing a Santa Claus hat, be advised that it’s rather “a genuine dwarf hat from the coal mines of the Black Forest.” The hat reminds Ohly that much of his work is based on existing open source code. “I feel like I’m a dwarf standing on the shoulders of giants,” he said.

Ohly removed the hat, however, when discussing the aspects of the IoT Refkit solution that he wrote himself. “It’s getting kind of hot in here,” he added.

The full video is available below:

Connect with the Linux community at Open Source Summit North America on September 11-13. Linux.com readers can register now with the discount code, LINUXRD5, for 5% off the all-access attendee registration price. Register now to save over $300!