Embedded/IoT systems have proliferated in recent years. The rise of the Raspberry Pi, in particular, has made it easier than even for non-embedded system developers to build and ship IoT products. The RPi’s wide use in industry has even prompted the release of the RPi Compute Module system on module (SoM), which enables higher quality RPi devices to reach that market. I’d argue that the wide use of the RPi both in prototyping and in real products is, at least in part, due to the ease with which software can be set up on the system. In most cases, developers building devices using RPis, or similar single board computers (SBCs), use a desktop-like OS, such as Raspbian, Debian, Arch, or similar. And I think that in most cases this is a mistake. In this post I’d like to outline why I think this.

Package Bloat

Many of the mainstream Linux distributions (Ubuntu, Debian, Arch, Fedora), produce arm compatible builds, and support a wide variety of arm-based SBCs. Typically a default install of these systems consumes 1-2 GB of flash storage. Big deal right? Storage is cheap now. Unless you’re looking at building multiple thousands of devices, including an SD card or eMMC with multiple gigabytes probably isn’t a significant factor in your costs.

Even if you don’t care about storage cost, you should still reduce the amount of stuff you install on the device. The reason is because a small system, in which the developer has selected the installed packages in an understood system. If you’ve gone through the process of manually selecting a package to install, and at least spent a few minutes checking the configuration, you stand a much better chance of being able to debug that feature when it doesn’t go to plan.

There’s also an argument to be made that reducing bloat reduces the surface area for potential security vulnerabilities. Perhaps its a good idea to force the developer to perform some basic configuration for each package. Then they’ll have some understanding of potential security trade-offs in each package.

Software updates

For most people it’s really important to be able to remotely update both system-level and application software running on devices. For desktop-style Linux distributions, the system is updated on a package basis. For an embedded system, you really don’t want that level of control on a device, at least not in production. You can easily get into a situation where a fleet of devices are all running subtly different software. In such a situation, it’s not feasible to produce a definitive version number for the running software. This is always required when it comes to debugging a failure in the field. Fleet managers need to be able to distribute discrete updates to the system. Such an update will typically contain the core OS, user-space components, and the application code, all bundled into one monolithic (or sometimes incremental) update.

OS Images

Another shortfall of desktop-like operating systems is the difficulty of creating an installable image. You can, of course, just image the flash storage of a known working system and deploy it across a range of devices. Unfortunately this is pretty common when deploying RPi (and similar) systems. This is far from ideal, however, as it doesn’t encode the steps required to get to that point, which makes future upgrades and bug fixes really difficult.

There are a few bootstrapping systems for desktop Linux OS’s such as pacstrap on Arch or debootstrap on Debian. I believe these can be used to build a rootfs into a target folder, though I haven’t used either of them myself. I don’t think these systems would, by themselves, solve any of the aforementioned issues with packages and versioning. And I assume the build process would be more like: install Debian and then slim it down, rather than build up from scratch, though please correct me here if I’m way off.

The problem with Yocto and Buildroot

If you know something about this problem space, you’re likely thinking “Duh… that’s why people use Yocto/Buildroot.” And yes, I concede that these tools solve all of the aforementioned problems. They both take some input metadata describing what needs to be built, and (cross) compile an image that can be installed directly on the target board. The issue with these systems is they they’re out of reach for most developers. Let me explain why.

These systems are incredibly complex beasts, particularly Yocto, which I believe is the most widely used embedded Linux build system available. It takes some serious effort to get a basic working knowledge of the Yocto build system. Have a read through the Yocto Mega Manual before bed and you’ll see what I mean. Buildroot is much more approachable, though less supported and less feature rich, and still requires a good knowledge of fundamental Linux tools.

Investing the time and effort to learning the build tool isn’t enough though. The build tool is really just a assistant which helps you through the complexities of a compilation toolchain, kernel, bootloader, drivers, package configuration and compilation, rootfs build, and target image creation. In most cases you need to have some knowledge of each of these areas, as you’ll need to help the build tool along, especially if you’re working on custom hardware. For these reasons, many manufacturers look to contractors who specialise in these tools. Or, they just use Raspbian OS and live with the issues.

Potential solution

So what is the solution? I think that containers have a lot of promise in the embedded space. They allow an application image to be deployed on top of a core OS without having to manually edit files within package build scripts. They potentially solve the problems of imaging the OS, and also make updating the system atomic. They also potentially sidestep the difficulties of the Yocto/Buildroot systems, assuming a user-friendly package manager can be used to generate the container itself. In theory, the underlying build system would only be necessary for producing the core OS and board support, etc. There are already a number of OS solutions on the market employing containers, such as balena OS, Foundries.io Linux microPlatform, and TorizonCore. I’d be very interested to hear feedback from anyone who has used one of these platforms extensively. I’m unsure what the required system resources are for the container engine. Would these solutions be applicable to very small embedded systems? I’m also concerned that it’s potentially easy to reproduce the functionality of a desktop Linux OS within a container. And thus reproduce many of the aforementioned problems in the article.

Overall, I think a container based approach could be a big step forward in enabling developers to bring their application to market. I will be continuing to investigate this space in the near future.