Addressing the Unique Challenges of Embedded Software Development with DevOps

In modern web and mobile application development, the concepts, toolsets, and services of DevOps have been employed to speed up product development while simultaneously increasing quality. DevOps has been a major win for the software industry for the last ten years.

What about the software and firmware in embedded systems? What makes them any different? Surely they're benefitting as well, right?

Embedded systems are often the hidden, but critical piece to products in various industries, from automotive to medical to defense to consumer electronics. Unfortunately, their development and testing present unique challenges that traditional DevOps toolsets and services don't address. These challenges arise from two very distinct differences between web and embedded software.

Discontinuous Delivery

Traditional DevOps workflows for web and mobile applications employ Continuous Delivery (CD). The build, automated test, and deployment pipelines live and scale entirely in the cloud and typically pump out new releases daily or many times per day as the HEAD of the codebase moves forward relentlessly. Anytime a mistake makes it into the field, the CD backend quickly and effortlessly pushes an update or rolls back to a previous version. Users are none-the-wiser.

Not so with embedded systems. The embedded firmware image is usually a line item on the assembly parts list that gets programmed into the units at manufacture time. Almost like a resistor. Even with IoT and other field-upgradeable embedded systems, pushing firmware updates is complex, discontinuous, and usually requires some user intervention.

The firmware that makes it out the door has to work. You might not get a second shot at it.

Hardware-in-the-Loop

At the end of the day, the firmware requires the specific product hardware for which it was built, including testing the system prior to deployment. Hardware, especially for testing, is inherently expensive, finite, and messy. Plus, it lives in a physical, geographical location on a lab bench or in some rack connected to a host or network switch.

Which version of the product hardware supports which version of the firmware? How do I know what version of the firmware is on that field unit? How do I perform continuous integration and automated test with hardware? How do I scale across my test farm? How do I get this build into that system in the lab?

HIL makes everything slower and more difficult.

The Five Unique Challenges of Embedded Systems

These two differences lead to several unique challenges with embedded systems firmware development, test, and deployment.

Complex Toolchains

Cross-compilation and the use of different build host environments add layers of complexity. Embedded developers typically work locally with the build toolchain installed on their laptop and a development system connected to their machine. They code, build, and test locally on their laptop and connected test system. When it's time to push and integrate, they often face the "works on my machine" issue, where the build or testing fails for their colleague during integration or in the production build system due to differences in toolchains and build environment setups.

Complex Version Management?

Embedded systems often run multiple firmware versions across different product lines and customer deployments that originate from the same code base. You've got different versions of the hardware out in the field. Likely different versions of firmware also. Knowing who has what, and who's supposed to get what is non-trivial. Managing these versions, ensuring compatibility, and maintaining clear traceability of changes is a significant challenge.

Required Point-Fixes

The certification test house just kicked it back to you with a message that says, "Fix this bug only without changing anything else. If you change anything else, you will reset the certification testing process."

Crap, you can't give them your HEAD of the codebase, which is unfortunate, because you fixed that bug months ago. So now you have to recreate the source and build environment from that release, which happened four months ago. Does the release ticket contain all the commitIDs? What about the versions of the library dependencies? How about the specific version of the toolchain and build scripts?

Then, once you've fixed the bug, you've gotta prove that your new firmware contains only that fix.

Proof of Version and Forensic Analysis

Your customer is asking that you prove that the units that roll out of manufacturing actually has the firmware that was certified. Or, maybe, your support team has received a bunch of field returns. How do you know definitively what firmware they have, and how you get back to the source and build environment that created it?

Hardware-in-the-Loop

Software for embedded systems must be thoroughly tested on the actual hardware platforms it will run on. Discontinuous delivery puts more pressure on testing embedded systems prior to release. This dependency introduces complexity in testing workflows, requiring a heterogeneous test environment that can handle various hardware configurations and the specific test content that matches. Your test farm has finite and limited resources, but you need to parallelize across it as best you can. You want to test PRs, master merges, releases, and then conduct nightly and weekly soak tests. How do you manage all of this?

Addressing the Challenges

Although the challenges are formidable, we've had great success at using the concepts of DevOps with the addition of purpose-specific tools. Here are some ways we've addressed, overcome, and thrived through these challenges.

Common Build Environment

We create a common build environment (CBE) by using the once-but-no-longer-magic-in-the-embedded-world technology of containers. A CBE is a container that contains the correct version of the toolchain, plus access to the codebase and utilities repositories. Now, whether a developer builds locally or the pipeline is building in the cloud for integration, the exact production build machine is used. We've eliminated 100% of build issues due to mismatched versions or build environment differences.

Fuze Build, Package, Release, and Delivery Tool

We developed a tool that automates and integrates universal configuration management from build all the way through delivery. The key to solving complex versioning across a heterogeneous product family, recreating previous builds no matter how far in the past, and forensically proving what you have in your hand is what you said it would be, is to build configuration management into the process as a fundamental pillar of the automation. Much like Jira does with communications and git does with source code, making configuration management a fundamental pillar of the tool eliminates not only human time and process, but also human error.

Fuze starts with the build. As a generic build executive, it allows you to use the tools that you already have. It simply wraps your current build procedure, utilizes a CBE, and stores every piece of metadata associated with that build and the CBE so that you, the QA team, the product manager, and everybody else knows exactly what's in there and how to recreate it. If you build the FuzeID into the image itself (based on your requirements of security and obfuscation), now you have the perfect forensic tool to prove and know everything about this firmware image.

What goes into the package for the system delivery to manufacture? Images for multiple processors? Static config files? Documentation? No problem. Fuze also builds the release packages for you with 100% configuration management at its core.

When it's time to release and deliver, if even just a demo build through an FAE, Fuze is again the answer. Find the FuzeID of your intended release package, release it, and then deliver it, all without leaving the tool.

We create a single source of truth for configuration management -- the FuzeID.

With a FuzeID you know the following:

• The build - who, when, what CBE/tools, build command(s)
• Source commitIDs
• Dependencies and versions
• Build package contents
• Test results
• Release status - stage, by who, when
• Delivery status - to who, by who, when

Cloud2Lab Automated Test Framework

HIL testing gets little attention in the open-source and generic automated test framework world, so we built our own -- C2L.

C2L, like Fuze, takes the perspective of the person-in-the-lab. We believe and have seen the benefits of using the person as the focal point for automation. We ask, "what does a person do?" And then, we build the automation tool to allow that.

C2L uses the tools, test commands, and sequences that you already use if you're testing it yourself manually. No new language, API's, or scripting environment required. It gives you a straight-forward and easily understood environment for fully automating what you already do.

C2L also bridges the cloud to the lab. The build pipeline ran, or you built it locally, and now you have to push that image into a particular device or farm of devices on a bench or in a rack in some lab. We built C2L so it can do that, all through configuration. Plus, it handles the different versions and products across which you must test. Do you have 3 of these and 5 of those? C2L can parallelize according to how you configure it.

Debuggers, scopes, robots, and other control and acquisition required in your test setup? No problem. C2L handles it all and orchestrates your heterogeneous test content across the entire farm, then retrieves and organizes the test results.

You Can Do It (We Can Help)

Although embedded systems present some unique challenges, DevOps is still the way to help your team go faster and pump out products with higher quality. You just gotta know how to make it work for you.