Automated Testing of Embedded Software

Embedded systems can range from tiny 8-bit micro controllers with no OS to more powerful platforms running e.g. embedded Linux. In this article we focus on testing of smaller event driven systems with no OS, or with a simple RTOS. Such systems usually interact with the physical world in real time and with other embedded systems as well. The event driven execution style is a popular choice for scheduling tasks. Tasks are implemented using functions (in the C/C++ style) and invoked by a simple scheduler when messages are sent to them. The software is built on a developer PC (running e.g. Windows or Linux) and gets cross compiled into a executable that is downloaded to the target for execution. Atmel Studio or the IAR Embedded Workbench are popular IDEs used in this scenario and using the IDE the software can be debugged on target.

What is an automated test?

Additional to downloading the executable and manually activating the device to test if it behaves as expected, an automated test is a program that some how activates parts of the software that is under test and verifies that it behaves as expected. For the smaller event driven system that is used as example here, a test could be a small piece of code that constructs a message, sends it to a task, waits for the task to process the message, and then verifies that the effect is as expected - that the task did what it was supposed to. A proper system for automated testing will be able to run a suite of such tests without human aid, collect all results and publish them to the developers.

Automated tests on the target

The obvious and natural choice is to run tests on the actual target. This can be accomplished in many ways.

One way is to embed the test programs into the software as additional tasks or functions. Then when the device starts up in testing mode it will test itself by having some test task ensure that all tests are run. The results from the test can be stored or sent to a PC to be published later. The downside of this method is that the code under test is actually changed: the testing code is added as well and this may influence the behavior of the software. This strategy will also use extra resources in the target system.

Another way is to keep the testing code on the developer PC. The testing code sends messages from the PC to the target through some communication channel and activates the code under test in this manner. The results from running the code under test must be obtained by probing the resulting state of the target through the communication channel. Here no additional testing code is moved to the target.

Both the methods above test the entire application. Though perhaps only certain parts of the application gets activated by the tests, the full application is downloaded to the target prior to testing.

A fully automated testing system would automatically build the firmware, automatically download it to the target, and automatically activate the tests, retrieve the results, and report them to the developers on a dashboard. This requires a dedicated build server that runs the testing control software continuously and is attached to an actual target. An example of such a system could be a Jenkins build server using scripts written in Python or bash to control the build, download and execution of tests. Setting up such a system for an embedded project requires intimate knowledge about the target, communication protocol, and specific testing method used.

Automated tests on the host

A different approach is to test the embedded software, not on the target, but rather on the developer host. Though this may sound counter-intuitive there is a range of surprising benefits to this approach as we will see shortly.

Host based unit tests

A unit test is a small test that only tests a small part (a unit) of the software. In our case a unit could be a task. To run a unit test on a host machine it must be possible to compile the unit - in this case the task - on a host machine. But embedded software is often dependent upon how data is laid out in memory, specific compiler directives, non-standard macros and libraries, endianess, and many other domain specific dependencies that make embedded programming so challenging. Also a single unit may have a very large dependency extent, i.e. it may include many headers that include other headers and so on that make it hard to compile individually. A large dependency extent usually indicates that the unit requires that many other functions or libraries are linked with the unit. So in practice, just completing the first step of compiling the unit individually may be difficult. Our experience clearly shows that it is very beneficial to do this effort anyway. After a unit is compiled and linked for a host PC it becomes apparent what dependencies the unit has to hardware, other units and all the other issues mentioned. During this process it is possible to reduce these dependencies and clean up the unit. A unit that can compile individually on another platform than for which it was originally written, is of a much higher quality than a unit that can only compile in the context of the full software. For this reason host based unit testing is the single most valuable form of testing that embedded software can be exposed to. Even if the test would never find an error, the process of compiling it as an individual unit on a host platform would improve all aspects of the units design and quality significantly.

Host based integration tests

When testing a single unit, dependencies to other units are "mocked out". This means that if the unit calls another unit, that call is not actually performed, but rather handled by some stub, either generated automatically or manually. In integration testing several units can be tested together and their interaction becomes part of the testing. In the example of event driven systems an integration test could be comprised of the scheduler and one or more tasks. The test program would setup the scheduler, install the tasks under test and inject events into the system and verify proper behavior. If each unit can be compiled for the host, it is straight forward to compile the integration test for the host as well.

Host based system tests

When linking all units and the scheduler together on the host, the result is almost the same as producing the final executable for the target, except that the hardware layer must be "mocked out", since the hardware of the host is very different than from the target. It follows that the target hardware must be simulated when running on the host. It also follows that such a scenario where all units are compiled together on the host and linked with a simulated hardware layer resembles what is usually called an emulator. Being able to build an emulator of the embedded target can be very useful. The emulator can be set up to automatically reproduce common use scenarios and replay these scenarios on the build server. This is an important addition to unit and integration tests for keeping the software running after refactorings and feature additions.

Benefits of automated tests

Even simple automated tests on the target are very useful for discovering bugs early. But setting the bar a little higher and aiming for host based testing as well, brings the embedded software under complete control. In such a scenario, where embedded software is tested both on target and on a host in an emulated environment, developers shift their focus from debugging over to creating new features instead. Of course additional effort is required for creating new tests and maintaining the test system, but spending time on writing tests is much more fruitful than spending time on debugging. When a bug is found it is corrected and we are left with nothing. When a test is written, it stays around forever and is run continuously to ensure that the part of the software that is tested never breaks without the developer knowing it immediately.

Challenges of testing

Setting up an automated test system to test on target is obviously possible, but it can be hard to find time and manpower to do so in a busy workday with deadlines approaching (and missed) all the time. Also making the developers write tests during their every day development can be hard if it is not introduced and incorporated the right way. But the benefits of having an automated test system testing on target is immense.

Setting up a host based automated test system is a real challenge. It can be a daunting task trying to refactor a single unit and extract all dependencies to hardware, libraries, compiler specifics etc. The first time a single unit is compiled on a host environment using a host compiler the list of errors is usually discouraging. Performing this task of enabling host based unit testing requires a very specific skill set (and mind set) that can make it impractical for the in-house developer. But the benefits of "off boarding" embedded software and making it buildable on a host PC is huge and worth all the trouble.

Help getting started

If you are fed up with last minute bugs and feel frustrated about slow progress of your embedded software projects, you are not alone. We’ve helped dozens of embedded projects move from uncertain and risky to steady and reliable and we can help you too.

Contact the authors,

[email protected] or [email protected] for more info