Testing of Internet of Things (IoT) solutions

Although IoT testing benefits from general software engineering’s testing principles and good practices, however IoT solutions require special considerations for more effective/ efficient testing phase execution. Some of these considerations are listed below:

1.????Hardware specific considerations: Hardware components in IoT solutions needs to be validated from different perspectives than pure software testing. For example, hardware components (especially sensors) show a gradual drift in their operation and requires constant recalibrations. There should be test cases that should detect this drift and help neutralize the impact of these drifts.

Another concern regarding hardware devices is the device heterogeneity observed in IoT solutions. Lack of standardization of these devices further complicates the verification/validation activities as testing strategy/plan needs to take into account all the possible hardware variants/SKU's. Compatibility testing becomes important for the same reason.

Related challenge is unavailability of hardware device during earlier stages of IoT solution development and as a result testing team has to rely on device simulators. Simulators can mimic device behavior to a large extent but can’t cover all the device characteristics (especially issues related with performance and subtle race conditions). Testing team also needs to spent inordinate time in customizing available simulators to accurately match real device characteristics.

2.????Difficulty in replicating real-world scenarios: IoT devices operate in "real" world and not all the field conditions can be replicated in labs. Erratic connectivity, hardware drift (as mentioned in earlier point), battery life dependency on environmental conditions (e.g., ambient temperature variations, physical vibrations) are difficult to replicate in lab. Replicating production environment in lab is impractical and cost prohibitive primarily due to volume considerations (e.g. difficult to install 10,000 sensors in lab).

Additionally, "on the move" field devices encounter challenges like random switching between carrier technologies (e.g. frequent and random switching between “NB-IoT” and cellular is common in remote areas) and this require special test scenario’s/cases to be considered.

One way to handle the replicate the random scenarios that are prevalent in real world is to consider exploratory testing (chaos engineering) as part of test plan.

3.????Unavailability of skilled testing skills: All said and done, IoT is still an emerging field and there are less number of skilled persons who can conceptualize and develop “end to end” IoT test cases. Typically, testing people carry expertise in either hardware or software domains, and it is difficult to find a person who is equally well versed in both ends of the spectrum.

Another challenge is that testing persons are quite experienced in conceptualizing test cases where interaction is between human and software system where as interactions in IoT system is inherently complex involving labyrinthine interactions between human, hardware, connectivity and software systems.

Some IoT specific test cases that should be considered as part of IoT testing includes range testing, roaming and handoff scenario’s, device specific test cases (e.g. wakeup, hibernation, watchdog functionalities), battery characteristics w.r.t data load, connectivity status and ambient conditions (noise due to other RF devices in the vicinity).

4.????Strict compliance to regulatory standards: In addition to domain specific compliance requirements (PII, HIPAA, PCI DSS, GDPR, etc.) for backend systems (central server), IoT solutions need to comply with field device specific regulatory standards (like FCC, CE, etc.). IoT compliance requirements are continuously evolving and as a result, testing strategy and implementation needs to adapt on a continuous basis.

In scenarios where IoT systems performs actions in physical realm ("end to end" use cases involving sensification as well as actuation - robotic arm operating in smart manufacturing context), strict adherence to compliances is even more important as it can result in loss of human life and property.

5.????Inadequate time allocated for validation/verification activities: There is considerable “market pull” for IoT solutions due to “newness” of IoT applications and aggressive marketing initiatives. As a result, IoT solutions are required to be developed in short time lines and disproportionate impact is observed on testing activities as these tend to fall at fag end of solution development life cycle.

Some of the above-mentioned challenges can be alleviated by providing adequate trainings and by adhering to “shift left" testing principles whereby verification/validation activities are started early in the project life cycle. Better alignment of hardware and software development cycles (project schedules) will result in availability of hardware at "right time" thereby reducing overreliance on device simulators.

Figure shown at the beginning of the article lists various testing types that have relevance in IoT context and can be considered while drafting IoT test plan.

Please feel free to add additional testing challenges and related mitigations in the comments section.