Virtualized kiosk mode HMIs

In this article I want to give an outlook how a kiosk mode can be integrated in HMIs with display and touchscreen. Kiosk mode is making HMIs bullet-proof and whatever user could crash the system, virtualization is the key for a speedy recovery.

Human Machine Interface - Definition and History

Human Machine Interfaces (HMIs) can be found everywhere where a human comes into contact with a machine. The simplest HMI can be an on/off power switch.

In the past with more possibilities, alphanumeric LCDs with keypads were used in machines. Later on there was a switch to graphic displays with keypads. Resistive touchscreens were added, so keypads were realized within the display content. And with iPhone, the capacitive multitouch touchscreens were adopted with features like swipe gestures and Cover Flow visualization.

Have also a look into my article https://www.dhirubhai.net/pulse/smartphone-hmi-mcu-manuel-schreiner Smartphone HMI for MCUs.

Virtualization

Today with more available resources and capabilities within machines, software can be virtualized. This means a host system can run different independent software containers. Each software container can have a different job or responsibility. To run software containers on a Linux host, different virtualization layers like Docker, Podman or LXC can be used. For Docker and Podman, there are tools available which can directly connect to a CI/CD pipeline, so the deployment of HMIs can be directly triggered from the build pipeline.

By the way, this is the same way software can be deployed in the cloud or on a server on-premise. For traditional headless software services, software containers can have TCP/IP network connections to communicate with everything outside of the software container.

But for HMIs there is the need to connect the local user via visual, optical and touchable interfaces.

Connecting the outside into a software container

While normally the cloud or server on-premise has no need to access anything else than network to the software container, HMIs having the need to connect local peripherals into the running software containers. In terms of consumer available operation systems like Microsoft Windows, Android, iOS and macOS, forwarding peripherals into a virtual software container can become pretty complex. This relates mostly on security reasons and makes sense for desktop or mobile computers. In addition especially for Android and iOS, virtualization has pretty hard restrictions, so Docker or Podman is generally not available for these operation systems.

In Linux Systems, there are simple mechanisms available to forward peripherals via Docker, Podman or LXC directly into the software containers. The host system is normally secured by avoiding any access to local peripherals except there is a forward rule with access rights defined.

By this fact it is possible for HMIs to forward peripherals like USB, Input-Devices like the Touchscreen/Keyboard/Mouse, Sound-Devices and also the Framebuffer to output video.

HMI Production Strategy

Strategy means also to have a long term vision. As soon HMIs are used in public transport, for example in vending machines or information displays, there is an absolute need for robust designs. There is also the need to have a deployment strategy, so software can be updated and installed on the system.

Today, there is no John Doe's computer anymore required to pack the software and the trainee who is running with the USB stick to the device. CI/CD pipelines are used to compile, test and pack the software and to make it available for installation.

There are several ways to get software to the device in the field. I try to explain a simple one without too much dependencies. USB stick can be indeed still a solution for a deployment, but as soon there are hundreds of devices to deploy, it makes sense to have a different strategy. With Docker and Podman it is possible to provide the so called Docker registries. These can host images on a web server to create and update software containers easily. Gitlab for example has a Docker registry already integrated and has also the possibility to add deployment tokens for securing the access. Devices in the field can use a software container with Watchtower installed which can watch for new available versions in a specified time interval or fixed time and trigger the update of the other required software containers.

For the host system it makes sense to have a look into Yocto Linux with Rauc tools included. Yocto makes it easy to build an own Linux OS for the own need and to build and pack everything required in an installable base image with possible over-the-air (OTA) updates via Rauc. Rauc is using a dual boot strategy for OTA. If an update fails, it is still possible to boot the old system again. With Rauc, it also host system updates can be downloaded and triggered via virtualized software container.

HMI Software Technologies

Steve Jobs already made the correct statement as he separated operation systems by the device it is used for. For sure the base of an operation system can be the same for a different device types while everything else should have its own ecosystem.

For consumers, devices have to be customizable and developers must be able to make developments for their applications.

For HMIs there is the need to customize the content, but to take away the customization level from the end-user. Probably in 99% of all use cases, the content of an HMI can be fulfilled by a web browser. A web-browser running in a software container in kiosk mode can be easily realized. There is no need to make a customized native app anymore. The only thing that needs to be taken into account is how the running frontend in a web browser connects to its backend.

Traditionally a web server is running as service on a server which is forwarding dynamical requests to its backend service. Some backend services also include a web server for static content, so there is no need for an additional web server for static websites anymore.

So beside the web browser it makes sense to add containers for hosting static websites, dynamic backend and if the backend or the web server does not include proxy http requests, a separate application proxy. So in this case the main application consists of three different software containers.

Adding Real-Time Events to the HMI

In most cases, the HMI is interacting with the outside to get different contents or to have real-time data available. For sure, if connected to the internet, a backend-service running on the edge can also forward its information from/to a public available REST endpoint. But if other technologies are used within a vending machine like NFC or controlling the PLC for the barrier in a parking garage, it makes sense to build for each interface separate services as well and to choose a event driven technology like MQTT that connects to the backend as real-time entity.

Wrap-up

The complete picture got pretty big and virtualization looks like an overhead. But indeed virtualization can be very efficient also in terms of resources. Furthermore virtualization makes it possible to bring different technology stacks together in one system without dependency issues. Also different Linux versions and different versions of libraries can be put together in just one system running virtualization. By this fact, different teams can work on different technologies in parallel. The use of a normal web browser as HMI helps to use a modern way of software development which also does not exclude interfaces to machine components like a PLC or NFC reader. The result is a bullet-proof HMI which have speedy recovery times in case of crashes.