AI/ML (Livin’) on the Edge…

The next and one of the most important chapters in the story of IoT will be moving of?AI/ML?(Artificial Intelligence/Machine Learning)?from the Cloud to the Edge. This will be the inflection point where the edge devices become truly?“Smart”.

Today devices are considered “Smart” as long as they connect to the network, but that definition needs to expand and include Intelligence (read inference) in addition to just pure Connectivity.

At their most basic, IoT devices connect?real world with the digital world. They sense?(3 Vs — vibration/voice/vision)?whatever is going on in their surroundings, collect that data, process it, communicate with the network, and then act on it as needed or as told by the cloud. As the?Digitalization and Electrification?trends continue, we will see these capabilities coming to more and more devices around us.

In the last 8–10 years edge devices started to connect to the cloud (mostly over wireless) which kick started the IoT “revolution”. These devices gather information, send it to the network and wait for the next step where actual AI/ML took place.

Amazon’s Alexa?is one of the most relevant examples of this architecture. When you ask Alexa to turn on the lights almost everything other than key/wake word (Alexa) detection to answer or act on this question was done in the cloud.

There are multiple drawbacks/limitations of this cloud only AI/ML architecture:

1.?Latency

Even though our network speeds are amazingly fast even then they cannot keep up with real time needs of certain applications. Today edge devices need to wait for the cloud to give instructions which is not great for use experience when it comes to real time applications.

You should be able to turn the lights on or off instantaneously and not wait for few seconds for it to happen.

2.?Increased energy/power usage

Going to the cloud and coming back over wired or wireless connectivity takes time that results in increased energy/power usage. This also results in thermal concerns for the device and its housing resulting in higher costs.

3.?Security risks

One of the biggest drawbacks of this cloud only architecture is sending data on the network and storing it in the cloud which is a huge security and privacy risk.

4.?Higher overall cost

All of the above results in higher total cost of ownership for the end user. Such IoT devices are expensive to maintain and operate.

But now?we are starting to see devices that have enough compute capabilities that a lot of intelligence/inference can now happen on the edge device itself. This is being enabled by new class of MCUs and MPUs or a combination of the two with enough horsepower to take care of the AI/ML workloads. We know that mobile phones have had this capability for a few years given their very high compute but now we are starting to see this capability show up in the non-mobile IoT devices.

These SoCs (MCU and MPU) are the heart and soul (and brains) of these IoT devices. Some of the key ingredients needed for Edge AI/ML SoCs enabling intelligence on the edge are:

1.?Energy efficient

Lot of IoT devices are battery powered so they need to be energy efficient for long battery life. SoCs going in to these device need to have independent power domains within multi-domain architectures with very specific uses for each application domain. Each domain should only be powered when it is needed and should go to sleep once it is workload is done.

We are seeing and hearing about SoCs which are consuming less than 10uA in sleep mode and less than 50uA/MHz in active mode.

2.?Scalable compute w/ML intelligence and enough memory

Most of the OEMs do not just build a single device for their customers. Instead they build family of devices with multiple SKUs which have different compute (AI/ML), memory, security, sensing and connectivity needs.

There is a definite need of a platform level thinking from the SoC vendors such that their OEM’s can leverage HW and SW efforts done for one SKU on to multiple other SKUs.

In addition, having enough memory (SRAM and non-volatile) in the system is of paramount importance because all the compute will be wasted if the required memory is off-chip. This will add latency and power which will bring us back to the limitations listed above.

3.?Security & Privacy

Security will apply to the code being run on the device, the data being collected and processed and whatever needs to be communicated to the cloud.

These SoCs will require isolated security sub-systems which will not be accessible by the application therefore ensuring that code and data is secured throughout the lifecycle of the device.

4.?Connectivity

Yes, still one of the key components of any IoT device…

Even if the intelligence is done on the edge some data still needs to be shared with the cloud which will aggregate data for all the other devices on the network, do the additional processing, run the AI training models and prepare the updated models to be transferred back to the edge device over connectivity.

5.?Sensor/Peripheral interface

Need to have all the right interfaces to be able to collect 3 V’s, vibration/voice/vision, data.

Digital and Analog interfaces and peripherals for microphones, cameras and different kind of sensors need to be part of the SoC.

Synchronized multi-modal HMI will be one of the key use cases going forward in the IoT devices and SoCs that enable this capability will be highly successful with the developers

6.?Integration

Single chip (or packaged solutions) will be key for space and cost because guess what? IoT devices are space and cost constrained.

Having compute, memory, connectivity, security, sensors in one device beats trying to get the same functionality from multiple devices.

7.?Software development

All AI/ML on the edge requires lot of new tools which will help with training the models, quantizing, deployment, inference and continuous re-training. Developers need to be equipped with tools that will make it easier for them to build their own applications and not get bogged down with the details on how to train and deploy a AI/ML model.

Conclusion

Next 3 to 5 years will see more and more IoT devices with edge AI/ML capabilities coming on the market. This will be made possible by next generation of SoCs being introduced in the next 12 to 24 months.

The next stage of IoT will be won with the devices that are able to do AI and ML inferencing on the edge with some support from the cloud. These devices will be small, scalable, secure, power efficient and cost effective. This new generation of devices will make?living on the edge (devices)?a lot more exciting.

Fawad Khan, San Francisco — 6/26/2022