What's needed to bring generative AI to embedded devices?

By: Zachariah Peterson | Originally published 7/24/2024 on Electronics360

Artificial intelligence (AI) has become such a massive trend in software and hardware that some topics deserve a second look. Take embedded AI for example; the one area of computing where implementation of AI without a connection to a data center is most difficult is in embedded systems. Small MCUs and field programmable gate arrays (FPGAs) can be used to implement neural networks for inference or very small-scale training, but a totally different architecture and ecosystem will be needed with high parameter count models that make up LLMs.

Since earlier this year , many companies have been making real strides bringing AI capabilities to embedded systems. This often comes from a set of software approaches, where models are optimized before deployment to an edge device. The other direction is the hardware direction, where unique hardware is being developed as alternatives to GPUs in a data center.

If you want to bring advanced AI capabilities to edge devices, here is what you can expect from technology going forward.

AI chipset startups for the edge

Currently, the industry largely relies on legacy compute solutions to implement any kind of AI, including AI at the edge. There are many processor options for implementing embedded generative AI:

• GPUs — If you can get a small GPU into an edge device, you can leverage a proven hardware platform for training and inference.
• Specialized ASICs — There are many startups building ASICs as accelerators specifically for highly scalable AI processing.
• FPGAs — Although FPGAs are less-often cited for AI, they do allow developers to implement any model architecture alongside other (non-AI) features as custom logic.

Aside from specialized ASICs, some form of generative AI, albeit at small scales with very specific applications and inputs, will still be possible with FPGAs or large microcontrollers running an embedded OS.

Despite all the market focus on Nvidia GPUs, there are some potential contenders in the AI chip wars, some of which could bring AI capabilities to the edge. We’re not talking about the mega-cap technology companies; we’re talking about startup companies developing unique products that are hyper-focused on embedded AI capabilities. Here are three AI startups that are developing hardware for AI at the edge.

• Mythic — The analog computing approach, which has been earnestly explored as an AI computing paradigm for nearly a decade, promises much lower power and resolution than digital AI processors. Mythic implements analog computing, where logic cells are not required to take discrete states but instead vary continuously. The lower power consumption of this approach is a big benefit for edge AI systems.
• Hailo — Another company focusing on energy-efficient deep learning processors, Hailo’s chips target smaller form factors with low-power consumption. Potential application areas for these chips include small robots, wearables and some automotive applications (e.g., sensor arrays).
• Groq — The company has developed a unique, scalable chip architecture that is designed for low-latency AI. The company’s target application area is AI accelerator ASICs, which could be used in multiple settings for fast inference, including in embedded systems or in edge computing.

Other companies are still targeting the supercomputing or data center infrastructure markets as alternatives to GPUs or as accelerator ASICs. The two best-known companies are SambaNova Systems and Cerebras Systems, which are developing large chips (physically and in terms of compute) for training and inference with very large models. These companies are also developing software suites for implementation and management, something which is still lacking in embedded AI.

The software stack

AI is at its core a piece of software that lives on hardware, and so the AI chip vendors need to provide a software stack for model development. Oddly enough, the embedded chipset vendors are still not at the point of developing these software suites or IDEs for developing and optimizing AI models. Instead, they tend to rely on open-source resources and libraries for building new models and deploying them on end devices.

Enter 3rd party companies like Edge Impulse; the company has built an edge AI platform that enables developers to quickly perform all the core functions needed to build and deploy an AI model on edge devices, including hyper-optimized LLMs:

• Compile datasets and create synthetic data
• Train models against compiled datasets
• Optimize models through parameter reduction and quantization
• Create containerized applications that use an AI model
• Compile models and deploy on an edge device

Given the size of the data center and cloud computing markets, companies developing hardware for those markets will need to develop entire platforms to support their hardware and manage model execution. The embedded systems market is also quite large, but expect the developer ecosystem to be built and maintained by 3rd parties rather than by the AI chip vendors.