ACAD 49: Project GreyMatter

Intelligence isn't just about inventing new technologies; it's about using them effectively while ensuring privacy and responsiveness. Apple’s upcoming World Wide Developers Conference (WWDC) on June 10th is going to be different this time. Developers and tech enthusiasts across the world are eagerly waiting to learn what Apple will unveil alongside iOS 18 to break its long-held silence in the Generative AI space.

The last one year has been quite tough for Apple. Numerous issues have affected the company’s performance and investor sentiments, including dipping sales in leading markets such as China in the face of tough competition from Huawei, Patent lawsuits over the pulse oximeter used in Apple watch, and the divestment of the autonomous car research division to feed its Generative AI body of work[4]. Apple’s stock grew 7.6% over the last year, while the S&P 500 index grew almost 26%. In contrast, Microsoft and Google have exceeded expectations both for end users and investors, partly due to their relatively clean integration of Generative AI features across their product suites.

While the near-term context may look gloomy for Apple, the resurgence of the company’s position will largely depend on the partnerships it can quickly form to command significant differentiation in the AI space and maintain its global market reputation. Preliminary data suggests shipments of generative AI smartphones will exceed 100 million units in 2024.[4]

Apple's "Project Greymatter" is one such step. Apple plans to introduce a set of AI tools that will integrate into core apps like Safari, Photos, and Notes, as well as enhance operating system features such as improved notifications. The highlight is that these AI tools won't just run on the cloud; in fact, they will harness the cutting-edge hardware on the device to provide prompt responses to user queries [1]

On-Device AI

We’ve been quite well primed by tech leaders that as the number of parameters in a Large Language Model (LLM) go up, the model starts exhibiting reasoning characteristics that mimic human intelligence. However, more parameters also translate to higher costs, increased latency, and reduced privacy.

In Apple's case, real-time cloud interactions for every user query could expose the device (and the user’s data) to cyber attacks, potentially harming Apple’s strongly-held privacy positioning in the market. Apple has a strategy. AI processing will be a hybrid setup with intelligence distributed between the cloud and devices.

According to Qualcomm, future architecture will need different offload options to distribute processing among the cloud and devices depending on model and query complexity factors. For example, if the model size, prompt, and generation length are below a certain threshold and provide acceptable accuracy, inference can run entirely on the device. If the task is more complex, the model can run across both the cloud and the device.

How Does It Work?

Several use cases would benefit from the reduced latency and enhanced privacy that comes from running AI applications on devices. For instance, autonomous cars and personal healthcare monitoring IoT devices need algorithms on-device to respond with blazing speed and mitigate risk to human lives.

In the case of autonomous cars, the object detection and image segmentation algorithms in the car have less than 35 milliseconds to collect, run, and deliver insights to inform real-time decision-making.

On the other hand, every time you click a photo, more than 20 algorithms work harmoniously on the phone device to give you that perfect shot satisfaction.

At the end, AI sophistication = Data X Algorithms X Compute. To make On-Device AI work, several capabilities would need to evolve on each of these frontiers to deliver the intended experience.

• Data: Apple has an edge with its market leader position in the smartphone space. Apple continues to dominate the premium smartphone market (devices over $600), with a 71% share in 2023. If Apple can find a way to fine-tune its suite of AI algorithms on this existing bolus of user data without compromising data security, it can retain its edge for a long time.
• Algorithm: More than 100 open-source algorithms are available for on-device execution, such as Stable Diffusion, Llama, and Fuss-Free Network (FF Net). These algorithms have simplified architectures focussed on specific tasks, with parameter counts in millions and model sizes below 100 MB [2].
• Compute: Apple recently introduced the M4 chip as part of its latest iPad release. This chip comes with a dedicated Neural Processing Unit (NPU) to handle up-to 38 TOPS (Trillion Operations Per Second). According to Intel, running sophisticated AI applications like Microsoft Copilot could require upto 40 TOPS of computing power but Apple, as usual, may lead the way in achieving this outcome more efficiently. [3]

Conclusion

Apple’s future hinges on its ability to innovate while maintaining its core values of privacy and responsiveness. Project Greymatter is a critical step in this direction. By focusing on hybrid AI processing and leveraging its market position, Apple is poised to set new standards in the tech industry. The upcoming WWDC will be a defining moment for Apple, potentially marking a new era in AI integration and user experience.

Resources:

1. https://www.imore.com/ios/wwdc-2024-to-reportedly-showcase-project-greymatter-apples-idea-of-ai-for-the-masses
2. Introduction to On-Device AI - DeepLearning.AI
3. https://www.tomsguide.com/tablets/ipads/apple-unveils-m4-chip-for-ipad-pro-heres-what-it-can-do
4. Apple unplugs self-driving electric car project, reports say ( bbc.com )