PyTorch 2024 and General AI Trends

I'm finally getting a chance to catch up with my notes and thoughts from PyTorch 2024. I wanted to highlight some interesting trends based on the sessions and conversations I had with attendees at the conference, which included data engineers, data scientists, ML Engineers, MLOps experts, and many more.

Core LLM Advancements

Large Language Models (LLMs) remain a major focus in the PyTorch ecosystem, with considerable efficiency, performance, and application developments.

• Model fine-tuning, including how to improve the process as well as increase the scope of data available for training
• Ecosystem enhancements, especially in the Llama stack, included new models, tooling, and an improved developer ecosystem with better APIs and out-of-the-box prompt guardrails.

Expansion of inference to edge devices

There's a notable focus on optimizing PyTorch models for edge devices and mobile applications. Computing at the edge enables real-time, on-device inference, opening up new possibilities for AI applications in resource-constrained environments but, more importantly, privacy and security-conscious enterprises that cannot afford to centralize their workloads.

Further, focus on Scientific Discovery.

While current iterations of LLMs and narrow AI applications have proven successful in areas such as customer support and information retrieval, quite a few sessions focused on tackling complex problems in physics, chemistry, and biology. This is promising because it expands the horizon of the applications that can be built over time as new architectures and best practices are developed for these additional problems.

Advancements in Chip Architectures

Semiconductor companies are increasingly developing specialized chips optimized for AI workloads, particularly for large language models and generative AI. A common topic of discussion was the need vs long-term viability of specialized chips for distinct types of workloads. As we know, certain manufacturers can build solutions for both inference and training workloads. However, there was a notable number of vendors representing inference-only solutions. While there is debate about the long-term viability of inference-only solutions, it will be interesting to see how the new chip architectures can keep up with advancements and changes in model architectures. For example, the Llama stack and other open-source AI models drive innovation in the AI software ecosystem, influencing hardware requirements.

The need for better AI Infrastructure Security and easing the MLOps burden

I had consistently similar conversations with employees across companies of all sizes who expressed how challenging it can be to build and maintain the infrastructure associated with self-hosting an AI application. This was further reinforced by the number of vendors whose primary product was to expose access to models as a service, abstracting the complexity associated with wiring up all the necessary components to get a model and respective components up and running in a production setting. Most of these solutions can benefit from simplification of maintaining up-to-date open source components that power these applications.