Microsoft Unveils Phi-3: A Breakthrough in Small Language Models

Introduction

Microsoft has announced the launch of its Phi-3 family of open small language models (SLMs), positioning them as the most capable and cost-effective models of their size. The Phi-3 models represent a significant advancement in AI, outperforming larger models on language, coding, and math benchmarks. This innovative leap is made possible through a novel training approach developed by Microsoft researchers, which sets a new standard for efficiency and performance in AI.

A Shift Towards a Diverse Portfolio of Models

Sonali Yadav, Principal Product Manager for Generative AI at Microsoft, highlighted a strategic shift in the AI landscape: “What they are going to start to see is not a shift from large to small, but a shift from a singular category of models to a portfolio of models where customers get the ability to make a decision on what is the best model for their scenario.”

Introducing the Phi-3 Family

The first model in the Phi-3 series, Phi-3-mini, is already available with 3.8 billion parameters. Despite its compact size, Phi-3-mini outperforms models twice its size, such as Mixtral 8x7B and GPT-3.5. Following this, Microsoft plans to release additional models like Phi-3-small (7B parameters) and Phi-3-medium (14B parameters), further expanding the capabilities of the Phi-3 family.

Key Benefits of SLMs

Luis Vargas, Microsoft VP of AI, emphasized the versatility of SLMs: “Some customers may only need small models, some will need big models, and many are going to want to combine both in a variety of ways.” The smaller size of SLMs enables on-device deployment for low-latency AI experiences without network connectivity, making them ideal for applications in smart sensors, cameras, farming equipment, and more. Additionally, on-device deployment enhances privacy by keeping data local.

SLMs vs. LLMs

While large language models (LLMs) excel at complex reasoning over vast datasets—ideal for applications like drug discovery—SLMs offer a compelling alternative for simpler tasks such as query answering, summarization, and content generation. Victor Botev, CTO and Co-Founder of Iris.ai, commented on the paradigm shift: “Rather than chasing ever-larger models, Microsoft is developing tools with more carefully curated data and specialized training. This allows for improved performance and reasoning abilities without the massive computational costs of models with trillions of parameters.”

Breakthrough Training Techniques

The quality leap in Microsoft’s SLMs was achieved through an innovative data filtering and generation approach inspired by bedtime story books. Sebastien Bubeck, Microsoft VP leading SLM research, shared insights into this approach: “Instead of training on just raw web data, why don’t you look for data which is of extremely high quality?”

Ronen Eldan’s nightly reading routine with his daughter sparked the idea to create a ‘TinyStories’ dataset of simple narratives. This approach proved successful, with a 10M parameter model trained on TinyStories generating fluent stories with perfect grammar. Building on this success, the team created the ‘CodeTextbook’ dataset, synthesized from high-quality web data vetted for educational value.

Ensuring AI Safety

Microsoft underscores the importance of safety in deploying Phi-3 models. Their multi-layered approach to managing and mitigating risks involves reinforcing expected behaviors through further training examples, assessing vulnerabilities through red-teaming, and offering Azure AI tools for customers to build trustworthy applications atop Phi-3.

Conclusion

The Phi-3 family of small language models marks a significant advancement in AI, offering powerful capabilities in a compact, cost-effective package. By combining innovative training techniques with a commitment to safety, Microsoft is paving the way for a new era of AI applications that are both versatile and secure.

For more information, you can explore the Phi-3 models in Azure AI Model Catalog, Hugging Face, Ollama, and as an NVIDIA NIM microservice.