Microsoft at NeurIPS 2023

This week, many AI researchers and practitioners were at NeurIPS 2023 in New Orleans, the largest and perhaps most important conference in AI research. NeurIPS is always a valuable opportunity to see what's happening at the frontier of AI and to hear what’s on the minds of the folks moving the state-of-the-art of AI forward. Microsoft has been a participant in and sponsor of NeurIPS for many years. This year in particular, I'm enormously proud of what my colleagues at Microsoft Research and other parts of the company have contributed to the conference and to the field.

You can check out a full accounting of those contributions at the MSR NeurIPS 2023 website . If you'd like, you can even chat with our MSR NeurIPS Copilot , which has access to the full text of the 100+ papers by MSR authors accepted this year at NeurIPS, plus our full catalog of 2023 AI publications. I encourage folks to take a few minutes to explore some of this work, and, if you'd like something explained that doesn't make sense, to consider chatting with this Copilot . If for instance, you want to know what is interesting about the paper A Theory of Unsupervised Translation Motivated by Understanding Animal Communication by Goldwasser, Gruber, Kalai, and Paradise, you can ask and get back:

The paper "A Theory of Unsupervised Translation Motivated by Understanding Animal Communication" presents a theoretical framework for analyzing Unsupervised Machine Translation (UMT) when no parallel data are available and the source and target corpora do not share related subject domains or similar linguistic structure. It is particularly interesting because it applies this framework to the ambitious interdisciplinary initiative, Project CETI, which aims to understand animal communication, specifically that of sperm whales, using machine learning tools.

Pretty cool, right?

In addition to all of this good work being presented by Microsoft researchers and their co-authors, we had a couple of other interesting announcements while NeurIPS 2023 was underway. As a follow-up to their Medprompt work, which shows how the generalist GPT-4 model can perform as a specialist on medical challenge problem benchmarks, our Chief Scientific Officer Eric Horvitz and colleagues shared new approaches to steering frontier models to better performance through prompting strategies, which we are releasing in?promptbase , a collection of resources on GitHub. ?Using a modified version of Medprompt, the team steered GPT-4 to achieve the highest score ever reported on the complete MMLU challenge, a series of benchmarks that tests general reasoning and knowledge capabilities of large models.?I have?been continually impressed by their ongoing work to more effectively steer large models with simple, zero-shot prompts; it shows that we’ve far from exhausted the capabilities of existing frontier models and have much more to learn about their capacity for specialization and reasoning.

Similar efforts to extend the value and impact of language models have come from our researchers in MSR’s Machine Learning Foundations team, who this week shared details on Phi-2 , the latest in a suite of SLMs that achieve remarkable performance on a variety of benchmarks, surpassing all other models in its class, including the recently announced Gemini Nano from Google. Phi-2 is a relatively small model (2.7 billion-parameters), but in complex benchmarks, Phi-2 matches or outperforms models up to 25x larger, thanks to new innovations in model scaling and training data curation.

All of this caps off what has been a truly extraordinary year in the field of AI and in the technology industry as a whole. It has been, without question, the most exciting and interesting year in technology that I’ve seen over a fairly long career. It bears mention that I’m pretty sure I said more or less the same thing at the close of 2022, and I suspect I’ll probably be saying the same around this time next year and each year for the foreseeable future—the point being that in AI right now, we’re experiencing a period of sustained exponential growth which represents perhaps the most profound technological progress that we as a species have ever seen.

And it’s really only the beginning. Modern generative AI is still in its infancy, and we’re learning as we go. While it feels like we’ve lived with them for ages now, 2023 was really the first year that powerful AI tools like ChatGPT and Microsoft Copilots meaningfully entered the public vernacular as useful helpers to make people’s lives easier. By the time next year wraps up, we’ll have many new experiences, apps, and tools to add to that list, that in the limit, will create cascading benefits for more and more people on the planet. Though the amplitude of hype and acceleration rate of AI’s growth can keep folks fixated on each subsequent “next big thing,” if we step back just a little bit, it’s easier to see that the opportunity in front of us is astronomically greater than what we’ve already achieved.

I mentioned Phi-2 above, which I believe is some of the most exciting research happening not just within Microsoft, but in all of AI. This week I had the opportunity to have a conversation with Sébastien Bubeck, one of Phi-2’s creators, about his work, what Phi-2 can do, and what it suggests for the future of how we build and approach different kinds of AI models. Give it a watch here .

All the best to you and your families during the coming holiday season. May 2024 continue to bring the excitement of discovery and continued innovation for us all.