Building Copilot: 7 Learnings Since the Feb 7 Launch of the New Bing and Bing Chat

One year ago today, we announced the new AI-powered Bing including Bing Chat, which was later rebranded Microsoft Copilot . It’s been a year full of innovation and learnings. Here are my top 7 takeaways:??

1. Real customer feedback from real usage is vital.?

We launched the new Bing in Preview because we knew it was the beginning of a journey. There’s only so much you can iterate in a lab environment, and there’s no replacement for real customer feedback. Of course, customer feedback can be loud and passionate and when media are involved it can get amplified. It’s important to listen and adapt to quickly fix any issues. What keeps us energized is that the vast majority of people using Copilot have been telling us that they have a very positive experience. Over time, real customer use and feedback have helped us make Copilot better including in important areas such as our Responsible AI efforts.

2. As new technology emerges, look for unmet needs and innovate.?

While AI solves a lot of problems, it also creates new needs for our customers. For example, we heard that a lot of people were wanting to use generative AI tools at work, but that companies were limiting access for fear that sensitive company data would leave the company boundary or would be used for training Large Language Models (LLMs). We knew we could help and quickly offered Bing Chat Enterprise which is now a version of Copilot that offers commercial data protection, a secure chat option that does not store any of the chat data or use the data for training. We also heard that some power users (researchers, developers, creators, etc.) wanted faster, higher priority access to our latest models with longer conversations. We also heard the Microsoft 365 customers want access to Copilot in Microsoft 365 outside of work. We shipped Copilot Pro in response to both of those needs. Lastly, Publishers told us that they wanted more control over their content on the chat experience so we announced new tools and controls for them. ?

3. We’re doing what we set out to do: helping people with complex tasks.?

When we integrated Open AI’s GPT-4 into Bing, I was amazed by the new Bing’s ability to provide more complete answers to complex, natural queries, which no previous search engine was able to do. We knew that search would never be the same.... Users quickly learned and progressively used our product to complete tasks in complex and professional areas as you can see in the chart below. Chat activity is still deeply rooted in search — we estimate over 80% of chats are still searching for information to some extent. What the chart doesn’t tell you is that web searches are also progressively becoming longer, more natural, and more complex.

4. There’s a lot of joy in creation.?

Many of the chat experiences in Copilot include creation, whether it’s writing letters, code, generating images, songs, poems, or more. I am a big fan of image creation and, with over 5 billion images created to date, so are many of you. I find it inspiring or just good fun — my family and I like to create images of shih-tzus (like our dog, Oliver) going on adventures. We were first to ship image creation in a chatbot back in March with Dall-E 2 and saw a huge spike in usage. Then in October we announced Dall-E 3 and the difference in quality of the models increased usage by 10x. Copilot Pro users enjoy even more image creation options such as landscape images. My favorite party trick is to take a photo in the Copilot app and have it generate a similar image in a different style , which leverages the multimodal and Dall-E 3 models within Copilot. Also, check out the new Designer in Copilot features we just announced today .?

Another fun feature in Copilot is the ability to create a song about any topic and in any style. We were also first to ship music creation in a chatbot back in December with the Suno AI plug-in.?The example below is for the prompt "create a song about my dog Oliver going on an adventure, in country style ." ????

5. Copilot is helping Bing grow.?

We have done multiple cohort studies that show that, when a user experiences Copilot in Bing, they become more engaged and also use traditional Bing search more, which means more searches and more traffic to publishers. Bing Daily Active Users (DAU) have been growing fast and crossed 100 million last year. There are different reports about search market share which measure it with different approaches. I look at Comscore because it’s the most focused specifically on search and is based on a large user panel, so I trust it more than other metrics. Comscore shows that Bing is continuing to grow, for example in the table below which they published recently .?

6. In a world of LLMs, search is more important than ever.?

For a while, many people were wondering if LLMs would replace web search, but if anything, web search has proven even more important than ever since it is key to make LLMs more accurate and up to date. Grounding with search results (RAG) in Prometheus , helps us answer questions about recent events with the latest web data and also reduces hallucinations. We’re seeing other companies and products recognize this as well as, for example both Meta and ChatGPT use Bing search for grounding their LLMs.

7. Copilot will bring even more innovation in the next year.?

In my 20+ years at Microsoft, this has been the most transformational, exciting and intense year of my career. So, what’s next? Honestly, I can’t wait to see —despite all these learnings I still haven’t learned to predict what’s ahead. But I do know that Copilot and AI are the interface of the future, and if this next year is anything like the last, we’ll continue to innovate and learn a lot. I expect we’ll see more focus on task completion across apps, GPTs, and plug-ins. I know our users will continue to push us to think bigger and better and help them accomplish more.

Feel free to comment and let me know your learnings and favorite moments from this past year. Thanks for all your feedback and support on this journey — we wouldn’t be here without you.?????