From Search to Research

Everyone now uses a search engine to find information almost daily. You may have even found this post via search.

Search has been a big part of my life for more than a decade. In 2007, I left my job as a computer vision researcher at Microsoft’s research lab in Beijing to join the Bing team in the US. Satya Nadella was leading the small team at the time, and we were excited about learning the search ads business. Most importantly, we were trying to create a different search experience that was more helpful to people.

Bing’s continued growth has been very exciting and encouraging to me. Bing now powers 33% of searches on PCs in the US and has a worldwide presence that continues to grow – including 25% search share in the UK. And not only does Bing integrate data from many external partners like Reddit, it powers sites like Microsoft Academic which provides a comprehensive way to find research papers.  

The progress that we’ve made to date is great, but what’s next? It’s to move from search to research.

I spoke about this topic today in Los Angeles at the 11th ACM International Conference on Web Search and Data Mining (shortened to WSDM, which is great considering the wisdom you can get from search!) and underlined how much further we can go to create the next generation of intelligent search.

Traditional search relies on keywords and query-URL matching to find answers. Today, there is not much difference between leading search engines when it comes to finding web sites. But better speech recognition, interaction with personal assistants and new mobile experiences have changed the way people ask questions. Today, questions are based on natural language and are longer, making it necessary to deliver more precise answers. How?  

With advances in AI like machine reading comprehension, we are working to make Bing a research tool that delivers direct answers, different perspectives and clarity through dialogue. 

Direct answers. Our ability to provide more direct answers changes dramatically with better machine reading comprehension (MRC). Just a few weeks ago, our team at Microsoft surpassed human reading comprehension in the SQuAD benchmark. MRC helps us extract the best answer from a passage in the web document by matching the semantic meaning of the query and the passage. We also offer contextual answers by making inverse inferences from machine learning results into something users can relate to. For example, when you search “the size of Iraq” on Bing, you get not only “169,235 square miles” but also “about equal to the size of California.”  

Multiple perspectives. Providing an answer is simple until you consider that there are many different opinions. How do you know the right answer if you don’t have different perspectives? Consider this: a question that is framed positively – e.g. “Is kale good for me?” will likely result in an answer about kale’s benefits. However, a question framed more negatively – e.g. “Is hot yoga bad for you?” will likely return results that do not acknowledge the greater benefits of hot yoga. To address this issue, we’ve added Multi-Perspective Answers to Bing. This isn’t new – in fact, we announced this back in December. When asking a question that may return multiple valid perspectives from authoritative sources, you will now see two answers that offer different opinions on the same topic. We are also able to provide more authoritative answers by letting users know when MRC finds multiple reputable sources that provide an equivalent answer or different attributes.

Dialogue is a critical part of evolving search. The goal is to understand your intent so every search helps you quickly complete the task. We've created a new way to search that is interactive and can build on your previous searches to find the best answer. Now, if you need help figuring out the right question, Bing will help you with clarifying questions to better refine your search and get you the best answer the first time.

Here’s an example: When we search for “headache” on Bing, additional options are presented to help us narrow down what we want to know. When we click on “types of headaches,” it takes us to other options until we find what we’re looking for.

A new area we’ve been working on is “Conversational Context”, which is critical to dialogue. Our model uses sequence-to-sequence with context embedding for better conversational query understanding. On the backend, both the query and its context are used as the input. Then, multiple layers of LSTM and attention help generate the reformulated query and eventually provide and answer. Take a look:

Simply put: this is an important capability to make conversations between people and machines more natural, and we will be shipping it with Cortana in the coming weeks.

We are very excited about the future of Conversational AI. Personal assistants like Siri, Alexa and Cortana have used IQ skills to help users with proactive and reactive requests. One of our latest innovations is the social chatbot that relies on EQ to build an emotional connection with humans. Microsoft started on its social chatbot journey in 2014 with Xiaoice – our most popular bot from China. We now have social chatbots in Japan, the US, India and Indonesia. From interacting with these 100-millions of social chatbots we may learn that intelligence is indeed in the dialogues.

I expect that some of you reading this are not avid Bing users, or haven’t been to Bing in a while. Maybe it’s time to give Bing a try. With new features and innovations to drive direct answers, perspectives and dialogue that help bring more intelligence to the search experience, we’re changing the nature of the search that you do every day.