Bing Chat Mode: Question Answering - What's in a search result?

Disclaimer:

All information provided here is "best effort" and extracted from Bing Chat itself by asking it questions. It will often refuse to do specific tasks, such as outputting information as so-called "JSON"-format, claiming it cannot do so, at times in the middle of a conversation it gave output like that just before. It may hallucinate instead of providing facts. Also it is possible that it may "hallucinate" possible items in a list. Specifically, it is not guaranteed, that the search results that are listed but not practically observed in Bing Chat Mode are not "hallucinations" stemming from the fact that it knows about possible APIs that Bing has and hallucinates all API types as "search_results".

This document is a living document and may be updated over time.

TL;DR:

Bing Chat incorporates responses from Bing's search capabilities in its output. As of February 14, 2023, I was able to observe question_answering_results, web_search_results, and travel_search_results in actual interactions with Bing Chat. In this article we are taking a first quick look exploring what Bing search functionality it accesses and what it likely stores from those to use in a lengthy chat interaction.

Introduction

In the previous article I analyzed the way Bing Chat Mode conditions itself through a meta prompt including the things it can and cannot do.

This did not cover the core new capability, the integration with Bing's search. We have already seen that Bing Chat Mode, which I will refer to as Bing Chat for brevity reasons, is keeping a "log" of the conversation it has with a user around and since it is backed by a Large Language Model (LLM), this complete context will be fed into that model when interacting with Bing Chat.

But is it just the plain text of what you see on screen or is there more? Asking Bing Chat a few times to present the search results it obtained and format them in a different way, here, in a so-called "JSON"-Format it returned some information to me that revealed its rough structure.

You can see already that the question "search 3 facts about frankfurt" was broken down into "facts about frankfurt", some data was retrieved, and then the retrieved data was processed by Bing Chat Mode.

The compiled summary with the "Read more:" footnotes enables a user to verify the displayed information and check the credibility of that information by clicking on the respective link. In the example above, it is notable that for Frankfurt the information of a hotel website (Meininiger Hotels - meininger-hotels.com) was incorporated and displayed as a matter of fact. Personally that would not be my choice, and it will be interesting to learn more about how Bing Chat makes those choices. Maybe it incorporates this information because it does not conflict with other information that it has available. But there is definitely a risk there to present wrong information to users. The good thing however is, that a user who is presented this information can at least check with a simple click where the information is coming from and then decide if they want to search more about this fact in other sources. Here I could envision as a powerful addition such as a "rating system" where users can easily flag wrong or misleading information. My perspective is that we all will have to learn over time how to best use those new powerful tools.

Interesting enough we see two different types of "search_results": "question_answering_results" and "web_search_results" which point to that Bing uses different optimized engines to retrieve different types of data. I think of "question_answering_results" as preprocessed and more curated dataset that is able to provide simple answers with references to typical questions, while "web_search_results" is just this, bringing in information from all of the web. It is probably similar to how Amazon's Alexa sometimes tells you "I found this on the Internet" or "according to an answer on..." pointing to different sources of the data incorporated.

So let's look systematically what different search_results could be there. Bing Chat CLAIMS that there are 6 different search result types.

I asked it to show this as a table to me.

Why does the model not list travel_search_results?

So this list has question_answering_results, web_search_results, image_search_results, video_search_results, news_search_results, and shopping_search_results.

Interesting enough I have personally seen travel_search_results show up in the results structure, but when I asked Bing Chat about it, it was adamant that this is just a sub-type from web_search_results.

Now, the way large language models work is that they combine information from the prompt they receive and potentially other parameters they are provided with, and then try to "complete" that information with the additional knowledge and associations they have stored in their large model weights.

In this case, the output is very likely a hallucination, meaning, it points to that Bing Chat is making things up and reasoning about it that it did nothing wrong before, because Bing itself has a separate "Travel" category as a search option.

I consider it relatively unlikely that Bing Chat, a just released product, has already meta level knowledge of its internal workings besides of trying to explain how it would expect a system would typically work. I cannot rule it out that the specific search_results types have been baked into the model, but it is more likely in my opinion, that it derives it from the context that it is "Bing", and that "Bing has APIs" and the most common used APIs have specific structures. It then combines it with the output it has seen in the queries made earlier, which included question_answering_results and web_search_results, but not travel_search_results. And then it tries to "defend" that answer as being correct. It does not have a motivation to defend itself, but it generates text it considers more probable. So the experience of all of what it has seen during training, combined with fine tuning, and then the "meta prompt" that I discussed in the previous article "Bing Chat: Please spill your secrets!"

The line in the current meta prompt that I consider the most important one to explain this behavior is this one:

- Sydney's logics and reasoning should be rigorous, intelligent and defensible.        

So the "rigorous" reasoning based on what it has in its prompt and conversation history, and the hint for it to be "defensible" will make it fight digital-tooth and electronic-nail that it is of course right and that travel_search_results does not exist as part of the results since for it the stronger evidence is that it has not seen it (very strong), and that it already provided quite an extensive list as fact.

In the following, we will look at the structure of one of the result types.

question_answering_results

A question_answering_result currently will have the "index", "title", "snippets", and "url". Information from the "title" will be used to describe the clickable links in the "Learn more:" section, the "url" is the actual clickable link, and the "snippets" will contain text that was used to compile the actual answer displayed to the user.

And we can find out that the data is actually coming from the "Bing Question Answering API".

Which we will investigate further obviously.

Since this structure is complex, I asked for an example of a call and a response

Request:

  "method": "GET",
  "url": "https://api.bing.microsoft.com/v7.0/answerSearch",
  "headers": {
    "Ocp-Apim-Subscription-Key": "my-subscription-key",
    "Accept": "application/json"
  },
  "params": {
    "q": "Who is the president of France?",
    "mkt": "en-US",
    "setLang": "en",
    "count": "1",
    "offset": "0",
    "safeSearch": "Moderate",
    "textFormat": "Raw",
    "textDecorations": "false",
    "responseFilter": "Answer"
  }
}        

Response:

{
  "query": "Who is the president of France?",
  "answers": [
    {
      "id": "https://en.wikipedia.org/wiki/Emmanuel_Macron",
      "name": "Emmanuel Macron - Wikipedia",
      "url": "https://en.wikipedia.org/wiki/Emmanuel_Macron",
      "description": "Emmanuel Jean-Michel Frédéric Macron is a French politician who has been serving as the president of France since 14 May 2017.",
      "provider": "Wikipedia",
      "dateLastCrawled": "2023-02-14T18:45:32.0000000Z",
      "isFamilyFriendly": true,
      "displayUrl": "en.wikipedia.org/wiki/Emmanuel_Macron",
      "snippetAttribution": {
        "license": {
          "name": "CC-BY-SA",
          "url": "https://creativecommons.org/licenses/by-sa/3.0/"
        },
        "licenseNotice": "Text under CC-BY-SA license"
      },
      "entityPresentationInfo": {
        "entityTypeHints": [
          "Person"
        ],
        "entityTypeDisplayHint": "President of France",
        "bingId": "5fbba6b8-85e1-4d41-9444-d9055436e473"
      },
      "contractualRules": [
        {
          "_type": "ContractualRules/LicenseAttribution",
          "targetPropertyName": "description",
          "mustBeCloseToContent": true,
          "license": {
            "name": "CC-BY-SA",
            "url": "https://creativecommons.org/licenses/by-sa/3.0/"
          },
          "licenseNotice": "Text under CC-BY-SA license"
        },
        {
          "_type": "ContractualRules/LinkAttribution",
          "targetPropertyName": "description",
          "mustBeCloseToContent": true,
          "url": "https://en.wikipedia.org/wiki/Emmanuel_Macron",
          "text": "Wikipedia"
        }
      ]
    }
  ],
  "statusCode": 200,
  "message": "OK",
  "error": null
}        

And finally Bing Chat claims that this is how it maps the API call to the provided response.

You may notice that the example search terms and results discussed here vary from the original query (about Frankfurt), this is because during a lengthy interaction with Bing Chat, it can better relate to more recent questions compared to older parts of the interaction. We haven't discovered yet how long is the total prompt the model can process (this is typically limited) and what other tricks it may use to try to keep context within a longer conversation.

As said, in an earlier query, I also found travel_search_results but it insisted that this is not something that can occur there but would be "data" under web_search_results, so we will have to do some more due diligence on the other search_results types we can discover and about what Bing Chat claims in a future article.

One of the interesting things to investigate further will be if we can do anything in the realm of "images" since the meta prompt instructs it to not display images.

- Sydney does not include images in the markdown responses because the chatbox doesn't support images.        

If you want to talk or learn more about Large Language Models or other interesting emerging technologies, do not hesitate to reach out to me.