Building the New Bing: Multi-Modal Capabilities

One of the many new features we announced on May 4 was Visual Search in Chat. We have been running test flights of this feature in Bing Chat, and I’m happy to share that it is now fully rolled out on desktop and Bing mobile. As we’ve done throughout the preview, we are seeking feedback on the experience and will apply learnings as we continue to iterate on this feature.?

Adding multi-modal capabilities will allow people to engage with AI through the input of images, as well as text. With Visual Search in Chat, you can enter the URL of an image, drag and drop it into the search box, or upload your own from your device and Bing Chat will provide answers to your questions with an understanding of the image you provided.?

Whether you want to ask, "what’s in this image," "how do I make this?," "how can I save this plant?" or "what’s a recipe I could use to make this at home?," Visual Search in Chat will provide answers to your questions taking the image you have supplied into consideration.?

One of my favorite ways to use multi-modal is in combination with Bing Image Creator. You can for example upload an image that you like and then ask Bing to create a similar image but with some changes, like in the example below.?

As we continue to bring new experiences to the preview, responsible AI remains at the center of everything we build with the new Bing and Edge. In addition to partnering closely with OpenAI on their alignment work, we have developed a safety system that is designed to mitigate failures and avoid misuse with things like content filtering, operational monitoring and abuse detections, and other safeguards. For example, we blur the faces of people in images prior to the image being run through the model to protect privacy. Users can report concerns via the feedback tool or Bing’s Report a Concern form.?

We can’t wait to see how you’ll use Visual Search in Chat and as always, are eager to hear your feedback.?