Microsoft Bot Framework

MBF - Microsoft Bot Framework

The Microsoft Bot Framework -

is a powerful set of services, tools, and SDKs that provides a foundation for developers to build and connect intelligent bots.

simplifies the build and deployment Exploring the Future of UI/UX: Introducing the Microsoft Bot Framework Transcript In this video, we're going to introduce the Microsoft Bot Framework. So this is a bot platform made by Microsoft, it connects to many platforms, including the Cortana platform. And the bot builder is used to build bots, it supports .NET, Node.js, and REST. And there's an Azure Bot Service which Microsoft also has, which is an environment in which you can build and run bots. So you use the bot builder and the bot platform in the Azure Bot Service. And then finally, there's what's called Microsoft Cognitive Services, and those can be used to make intelligent AI bots. So that's kind of the framework of how all of those things fit together. So what are the core framework concepts? Well number one is, you have this idea of a channel. And the channel is a connection to an external app, so something like Facebook Messenger or Skype. And then we would have the bot connector, and that is basically this idea of connecting your channels, connecting your bot to your channels, without needing to write code for proprietary APIs.

So it's a connector that helps to get around those things. And then there's activity, an activity is the interaction between the user and the bot. And then there's a message, which is a type of activity, whether it's got text or attachments or whatever. As well as that, this is idea of a dialog, and that is a subprocess that can happen during the conversation. Like opening a dialog window, for example, and that can be useful for managing state within the conversation. As well as that, there's this idea of rich cards, which most bot platforms have, which is where messages have more advanced UI elements, like they have cards or animations or audio. And then if we look at Microsoft Cognitive Services, well, what is that? Well, it's part of the Microsoft Azure Platform, and it has services that can be used to make intelligent bots. So it's got something called the Language Understanding Intelligent Service. It's got custom speech service, computer vision API, face API. It's got a long list of additional tools that can be used to make really intelligent bots. So that's a quick introduction to the Microsoft Bot Framework.

of functional chatbots.

MBF - An Introduction

Bot development using Microsoft Bot Framework can be done in more than one way.

• Using C#
• Using NodeJS

In this course, we are focusing on the development using C# and Visual Studio 2017.

MBF - Channel

Channel is simply a link or connection between the bot and an app in which it is deployed.

Examples of channels include websites and mobile applications including Slack, Facebook, email, etc.

Using MBF, you can build the bot and publish it to multiple channels with little effort without having to make different bots for different apps.

MBF - Utterance

Utterances are essentially the messages that the user of a chatbot sends to the chatbot.

For instance, in the above image, the blue bubble is an utterance.

MBF - Intent

Intent denotes the intention of the chatbot user by the message that he sent to the bot.

The bot's machine learning model will recognize the intent so as to give an appropriate reply.

For example, in

Book me a ticket for today's movie.

the intent could be book ticket.

The next card shows few more example messages along with their corresponding possible intent.

MBF - Intent Examples

Utterance

Intent

Hello

greetings

How are you?

feelings

Help me!

help

Show me Pikachu

info_pokemon

Bye

goodbyes

MBF - Entity

An entity is something that adds more information to an intent. It modifies or fine tunes the meaning of the intent.

It represents an object or a value that the bot can process.

In

Book ticket for the flight at 8:00PM on Friday.

Intent is Book ticket. And the entities are flight, 8:00 PM, Friday. These are values that can be used.

The ticket could be for a bus or a train or something else. The entity flight narrows it down.

Friday and 8:00 PM fine tunes the intent further.

MBF - Entity Examples

Utterance

Intent

Entity

Could you book a restaurant in Manhattan?

RestaurantBooking

Location: Manhattan

Please turn off the blue light.

TurnOff

Device: blue light

Play pop music

PlayMusic

Genre: pop

MBF - An Example Utterance

An example utterance with the intent and entities outlined.

You can see a simulation by Facebook's wit.ai here where you can ask something and the intent will be displayed.

MBF - Prerequisites for .NET development

For using and deploying bots using MBF, you need the following:

• Microsoft Visual Studio 2017
• LUIS (Language Understanding Intelligent Service)
• A Microsoft Azure subscription

MBF - Environment setup

To set up the template for bot development in Visual Studio:

• Download .vsix file from Visual Studio Marketplace through this link to use the latest version.
• Extract the contents of the file and copy to the project templates of Visual Studio (in a path like%USERPROFILE%\Documents\Visual Studio 2017\Templates\ProjectTemplates\Visual C#\)

MBF - Basic bot creation

Open Visual Studio 2017 and under Visual C# section, select Bot Builder Echo Bot and click OK button to create the project.

MBF - Project Structure

The project structure should be now as shown in the above picture.

Build the project to restore all the NuGet packages.

MBF - Bot emulator

The Bot Framework emulator is a Desktop application that can be used to test the bot developed using the Microsoft Bot Framework locally.

It can be downloaded from here.

The emulator needs the endpoint URL of the bot. Microsoft App Id, password and locale are not needed if you are testing the bot locally.

MBF - a Walk-Through

The MessageController class will be a single HttpPost method with an Activity object as its parameter.

Activity object is used to pass messages between the bot and the Channel.

A channel is the link between the bot and the app in which the bot is deployed.

The activity types supported in bot builder for .NET include

• Message
• Typing
• EndOfConversation

The most common one is Message.

MBF - a Walk-Through

The message received from the user invokes the root dialog - RootDialog.

• RootDialog is a class that inherits the IDialog<object> interface.
• StartAsync(): This is the method that is invoked on navigation to the Dialog class.
• MessageReceivedAsync(): The message received is processed and a response is sent.
• Context.Wait(): This defines which method the current dialog should execute when the next message of the user is received.
• Context.PostAsync(): This reply back the response to the user. It is similar to the console.writeline()

MBF - an Example

In the example shown above, at first, the root dialog invokes the 'new order' dialog. This is because everything starts with the root dialog.

At this point, new order dialog takes over and does not relinquish control until it invokes another dialog or it closes.

When the 'new order' dialog closes, the control is returned back to the root dialog.

Otherwise, new order could call another dialog like Product Search Dialog.

MBF - Dialog Life Cycle

A dialog takes over control when it is invoked. All new messages will be processed by that dialog till it gives up control by closing or invoking another dialog.

• context.Wait() method can be used to specify a callback function to call when the next message from the user is received.
• context.Done() may be used to close a dialog.

All dialog methods must end with context.Wait(), context.Fail(), context.Done() or invoke another dialog with context.Forward() or context.Call().

This is because the application should know what to do the next time the user sends a message.

MBF - QnA Maker

QnA maker is a REST API that teaches a machine learning model to respond to users' questions in a natural way.

MBF - QnA Maker

QnA Maker can extract the data in a structured or semi-structured document like an FAQ or a user manual so that it can build question-answers based on that.

This will come in useful in deciding the appropriate response of the bot.

A knowledge-base is created from the source documents.

It is based on the cloud and requires no code to generate the knowledge base.

It is scalable.

QnA Maker - Process

QnA Maker has two main facilities:

• Extraction
• Matching

Image Source: Microsoft

QnA Maker - Process

Extraction: The question-answer data is built from the source documents.

Matching: After training the AI model with the extracted data, the knowledge base is published by which an endpoint is created.

The user messages are given to this endpoint and the model assigns scores to the possible set of answers and selects the top scoring answer as the response.

QnA Maker - Knowledge Base

A knowledge base of QnA Maker consists of a set of questions and their answers along with some metadata.

Once the app is published, the user inputs are analysed by the AI model and appropriate response is rendered.

At the same time, the model will keep learning to cope with new conditions.

QnA Maker - a Scenario

In the IT helpdesk of an MNC, there are many standard questions that the users might ask. Even though most of these questions are answered in the FAQ pages, users might find it inconvenient to find the FAQ page and search for the question in which they are interested.

To save time, a bot powered by QnA Maker could be employed to answer the frequently asked questions.

MBF - Dialog

Dialogs are used to model the conversation during Bot creation.

User interface of bots is made up of dialogs which denote messages.

Dialogs enable the bot developer to logically separate various areas of bot functionality and guide conversational flow.

They are used to build a Conversation User Interface (CUI).

MBF - Dialog

It is better to use different dialogs for different use cases.

For example, different dialogs must be used for booking a ticket for a movie show and for searching the list of available movies.

When switching from one dialog to another. The most recent dialog will be on top of the stack.

It is advisable to create a dialog stack so that we can return to a previous point if needed, unstacking dialogs on the way.

MBF - PromptDialog

PromptDialog is a subclass of the Dialog class used in Microsoft Bot Framework that can be used to present simple prompts to the user.Column 1

Column 2

PromptDialog.Text()

Prompts for a string input

PromptDialog.Choice()

Prompt which provides options to the user for selection

PromptDialog.Confirm()

Prompt for the Yes/No selection only

PromptDialog.Attachment()

Prompt for an attachment upload to the bot

PromptDialog.Number()

Prompt for a double

MBF - MultiDialog

MultiDialogs are used when we need to establish a connection between two dialogs.

It is achieved using

Context.Call(ChildDialog, ResumeAfterDialog)

This must be used in the parent dialog from where you would like to navigate to another dialog (child dialog). ResumeAfterDialog is the method that will be executed after the child dialog is done and control returns to the parent dialog.

MBF - MultiDialog

Another way to establish connection between two dialogs is by using

Context.Forward(childDialog,ResumAfterDialog,Object,CancellationToken)

Object has the message which is to be communicated between the two dialogs.

Ensure that you specify Context.Done() in the last method of the child dialog as it closes the context of the current dialog and goes back to the parent dialog. As the dialogs are added to the top of the stack, the closure needs to be taken care.

MBF - Scorable Dialog

The user might interrupt a conversation with the bot to ask something else when the bot is expecting a reply to the previous message. The user could try messages like 'help', 'cancel', 'start over', etc for this.

Scorable dialogs can be used to handle these interruptions in a graceful manner.

All incoming messages are given a score between 0 and 1.

After completion of the scorable dialog execution, the conversion is returned back where it is left off.

MBF - Scorable Dialog

To make a dialog scorable, we must inherit the abstract class named ScorableBase.

And you must provide implementations for the following IScorable methods:

• PrepareAsync(IActivity item, CancellationToken token) – accepts the incoming message activity, analyzes and sets the dialog’s state.
• HasScore(IActivity item, string state) – Checks the state property to determine if the scorable dialog should provide a score for the message. If returned false then it is ignored.
• GetScore(IActivity item, string state) – triggered only if the HasScore() method return true.
• PostAsync(IActivity item, string state, CancellationToken token) – defines the core actions to be performed for the scorable class.

For more about dialog management, see https://channel9.msdn.com/Events/Build/2017/P4070.

MBF - FormFlow

Formflows are useful to manage a guided conversation in a chatbot.

The bot may be needing a minimum level of information in order to help the user. Formflow enables the chatbot to ask appropriate questions to the user to fill in the gaps in the available information.

Though it is easier to set up a functional bot with FormFlow, it is at the cost of flexibility that would be possible with Dialogs.

MBF - FormFlow

With Formflow we can define our form fields and have the user complete them, whilst getting help along the way.

You can break the conversation in between and modify the previous conversation data.

It can be used with Dialogs to increase its functionality.

Formflow automatically generates the dialogs that are necessary to manage a guided conversation

MBF FormFlow - Basic Features

The information or prompt questions need to be specified beforehand for the creation of the Formflow.

The bot will collect the corresponding data from the user as answer to these questions.

You can define the form by creating a C# class that contains one or more public properties to represent the data that the bot will collect from the user.

MBF FormFlow - Implementation

To use FormFlow, you should bring in its namespace into the program. It can be done like

using Microsoft.Bot.Builder.FormFlow;

The form is created using FormDialog.FromForm(BuildFormMethodName) method.

BuildFormMethodName, in turn, will take care of the FormBuilder<> creation.

MBF FormFlow - FormBuilder Class

The FormBuilder class will have the following methods:

• Field(): The property name defined to prompt the user.
• AddRemainingFields(): This specifies to include all the leftover property prompts apart from the property names mentioned using the Field().
• Build(): Once all the prompts are fulfilled, it builds the user responses and stores in the class object provided for the FormBuilder.
• OnCompletion(): It defines which method to be executed once the FormBuilder is completed.

MBF Rich Cards - Overview

The messages between the bot and the user are usually text strings.

But some channels support multimedia content. MBF allows this in the form of rich cards.

It should be noted that some channel does not support some of the rich cards.

There are eight types of rich cards available:

• Adaptive card
• Animation card
• Audio card
• Hero card
• Thumbnail card
• Receipt card
• SignIn card
• Video card

MBF Rich Cards - Animation Card

Can play animated GIF images and short videos.

Its properties include autoloop, autostart, image, subtitle, sharable, etc.

MBF Rich Cards - Hero Card

A Hero card usually consists of a large image, buttons, and text.

Its properties include images, buttons, subtitle, tap, text, title, etc.

MBF Rich Cards - Thumbnail Card

This is like a version of the Hero cards with a small image.

It usually contains a thumbnail image, buttons, and text.

Its properties include buttons, images, subtitle, tap, text, title, etc.

MBF Rich Cards - Video Card

This is the card used for playing videos.

Its properties include aspect, autoloop, autostart, image, media, subtitle, text, title,

MBF Rich Cards - Sign-In Card

Used to enable a bot to request the user to sign in.

Usually contains text and buttons that the user can click to initiate the sign-in process.

Its properties include button, text, toAttachment, etc.

MBF Rich Cards - Receipt Card

MBF Rich Cards - Adaptive Card

Adaptive cards are customizable.

Can contain any combination of text, speech, images, buttons, and input fields.

You can tailor it to your needs.

To create an adaptive cards, install Microsoft.AdaptiveCards from the NuGet Package.

See https://adaptivecards.io/ for more.

Allows the bot to furnish the user with a receipt.

Typically has a list of items to include on the receipt and other related information.

Its properties include buttons, facts, items, tax, title, total, vat, etc.

MBF Rich Cards

The usage of rich cards enriches the Conversation UI (CUI) of the bot.

Rich Cards are converted to attachments and are add to the Attachments property of the Activity object.

The only limitation of the Rich Cards is the channel in which bot is hosted as some channels may not support some cards.

Multiple rich cards can be shown at once as a list or a carousel.

MBF - LUIS

Language Understanding Intelligence Service (LUIS) is part of Microsoft Cognitive Services that can be used on any device or platform.

• A cloud-based API service
• Applies custom machine-learning intelligence to predict overall meaning, and pull out relevant, detailed information.

Microsoft's Cortana uses LUIS for language and semantic interpretations.

MBF - LUIS Entities

An entity is a word or phrase in the utterance which will provide the bot additional data.

It's not mandatory for an utterance to have an entity.

An entity represents a class or group of objects which have something in common.

Entities add more information to the intent.

An entity is not owned by a single intent. They are common to all intents.

LUIS - Entity Types

LUIS has different categories of entities. They includeLUIS entity

Description

Pre-built entities

LUIS provided built-in entities such as Number, datetime, geography etc

Simple Entity

describes a single concept

Composite Entity

made up of other entities that form parts of the whole

Hierarchical Entity

members are in the form parent-child

List Entity

A list of values

Regex Entity

Allows LUIS to extract well formatted from an utterance using a regular expression

LUIS - Best Practices

Do

Don't

Define distinct intents

Add too many example utterances to intents

Tread a middle path between too specific and too generic for intents

Use LUIS as a training platform

Build app iteratively

Add many examples of the same format ignoring other formats

Add phrase lists and patterns in later iterations

Mix definition of intents and entities

Add example utterances to None intent

Create phrase lists with all possible values

Leverage suggest feature for active learning

Add too many patterns

Monitor performance of app

Train and publish with every single new example utterance

LUIS - App Publication

After the publication of the LUIS app, the user utterances first goes to LUIS for analysis.

LUIS sends a JSON response to the bot based on which it responds to the user.

The LUIS app ID and key need to be noted after the app is published.

The app ID and key need be specified in the web.config file and provided to the constructor of the LUISDialog.

public LUISDialog() : base(new LuisService(new LuisModelAttribute(

        ConfigurationManager.AppSettings["LuisAppId"],

        ConfigurationManager.AppSettings["LuisAPIKey"],

        domain: ConfigurationManager.AppSettings["LuisAPIHostName"])))

{      }

MBF - Use Case

Megha who joined recently in a team finds difficulty in knowing the company processes, compliances, project information etc.

Chatbot named Atom came to her rescue.

Atom identifies the Megha’s identity and answers the questions asked.

It helps to raise a ticket in case of any issue internal to the company.

It even provides a way to find the project documents.

Atom is available in the Skype for Business which is used as the channel for Megha's communication in the team.

MBF - Use Case

Let us examine the logic flow of Atom:

1. Employee's identity is configured using Azure Active Directory.
2. Using the Cognitive Service QnA maker, it answers the FAQs.
3. By integration with the APIs, it helps to raise a ticket.
4. Easy access to the project documents using the Azure Search.
5. The intention of the user's query is understood with the help of LUIS.
6. The availability of the Atom bot in multiple channels for easy access of a wide range of users.

Azure - Publishing a Bot

Once we are done testing the bot locally, it may be published to Azure so that it can be accessed across channels.

Bot publishing using Microsoft Azure may be done in two ways - from visual studio or from Azure.

Azure - Steps for Publishing a Bot

Visual Studio and Azure Portal

1. Clean and build the project to avoid errors and missing any NuGet packages.
2. Right click on the 'Project' menu and select the 'Publish' option.
3. Select the 'App Service' as the publish target with 'Create New'.

Azure - Steps for Publishing a Bot

Visual Studio and Azure portal

1. In the right hand corner, ensure that the correct ID is shown as your Azure subscription.
2. Fill the required fields and select 'Create'.

Azure - Steps for Publishing a Bot

Visual Studio and Azure portal

1. Once published, find the bot's public URL.
2. Go to the Azure portal and create new bot channel registration source.
3. Ensure that Microsoft AppID and Password are copied back to the app service created for bot from Visual Studio.
4. Test the bot for successful conversation using the 'Test in Web Chat' option in Bot Management.

Azure - Steps for Publishing a Bot

Azure Portal

1. Log in to the Azure portal.
2. Click on 'Create New Resource' and from under 'AI + Machine Learning', select 'Web App Bot'.

Azure - Steps for Publishing a Bot

Azure Portal

1. Fill all the required fields and click on 'Create'. This creates an Azure bot service, which in turn creates an associated app service.
2. Go to Visual Studio and select 'Publish' from 'Project' menu.
3. Select 'Select Existing' for the app service publish target and click 'Publish'.

Azure - Steps for Publishing a Bot

Azure Portal

1. Select the app service created in the Azure portal.
2. Get the URL of the bot after the bot is published.
3. Test the bot for successful conversation from 'Test in Web Chat' option in Bot Management.
4.