AI Object Detection in Power Apps

Object detection can help expedite or automate business processes. In retail, it can help streamline inventory management, allowing retail leaders to focus on onsite customer relationships. In manufacturing, technicians can use it to speed the repair process by quickly accessing the manual for a piece of machinery for which the UPC/serial number isn't obvious.

Organizations of any size can use AI Builder object detection to add these capabilities for their own custom objects to their apps.

Collect images

To train an object detection model to recognize your objects, you have to gather images that contain those objects. Adhere to guidelines for image quantity and quality for better results.

Format and size

The images you'll feed your object detection model need these characteristics:

Format:

• JPG
• PNG
• BMP

Size:

• 6 MB maximum for training
• minimum width / height of 256 pixels x 256 pixels

Data quantity and data balance

It's important to upload enough images to train your AI model. A good starting point is to have at least 15 images per object for the training set. With fewer images, there's a strong risk that your model will learn concepts that are just noise, or not relevant. Training your model with more images should increase the accuracy.

Another consideration is to make sure your data is balanced. If you have 500 images for one object and only 50 images for another, your training dataset isn't balanced. This can cause the model to be better at recognizing one of the objects. For more consistent results, maintain at least a 1:2 ratio between the object with the fewest images versus the one with the most. For example, if the object with the greatest number of images has 500 images, the object with the fewest images should have at least 250 images for training.

Use more diverse images

Provide images that are representative of what will be submitted to the model during normal use. For example, let's say you're training a model to recognize apples. If you only train images of apples on plates, it might not consistently recognize apples in trees. Including different kinds of images will make sure that your model isn't biased and can generalize well. The following are some ways you can make your training set more diverse.

Building a object detection custom model

1. Sign in to Power Apps, and then select?AI Builder?>?Explore.
2. Select?Images?>?Object detection?AI Builder custom model type.
3. Select?Get started.

Select the model domain

The first thing you'll do when you create an AI Builder object detection model is to define its domain. The domain optimizes the model for specific use cases. There are three domains:

• Common objects:?The default value. Use this if your use case doesn't fit the specific applications below.
• Objects on retail shelves:?Detects products densely packed on shelves.
• Brand logos:?Optimized for logo detection.

Select a model for your domain and click?Next.

Provide object names

Next, provide the names of the items you want to detect. You can provide up to 500 object names per model.

There are two ways to provide object names:

• Enter object names directly in AI Builder.
• Select names from your Microsoft Dataverse table.

To choose objects from a Dataverse table, choose?Select from database?above?Choose objects for your model to detect, and then choose?Select object names. If you change your mind before you select your table, you can select?Add objects manually?to switch back.

Enter names in AI Builder

To provide object names directly in AI Builder, just enter the name in the space where the object is detected in the image. Then press?Enter?or select?Add new object?to continue.

• To edit an object name, select the object name and then make your change.
• To remove an object name, select the trashcan icon.

Select names from a database

If your data isn't in Dataverse, go to Prerequisites for information about how to import data into Dataverse.

1. Choose?Select from database?to see tables in your environment.
2. On the panel to the right, find and select the table that contains your object names.
3. Select the column that contains the names of your objects, and then choose?Select column.
4. From the list of objects in your table, select the ones that represent the objects you want to detect.
5. At the bottom of the screen, select?Next.

Upload images

Now let's move on to the image upload step. The pictures you collected ahead of time will now come in handy because you need to upload them to AI Builder.

1. Prepare your images in the storage location where you want to add them from. Currently you can add images from local storage, SharePoint, or Azure Blob Storage.
2. Make sure your images follow the qualitative and quantitative guidance.
3. In AI Builder, select?Add images.
4. Select the data source where your images are stored, and then select the images that contain your objects.
5. Confirm the images that appear in AI Builder before you upload them. Deselect any image you want to exclude.

6. Select?Upload?<number>?images.

7. When the upload is complete, select?Close, and then select?Next.

Tag images

This section explains the tagging process that's a key part of object detection. You draw rectangles around the objects of interest, and then assign a name to the rectangle that you want the model to associate with this object.

1. On the?Tag the objects in your images?screen, select the first image in your gallery.
2. Draw a rectangle around the object. To do this, press and hold your mouse at the upper-left corner of the object, and then drag down to the lower-right corner of the object. The rectangle should fully encompass the object you want your model to recognize.
3. After you draw a rectangle, you can associate a name to the object from the list of names you already selected.

4. Your tag is created when you see it surrounding an object

5. Navigate from image to image, and tag at least 15 images per object name to build a model.

6. After you're done tagging your images, select?Done tagging. Your data is saved as you create rectangles.

7. In the grid view, you can view a summary of all the tags you created and which images you created. This lets you know how much more work is needed to move forward.

8. Until you reach the minimum for content quantity, you can't move forward. After you have at least 15 images per object name, you'll be able to select?Next?at the bottom of the screen.

That's it! created a training set for object detection.

Train and publish your object detection model

Verify your data and then how to train, test, and publish your model.

1. On the?Train your model?screen, verify that you have the correct names and the correct number of images.
2. If the data is all good, select?Train?to kick off the training of your object detection model.

Quick-test your model

After your model is trained, you can see it in action from its details page. More information: Mange your model in AI Builder

1. From your model details page, select?Quick Test?in the?Last trained version?section.
2. Upload?an image that contains your objects to test your model.
3. Your model will be applied to the image you uploaded. This step might take a while.
4. After your model has finished running, the rectangles found will be drawn directly on the picture.

How to interpret your model performance score

If you quick-test your model after it's trained, a performance score appears on its details page. This performance score indicates how well the model did on the images you uploaded. This score isn't an indication of how well it will perform on your future images because it hasn't seen them yet.

If you upload fewer than 50 images for a label, you're more likely get a high score—as high as 100?percent. This doesn't mean your model is bulletproof. It means your model has made no mistake on a subset of the images you provided (called the?test set). The smaller the training set, the smaller the test set, and the more likely it is that your model will be right when the performance score is calculated.

Model performance scores are more reliable when you have more than 50 images per label, and when these scores remain stable even when you change the training set.

Publish your object detection model

From here, you can run more tests with other pictures. If you're happy with the results, you can publish?your model to use it in Power Apps or Power Automate.