The way forward for AI Adoption: Optimization of the human-in-the-loop and data pipeline automations.

Computer vision as a technology gained lots of traction in the last decade. More and more enterprises put it in their top priority point that they want to pursue and put a plan, strategy and budget? under their digital transformation push for automation.?

Fast forward today, we see that 80% of those plans turned out to be a failure. Reasons are as varied as it could be. Sometimes enterprises couldn't find the talent, money or sometimes due to lack of understanding of use cases at leadership level. Which results in lost of patience.

Why did that happen?

It has multi-fold answers if you look into it.

Here are few most common reasons mention in volume of occurrence-

1. Enterprises couldn't find the right use case where they can get consensus from the stakeholders. This is definitely a great direction to deep dive as this requires a lot of awareness push via storytelling.?

2. Top leaderships don't understand the iterative nature of computer vision development and lose patience.

3. Computer vision development process is very fragmented and involves so many players, it is a miracle to manage them and see the success.

and many more...

But we're not talking about the unsuccessful ones, we have to look at those 20% of those who got successful to put at least 1 use case in production and what is the story behind them.

Computer development lifecycle

At high level, there are 3 stage that goes into computer vision lifecycle and those are-

1. Data collection
2. Getting data labeled
3. Model training and deployment

Enterprises get their data either by?

1. generating themselves or?
2. buy it from 3rd party data providers or?
3. use synthetic data recently in very few use cases where simulators are available(mainly autonomous driving).?

But the most crucial part in the whole process is to get training data in an easy to consume format by AI models.

Data annotation problem has been solved as there are a plethora of BPO companies to provide manual workforce and also numerous tools to manage the project which we generally call training data platform or data annotation tool.

But why still enterprises face such a hard time getting their data labeled.

Some of problems that they encounters are

1. The process is so manual and cumbersome and requires so much time, resource and budget. It is not uncommon to spend months/years to label/annotate million images/videos.
2. Errors in labels have a huge consequence as it would waste so much time and money to redo it and can exceed the timeline to double or triple to its initial estimate. Ensuring the label's quality is very critical and challenging.
3. With a large team and datasets it becomes very hard to manage the project, with no to very limited visibility to labels.

The core issue lies in not optimizing the “human-in-the-loop” powered data annotation process itself.

Human in the loop is a very opaque, inefficient and poorly managed part of this whole puzzle especially at a scale for medium to high stakes AI.

Recently, several platforms introduced features like active learning, autolabel, and pre-trained models to speed up the annotation process but introducing these is not enough. They can only speed up an already wrong process, which results in poor output.

How to optimize humans in the loop?

It requires a process which governs the complete lifecycle of training data generation activities involving data collection, data curation, data annotation and its quality assurance before the ingestion of training data into model training process. The process we’re proposing is the outcome of the best practices, implementing ML technique and experience that comes out of working on numerous different use cases.

We call it the "Smart feedback loop".

1. It essentially assists the annotators to get the real time feedback from their reviewers on their annotation, so that they can get their understanding better with every file that they label. It ensures that annotators won't have to perform repetitive tasks unnecessarily. It will help reduce the overhead from reviewer, supervisor ensuring annotators get the training of these error cases.
2. It identifies all the duplicate files and removes them from the pipeline as they come.?
3. In case of similar files, annotators can leverage his/her and team’s old work and save time.
4. Based on data uploaded, it suggests the best ML model that can be run and pre-label the majority of data, saving annotations time to draw the bounding box or segmentation.
5. Optimize the data pipeline to prioritize most informative data which need to be labeled first. Most informative data means putting focus on good data than big data maximizing the accuracy with the least amount of training data.
6. Always benchmark the quality of the label to the ground truth to save reviewer's time with metrics such as IOU in case of object detection.
7. Give high level and low level of analytics to the project managers about worker and all stakeholders efficiency and actionable steps.
8. Make the iterative process manageable and automated.
9. A number of mainstream and repetitive use cases such as pedestrian detection, traffic sign detection and driver distraction etc can be automated and provided in data curation activity such that human annotators get the guidelines to label only required labels which are subjective in nature or where the automated system is less confident. Eg a user upload 100,000 images for labeling with traffic lights, pedestrian, stop line, pedestrian crossing etc. The moment data upload is complete, users can choose to ask the system to analyze the data via labellerr’s prelabeling service curation screen. Analysis results would break the dataset into 3 buckets. First bucket where it is completely confident of images and labels such as pedestrians marked and need a quick small review. 2nd bucket could be where it requires a reviewer's feedback for small corrections on images and labels. the 3rd bucket is where complete human in the loop effort is required.

Benefits

1. By implementing "Smart feedback loop" into the computer vision data pipeline enterprises can optimize their human in the loop process. It can easily make humans in the loop 6-7X faster. Also reduces the time and effort made by review team by automating QA process.
2. Project management would become easier.
3. Computer vision experimentation would become affordable to even small and medium scale enterprises.
4. Increase in computer vision adoption.