Using Machine Learning to Build Better Machine Learning: How Tech Giants are Relying on AutoML to Build Better Neural Network Architectures

Automated machine learning(AutoML) is becoming one of the most popular topics in modern data science applications. Often, people see AutoML as a mechanism to use out-of-the-box machine learning models without the need of sophisticated data science knowledge. While theoretically, this argument makes sense the reality if a bit different. In the current stage of artificial intelligence(AI), most real world applications require some level of machine learning knowledge. The scenarios that you can solve with a vanilla API like the Watson Developer Cloud or Microsoft Cognitive Services are very basic and represent only a small percentage of the broader spectrum of machine learning scenarios. If that’s the case, then we should wonder what’s the real value of AutoML.

In our experience at Invector Labs, the best way to think about AutoML is as a helper or enabler of machine learning applications that complements the skill of a data scientists. More specifically, we believe the most viable case for AutoML in the real world is to help data scientists to select and architect the right machine learning model for a given problem.

The Challenge

Model selection is one of the most difficult aspects of building a machine learning solutions. Somewhat ironically, for all the science and match baked into machine learning applications, model selection remains a highly subjective tasks that relies on the opinions of experts. For any given scenario, the number of machine learning models that can solve it is incredibly large so how can we really know if we are using the most optimal model for the job? Even worse, even if we have selected the right machine learning technique, how can we be sure we have the right neural network architecture in place? And once we settled on a specific architecture, how can we know the correct hyperparameter configurations? Those questions hunt data scientists throughout the entire lifecycle of a machine learning application. Furthermore, the more accuracy is needed for a machine learning problem the more time is spent in the model selection process.

Not surprisingly, the process of selecting and architecting a machine learning model is an extremely time-consuming exercise that never delivers an exact answer. Paradoxically, this is the type of problem in which machine learning excels so can we get creative and model the process of selecting a machine learning architecture as machine learning problem in and out itself?

AutoML to the Rescue

Model search is one of the use cases that seems like a perfect fit for AutoML. Given a dataset, a series of optimization metrics and some constraints in terms of time or resources, AutoML methods should be able to evaluate tens of thousands of neural network architectures and produce an optimal result. While effective data science teams might be able to evaluate a dozen models for a given problem, an AutoML method can quickly search through tens of thousands of architectures in a relatively manageable time.

Using machine learning to build better machine learning models seems like something out of an IronMan movie ?? Is this really happening in the real world? Absolutely! Here are three of my favorite high-profile case studies for AutoML in mission critical applications.

Salesforce.com TransmogrifAI: The Brain Behind Einstein

Salesforce.com’s Einstein is one of the most widely adopted machine learning applications worldwide. Ultimately, Einstein solves a series of machine learning scenarios such as sales forecasting or lead prioritization which are omnipresent in sales and marketing applications. However, what makes Einstein unique is the fact that it machine learning models are able to operate across completely diverse Salesforce.com configuration in a self-service manner. Each customer might have completely different sales and marketing schemas and yet Einstein can still do the job.

The magic behind Salesforce’s Einstein is powered by an open source framework called TransmogrifAI. Conceptually, TransmogrifAI is an AutoML based framework for creating machine learning models against structure datasets(rows and columns). More specifically, TransmogrifAI leverages AutoML is five fundamental areas of a machine learning workflow:

· Feature Inference: Extracting features from a given datasets.

· Transmogrification: Transforming features into numeric values.

· Feature Validation: Reduce dimensions, identify potential bias, etc.

· Model Selection: Conducting search across thousands of potential models.

· Hyperparameter Optimization: Tuning the hyperparameter configuration.

Given the Salesforce.com footprint, TransmogrifAI might be considered one of the largest AutoML applications in the world.

Azure ML: Helping Developers Select the Right Machine Learning Model

Last year, Microsoft Research conducted an experiment of leveraging AutoML and probabilistic programming for automating model selection. The results were captured in a very popular research paper and represented a breakthrough from the performance. In a matter of months, the AutoML approached pioneered by the Microsoft Research team were implemented in Microsoft’s hallmark machine learning product: Azure ML.

The latest release of Azure ML leverage AutoML to streamline model selection. The platform includes an AutoML service that regularly recommends new machine learning pipelines to evaluate given a specific problem. The execution of the pipeline is done in the customer’s Azure ML instance while the AutoML service only sees the results and uses them to make better recommendations.

The AutoML implementation in the Azure ML stack is one of the most complete I’ve ever seen. The current version supports classification and regression ML model recommendation on numeric and text data, with support for automatic feature generation (including missing values imputations, encoding, normalizations and heuristics-based features), feature transformations and selection. Developers can use AutoML through the Python SDK or via Jupyter Notebooks.

Waymo: Automated Model Selection for Self-Driving Vehicles

A self-driving vehicle is something like a big group of machine learning models on four wheels ??. Machine learning enable all the intelligent features of self-driving vehicles like helping cars see their surroundings, make sense of the world, predict how others will behave, and decide their next best move. Alphabet’s subsidiary Waymo is at the forefront of self-driving vehicle technologies and, as a result, is constantly innovating in the machine learning.

Recently, the Waymo engineering team published a detailed blog post of how they are leveraging AutoML to automate model selection on different machine learning applications. Specifically, the Waymo team leverages an AutoML technique known as NAS cells which has proven to be very effective in image analysis algorithms.

At Waymo, AutoML is used to explore hundreds of different NAS cell combinations within a convolutional net architecture (CNN), training and evaluating models for Waymo’s LiDAR segmentation task. The experiment has been producing CNN architectures that are performing with 20–30% less latency and 8–10% lower error rate that hand-crafted models.

As you can see from these examples, AutoML is gaining a position as one of the important elements of highly scalable machine learning architectures. The tools and frameworks for leveraging AutoML in model searching are getting better and are becoming available to mainstream developers. While there are other great use cases for AutoML, model selection remains the one driving the biggest benefits in the context of real world machine learning applications.