Fail Fast AI, or the advent of AutoML
It is common knowledge that the path of a data scientists on a project that requires the use of artificial intelligence, is quite long... Let’s look at it step by step:???
Step 1: Think big but start small!??
This starts with the analysis of the need and, most of the time, its "refinement", because the needs expressed are very often too general, too ambitious or too vast. It is therefore necessary to break down this need into smaller "functional units", sometimes reducing the scope of the requirements, and always evaluating the threshold of "opportunity", above which the project begins to make sense and bring a return on investment.
Step 2: The data??
With this stage behind them, the data scientist will tackle the data. And this is often where the disappointments begin ... problems of volume, diversity, representativeness, not enough maturity in the industrialisation of the necessary data, etc. Moreover, it is often necessary to go and find 'new' data outside the initial scope of the use case, sometimes outside the IS, to avoid falling into the trap of predicting things that any expert in the field has known for years. ?
But let's be optimistic, the data is there, and it seems usable within the scope of the need – the data scientist is fairly convinced of this after having collected, evaluated and explored it. ?
Step 3: Prepare the data??
Now we have to think about preparing the data before using it in the learning cycle. Missing data, harmonisation, standardisation, encoding, creation of new features, labelling, data augmentation... these are all part of this often rather long and sometimes tedious step.?
Step 4: And AI then? ?
Finally, the work related to AI as such begins, i.e. the difficult choice of algorithms to support the chosen strategy. ?Is it better to use a deep learning model? But which one, or which structure should be recreated? Perhaps a LightGBM will do, as usual? Maybe another tree implementation? And why not transform the data set into an image to allow convolution? ?
These decisions will be based solely on the knowledge and experience of the person (or even better, the team) in charge of the project. Will these choices be the right ones? Perhaps not, unfortunately, but can we blame a data scientist for not mastering the myriad of possible algorithms that exist today and that can be implemented for a given strategy? But in this case, if the models used fail, how can we be sure that there is no other solution, unknown to the person in charge of the project, which could give the expected results? ?
However, let's move on. We have the data ready, we have an approach strategy, we’ve selected one or two algorithms. Now it’s time for the learning phase. ?
Separation of datasets, choice of hyperparameters, learning strategy, choice of evaluation metrics, evaluation itself... The results are not as good as expected? Well then... 3 or 4 more epochs even if it means overfitting a bit? Am I already going to adjust the learning rate ? What if I changed such and such parameters? Or try this other algorithm that I didn't necessarily think was the best suited to this case? ... In short, the data scientist starts again and again, iterates, explores, navigates ... Not everyone understands what they’re saying, but they seems to know where they’re going, because they’re making presentations with lots of metrics and pretty graphs and always have new avenues to check, try out, explore. And they keep asking for data, data and more data to do other, different tests, and because “with more data we should be able to improve the performance of the model”.?
But when does it all stop? To caricaturise: when the budget is spent (or to put it more positively: when the expected results are achieved) – generating frustration and disappointment at business level in the process.
Can we blame the data scientist for not knowing, from the outset, what it will take to achieve 96.8 accuracy with good recall? No, obviously, we can't blame them, because we are looking at the main issue of any project involving AI ?– uncertainty. This uncertainty is omnipresent from beginning to end, even when we eventually get to a result on the learning perimeter, when we generalise the models (go live). And it's not over yet, because these models must be monitored over time, as they do not react well to subtle changes in their environments... On the other hand, what we can reproach them for, is not knowing (or wanting) to stop before it gets this far!?
But ?where does AutoML come into the picture? ?
Well, the promise of AutoML is to industrialise this stage which I would call “wandering” – just to annoy – by systematising the search for the “best” model. But how? Simply by means of a rather brutal approach which, to oversimplify a bit, consists ?of trying all the models with all the possible parameters on a prepared dataset, and at the end of the training period to present the list of tested models sorted according to their performance.?
An AutoML system can therefore be summarised as:???
The data scientist then provides the prepared input data, selects the family of algorithms to be tested (preferably a detailed list), runs the AutoML and... waits for the results. But what they get is not the result of an experiment as they usually do, it is the result of dozens of experiments that would have taken weeks to implement and test... Half a day of AutoML for 3 weeks of work? That's probably a good ratio, at present.?
So, can we get by without data scientists by using AutoML? ?
Definitely not (at least to date ??), because AutoML does not do everything!??
Not all cases are currently addressable (multi-target regression, 3D segmentation...). But more importantly, all the upstream work at the data level still needs to be done “manually”. Moreover, AutoML only gives results the first time (i.e. with a model that can directly be put into production) in relatively simple cases. On the other hand, it is clearly a formidable tool for accelerating the first iterations on a project, giving an accurate idea very fast as to the project’s chances of potential success. It allows you to test several use cases in no time, and to focus on those that will have the best ROI/success rate (the “fail fast AI” approach).?
But beware, because platforms are evolving faster and faster! They already have a great deal of built-in data characterising the hundreds of thousands of training sessions that are performed, and are therefore actively working on improving stopping policies or even the choice of algorithms to be proposed at the start. We are also seeing platforms offering new services to automate even more stages, such as auto-featuring before learning. These platforms are also particularly active in integrating new models, making it all the more complicated to keep data scientists' skills up to date.?
The democratisation of AutoML platforms seems to be a given and it is clear that the massive use of these platforms will transform the job of the data scientist, probably by splitting it into two categories:?
Written by Jean-Paul Muller, AI Practice Manager Inetum.