Real World Machine Learning Pipeline (ML Engineering)

Few months back when I had posted a note on, what my network would like me to talk about (based on question I get asked frequently), little did I know I will end up covering, on "How real world enterprise machine learning" pipeline looks like as a series of articles, posts and video. Below was the presentation that I had shared on this subject to start with

Long story short the presentation was to highlight disconnect between academics/popular courses and enterprise in machine learning world

For most institutes, Data Science is all about ML algorithms and some Data Analysis. Data Engineering is lightly touched upon in handful of colleges

But for enterprise to implement machine learning solution most of the time and money is spent on Data Collection, Data Cleaning, Data Engineering, Model Deployment, Model Monitoring, Dev Ops, Stakeholder communication. ML algorithm is small fraction of entire lifecycle

60% of machine learning work is actually getting data ready (Data Collection, Data Analysis, Feature Engineering from domain understanding) for ML algorithm to work and 25% of time time goes in building frameworks for Model Deployment, Model Monitoring among others. Hardly 15% of time is spent in writing ML code that includes feature selection, hyper parameter tuning, model selection etc.

Interesting fact is to preform the last 15%, we also have AutoML framework to help us in some or most part of it

Top 5 challenges for enterprise as well confirm's that ML code (15% of work that academics focus on) even though very important has never been big of a challenge (Thanks to academics for cover it). Major ML implementation challenges

• Data Collection
• Deploying and Reproducing the model in production
• Model Monitoring
• Keeping model relevant by adopting to changing business scenarios
• Communicate and interpret model output to various stakeholders

While there are lot of content out there this article is to consolidate my experience working on Machine Learning and Data Engineering projects for large enterprise over the years and help students, researcher and new comers in this space understand how real world ML pipeline looks like in typical enterprise

This information is something I have been publishing for the last few months and will continue doing it for another couple of months covering in detail individual components. Below is how a typical pipeline in enterprise executing machine learning and artificial intelligence projects

Checkout my video covering end to end pipeline mentioned above

In additon, I have been sharing individual components of engineering ML solution through series of posts talking about individual component of ML lifecycle. You can find all of my previous and any future post on LinkedIn under hashtag #end2endds (LinkedIn Search with hashtag will get you all related articles on this subject)

You can also follow my YouTube channel (AIEngineering) where I have been creating videos on End to End Machine Learning lifecycle.

Currently Data Collection and Data Analysis video is already out there and will be publishing remaining videos in days to come

To subscribe to my channel you can use link above or click on link - AIEngineering