AI 101: Getting started with AI

There are countless courses, articles and posts about how to learn AI, I am documenting how I started learning about AI with the aim of building an AI app. Since my experience is around Commerce Platforms for B2B tech companies, the easiest use case was to build a "Product Recommendation" application. I will mention the steps and examples of how I built my first AI app on my AI learning journey.

Understand the #AI Big Picture

Starting with a general idea of #AI #ML #DL, these are just fancy ways of saying Algorithm, Technique or Statistical model. From now on, when we say "Model", we mean "a mathematical/statistical technique that predicts an outcome without specifically listing the conditions for each outcome".

When building an AI app, start with "What are we trying to solve"? "Do we have enough data to train a ML model", "if we run the data through the ML Model, are the results good enough to deploy it to a production system"? Lets break it down:

AI Application Building Blocks

Step 1: Define the Problem

Given that I have data on what customers purchased from an eCommerce store.

I want to recommend a Product to another customer

So that, it has a high likelihood that it will be purchased?

Step 2: Gather Data and run Analysis

For my product recommendation App, I gathered 104k entries of anonymized data consisting of Transaction IDs, items, users, date/time stamp, qty, product category, geography, user reviews. Ran data analysis to find missing entries, outliers (an indication of incorrect data) and other techniques to cleanup the dataset. I used Dataiku , you can use any tool including MS Excel, to make this analysis.

Step 3: Build ML (Machine Learning) Model

The Data Analysis helps us to figure out which data to include in the ML model for processing with the end result of getting a product recommendation as the output. Machine Learning is essentially a fancy way of using a Mathematical or Statistical model that will predict an outcome based on available data. I am using these 'existing' Models to see which model has more accurate predictions. This means for most applications you don't have to build a new model, you can use existing models and fine tune them for your application.

1. Item-Item collaborative filtering - Items closes related to each other are recommended
2. User-User collaborative filtering - Assumes similar users like similar items
3. Singular Value Decomposition - helps to simplify data, eliminate noise and perform better compared to other algorithms for predicting values)

Step 4: Evaluate Model

Using existing datasets, divide the data set into 80% Training and 20% Testing data. Run the Model and compare the accuracy of predictions from each model against what users eventually bought. From the three ML techniques, pick the one that performs best. in my case Item-Item Collaborative Filtering worked best and is less computationally intensive. This can be an issue when scaling on hyperscalers (AWS, GCP or Azure) as it increases cost.

Looking at the error numbers, Item-Item Collaborative Filtering performs better compared to other Models. Its prediction are closer to the mean (RMSE and MAE).

Step 5: Deploy Model

Once the ML Model is selected, deploy it into production, keep testing the model and improve it where needed by adding more data, more features OR or trying a mix of multiple ML techniques to improve prediction accuracy.

Now you build your first AI App

Using tools like AWS SageMaker, Dataiku, RapidMiner, KNIME and many others, you can start building your AI App. Data stores such as Kaggle or DriveData can be used for feeding your first AI/ML models and testing the results.

Sample AI Applications

Not an exhaustive list by any measure, but gives you an idea of what you can pick and work on to get started.

1. Question answering: chatbot that outputs an answer for a given question.
2. Text analysis: analyze texts from models specific to an industry domain such as finance
3. Predict if an item belongs to a category: an email spam filter
4. Predict a numeric/continuous value: estimate the value of a house
5. Detect abnormal behavior in application: spot when an IoT sensor is sending abnormal readings.For more AI Applications, where existing models can help you build an App see this page from AWS SageMaker: How to use Amazon SageMaker built-in Algorithms or Pre-trained Models - Amazon SageMaker

AI Hyperscaler Landscape