Recommendation Engines Demystified

In my childhood, I lived in a small village and we used to function as small communities. I remember, those days there was a strong trust bondage between the consumer and shopkeeper and the product recommendations was based on the shopkeeper’s knowledge on the previous purchases. Those days because of these personal kinships, consumers used to get great customer service, while vendors were able to reap all the benefits as they understood their consumers budget, needs and preferences.

Fast forward to today, I see a different scene because of our modern-day globalisation of services. Naturally I see lot of advantages in terms of efficiency, availability and productivity, but we lost that intimacy of personal consumer vendor relationships. However, this lack of personalised service doesn’t change the basics of business, the basic fact that, the key to successful business is understanding a person’s need. And nowadays we learn a person’s need with an algorithm. A Recommendation Engine…

What is a Recommendation Engine?

In simple words, recommendation engine is a machine learning algorithm that utilises big data to suggest additional products to consumers based on a variety of reasons. These recommendations can be based on items such as past purchases, demographic info, or their search history.

How it works?

Before we speak about the machine learning algorithms, lets talk about data. What are the data sources needed to build a recommendation engine? So, whenever a cookie hits a website, they do record its location, device, channel and the actions like navigation to certain pages, items added to the cart, etc. They also collect details about the purchase history. These are the implicit indicators of what the consumer preferences are and these recording becomes the input for a recommendation engine.

There are three popular method to build a recommendation engine.

1.      Popularity based

2.      Collaborative filtering

· Nearest neighbor

· Matrix factorization

3.      Content based

Popularity Based

This is the easiest way to build a recommendation engine. This recommendation is for the consumers who came into our website for the first time. They have neither purchased with us nor we have any clue about these consumers.  The algorithm will have a standard matrix developed for these consumers and a standard set of products gets displayed to these consumers.

Collaborative Filtering

Collaborative filtering is one of the commonly used algorithms. In general, what it does is, it takes a consumer transaction and try to find a similar consumer. Based on the learning from the similarity, it recommends a product to them. Collaborative filtering can be done in two ways,

1. Nearest Neighbor,

In this approach, the recommendations are made from the nearest neighbor data in other words similar consumer’s data. For our understanding, we can imagine it as a giant matrix where your X axis(rows) are basically the consumers and the Y axis(columns) are the items consumers purchased or interacted. We can find similar consumers by making all these interaction into vectors.  For each consumer there is a vector representation of the items that they viewed. For each item there is a vector representation of the consumers. We can do it either way. If we do it at a consumer level it is called user based collaborative filtering and if we do it at an items level, it is called item based collaborative filtering. Then we can use the cosine similarity of these vectors to find the similar consumers.

2. Matrix Factorization

This is an improvised version of nearest neighbour method. In the nearest neighbor approach, obviously the matrix will be sparse since not everyone is going to buy every other product. In Matrix factorization we use machine learning model to predict consumer ratings for unrated items or unbought items.

Content based

Imagine your vendor have released a new product and got no purchase history yet. How does this product gets recommended to a consumer? That’s where the importance of content-based filtering. Content based filtering is depends on text embedding and Image embedding.Content based filtering uses natural language processing to find similar products.

Thank you for reading.