How do you design a scalable and robust recommender system architecture?

While designing a data pipeline, the first thing is to figure out whether the plan is to use real-time insights to generate recommendations or periodic recommendations (in batch mode) also solves the purpose. There is trade-off between choosing one over the other. Next step is to think about the source of the information, processing, and finally storing them. Additionally, one aspect to think here is about decoupling the incoming data with services consuming data. Here messaging systems like Kafka, Azure Event Hubs, can be particularly helpful -- decoupling producer with consumers. This is followed by thought on processing systems like Apache Spark, Flink, Samza, etc. Finally, for source, columnar DBs like Cassandra are good choice

Recommendation engines is trained on preference data which is collected and present in the matrix format. In preference matrix, user-preferences are collected for a given product. There are two types of preferences 1) Implicit 2)Explicit. Data pipelines are used to collect explicit and implicit preferences. 1) Explicit preferences is directly stated by the user. e.g. movies he likes or dislikes 2)Implicit preferences are inferred on the basis of user behavior. Examples of implicit preferences are a)Click-through rate b)product ratings, which are extracted from reviews or identified by sentiment analysis c) time spent on watching videos d) purchase history of a person etc.

Building a scalable and resilient recommender system architecture depends on an efficient data pipeline. Utilizing a microservices approach with Apache Kafka for real-time event streaming and Apache Spark for distributed data processing is vital. Decoupling data processing into smaller services enhances scalability and fault tolerance, facilitating parallel processing. Integrating batch and stream processing through lambda architecture enables both real-time and historical data analysis. Implementing robust monitoring and alerting systems like Prometheus and Grafana ensures proactive identification of performance bottlenecks, maintaining system reliability and scalability.

Strating simple with a 2X2 design pattern of RecSys is my personal recomendation. The idea is to do some of the model work offline and do the ranking and result online for the users. Training of embeding model and embed items is also done offline. The retrival is then done online using the trained embeded model using ANN index. For reference see- NVDIA's "Recomender systems,Not just recomender models" and Eugene Yan's brillant write up "System design for recomendations and search"

When we think about scalability and robustness of recommendation engine. We should consider the following points. 1) The preferences matrix which is used in recommendation engine is generally huge. The preferences matrix can contain the preferences of millions of users and products. 2) The preference matrix is very sparse. E.g. one user can only watch hundred of movies. 3) Apart from that we can face cold start issues where we need to recommend items for a new user. The matrix factorization algorithm is more robust and scalable, as compared to other algorithms. Apart from that, hybrid recommendation engine is also used where we can combine multiple recommendation techniques together.

For a data engineer, deploying and scaling a recommender system is pivotal. I advocate leveraging cloud services with microservices architecture for this purpose. Cloud platforms like AWS or Google Cloud offer scalability, flexibility, and cost-efficiency. Microservices enable modularization, simplifying deployment and scaling. Containerization technologies such as Docker ensure consistency and portability. Serverless architectures like AWS Lambda reduce operational overhead and enable automatic scaling based on demand. By employing these strategies, we ensure reliability and performance while minimizing costs, complexity, and risks associated with deployment and scaling.

Suggest one way to check for demographic parity. We can, evaluate the recommendations for different demographic groups to ensure that no particular group is consistently over- or under-represented. Thats one way. Other options are like to opt for fairness awareness models by design.