Mock Interview - Notification service for a bookstore

Today I did a mock interview with a senior engineer to help them prepare for the upcoming system design interview, I will share some notes here and I believe it will be useful for anyone who is on a similar journey.

1. Design scope.

Requirements: design a notification service for a book store based on user searches. For example, user searches for "system design", if the next day a relevant book is added to the SKU, an email notification will be sent to the user with the details. The system should support 20 million users with 4 million daily active users, and assume each user will have 2 searches on average, information (meta data) about each book is fairly small, can assume 10KB for each. Non-function requirements are typically around scalability, latency and fault tolerance.

Note: In this process, the candidate should actively seek as much information as possible to identify the scope of the task, sometimes the topic is quite broad, you should narrow down the key functions, and sometime the topic is quite vague, you need to figure out the exact requirements. Keep the conversation going and write down the agreement between you and the interviewer, make assumptions clear.

2. High level solution.

Note: In this part, make sure you get the buy-in from the interviewers you are going to the right direction and covered all the core functions. Here you should focus on the functionalities and features, presumably for a small customer base or on a single server. This is also a good opportunity to clarify any ambiguities from the first step, and also correct any misunderstandings, as the interviewer saw your design they will have a better understanding of where you are going and if it will fulfil the requirements. No harm to confirm with them directly about the functions.

3. Deep dive.

Note:

Load balancer: this is recommended as the number of daily active users are quite high, we should route the traffic evenly across AZs or regions.

API Gateway: typical scenarios of different API endpoints, like search, browse, payment, etc. All functions of an API gateway can be discussed, like throttling, tracing, SSL termination, and also the reasoning.

IAM: for access control.

Search indexes: as the volume is quite large, 500k new items added each day, it will definitely improve the search efficiency.

SKU DB sharding and replications: read heavy with batch writing, consistency hashing by category, which means potentially rebalance the hash ring everyday. Might need further discussions and investigations, if anyone has ideas please leave in the comment.

Message queues for client queries: it will go to 2 different services, including the one we are designing - for notification purposes, we need a record of user search history, and only send relevant recommendations to the topic and the date when the search is conducted. The search items together with user information will be stored in a DB.

Notification services: this will run every 24 hours according to the requirements, it can load the details from config files, including frequency, time, and throttling settings, as we don't want to overload the servers with such requests.

Message queues for notification workers: the system is designed with future proof - we are only sending emails now but there might be other channels to be added, like push notification and SMS.

Notification Workers: it can be scaled down and up based on workload, gets information from queue and also static info like template and images from storage before calling 3rd party tools to send the notifications.

Third party tool selection: it is a mature market with tools to send out emails, it makes commercial and operation sense to choose the OTS tools rather than develop in house. The cost of developing, maintaining and upgrading such tools is high. They come with built in tracking and analytics.

Analytics: we can track how many notifications were sent out and what user actions are.

Logs and retry: it guarantees we deliver exactly once.

Monitoring: make sure the system health level is high and take actions when issues happen.

Remember, interviewers will challenge you on different parts and ask difficult questions, make sure you can be adaptive to your solutions, and take alternative options when given the hint your current design might not work in certain scenarios.

4. Summary & Further discussions.

Walk through the system from user perspective, what would happen at each component. The candidate added API with parameter, and data structure for the key component, which is a bonus point. I generally asked what would be the potential bottleneck and solutions, what would happen in case of failures, and I am quite happy with the answers in this case. So be prepared to have this discussions through the interview and especially towards the end.

Also, you can add further development considerations, like security and more details in tracing and storage designs.