My pet project architectural design

Large picture: https://vladimirkhil.com/content/docs/my-architecture.png

It takes 22 years to build this architecture for my pet project. Unfortunately, I can only work on it during free time - that's why it took so long to create this. But most time has been consumed by the creation of clients applications; complex server part exists for the approximately last 5-6 years.

So, let me explain my basic design principles at first. They could be applied to both products and processes.

1. Architecture is a dynamic thing. You cannot create fixed architecture and live with it forever. So, be ready to continuously evolve.
2. Start with something simple, like monolith. Eventually you'll see where to improve and when there will be a need to extract a separate service.
3. Do not apply anything thoughtlessly. Do not follow cargo cults. But do not hesitate to thy the things and keep only those that work for you.

So, originally I had only two services - my personal site (with database) and SIGame server which hosted all the games. And all clients communicated only with them.

Eventually I discovered that some parts of SIGame server never communicate with each other. And they work with unconnected parts of the single database. That was a sign that these parts can be extracted to separate services.

Why did I need multiple services if I was working alone? That allowed me to concentrate only on some small domain areas while upgrading a single service. The whole project is very complex now so it is not easy to remember everything. And multi service approach also reduced blast radius if some update goes wrong.

Also I was working on transitioning everything to Kubernetes and deploying all the system parts through Helm. That dramatically simplified the deployment process and supported easy rollbacks. In case of the global failure it will require reasonably small mount of time to recreate the running product from scratch.

So, what services besides personal site and game server have been created?

SIContent service: it is responsible for storing packages (files with questions text and media files) temporally. Clients upload game packages (which are ZIP archives with defined structure). Service extracts them to a folder. After that all content files are available for players during the game via Nginx. The same package is shared between players in different rooms. After some time, if nobody is using the package, it is deleted.

Potentially this service will be able upload packages to S3 storage to speed up media access.

Because this service is heavily loaded (in terms of network ant not CPU/memory), I had to install a second server with another SIContent service instance. That eliminated media load lags at peak hours.

SIStatistics service collects data about packages and games been played. I hope that its data will allow to increase the quality of answer validation system (currently automatic validation is based on Levenstein distance and could not always recognize right answer given by the players). It is also used to display current top packages and latest finished games in client applications.

So, this service has a huge and constantly growing database. And interesting permissions: it allows unauthorized writes but does not allow unauthorized reads.

SIStorage service is responsible for holding predefines packages library. New users usually start with playing on standard packages and create custom packages after a while. This service has database for storing metadata and file storage for storing package files. When you create a new game with standard package, SIContent service directly loads package file from this service without the need of any client upload.

The idea behind this service is the support of public storage protocol. It allows to connect more public package storages to game without rebuilding clients or server in the future (currently one external storage is already half-connected to the game ecosystem).

AppRegistry service works as the source of clients versions, their installers locations and provides information about applications releases. Clients auto update feature discovers new versions through this service and downloads the updates. This service also collects all the clients exceptions information (sent on users behalf) and allows service administrator (me) to mark errors as fixed.

MyBackend service: as currently my personal site is a ASP.NET MVC application, I plan to split it into pure React based client and pure Web API MyBackend. This is the place where the work is not finished yet. I will move all necessary server data from "Personal site" service here. And then implement the React application.

This service holds my personal content like apps links, articles, photos, news, About page etc.

Spard service is the running interpreter for my own SPARD programming language. As users could provide their own sources, code execution can go wild and eat a lot of resources. That's why it is crucial here to keep all executions in a separate service with well-defines K8s resource limits. Despite of all, this service is lightweight and could be easily restarted.

Nginx (CDN) is simple: it holds all static content unrelated to a specific service. For example, some documents and app installers.

SIHost service is a potential one. I plan to extract hosting games capability and all the state from SIGame service so the latter will become stateless. It will hold game discovery mechanism pointing to a correct SIHost instance hosting required game. That will simplify game server upgrade process because currently during upgrades I need to switch new users to a new instance of SIGame server, wait until all games on a old one finish and turn off the old one only after that. And there is a issue when several players who wanted to play together and arrive during the switch time appear to be connected to separate game servers and could not play together.

After SIHost implementation they will be always routed to a correct SIHost instance having game they want to connect to.

Identity service will keep user profiles. I do not plan to perform my own authentication anymore - I am going to use an external one. Yet authenticated users need to store their data (profile pictures) somewhere. But not a personal information, of course (oh, God!).

There are no queues as there is no need for them. There is also no Distributed Tracing as there is no too much conversation between different services - they mostly communicate with client applications.

So, the project is still evolving. But for the time being I am a little bit tired of all this server stuff and plan to have a break. It is working, it holds up to thousands of users at peak hours and can work for months without my intervention. Zero error tolerance made it really super stable, but that is a completely different story :)