CTO Notes: Granularity of microservices in the context of project management

Microservices are a popular topic in software architecture, especially for SaaS, chiefly because of their potential for scalability. But not every project needs that kind of scalability, and not every organisation / team is ready for microservices even if high scalability is a requirement. Like with everything in life, microservices have certain trade-offs which might even kill a project if not handled properly.

The Holy Grail of scalability is to achieve a "shared nothing" state, where no two modules are tightly coupled - or ideally, not coupled at all. The theoretical condition for perfect scalability is if every operation can be done completely independently from every other operation.

With web apps, we have pretty much completely separated the user interface from business code, and made web browsers to be the ultimate UI platforms, working on their own, and providing UI to the data prepared (and stored) by the servers. This naturally leads to structuring the backend side of projects in terms of a collection of orthogonal REST-like APIs, to be called by the frontend (and possibly 3rd party services) as they see fit.

It doesn't have to be that way - there are multiple frameworks on the web which basically control the UI and respond to user-generated events from the server side. This approach is usually called Server-Driven UI, and it too can be done so the UI part is somewhat separated from the rest. But still, there needs to be a scalable backend infrastructure in the first place to feed the generated HTML to the frontend, and this again is an API.

I see the design of microservice apps mostly from two angles:

• making clean, orthogonal APIs
• creating an organisational structure which can maintain those APIs

As always, there are many ways to achieve those goals, and early decisions will have a huge impact on the future development of the product.

The enlightened monolith

For now, let's pretend we didn't hear about microservices. The classic way to implement scalable web apps would be to use a web stack that includes a "proper" database like PostgreSQL, to make the database (possibly in a combination with an in-memory cache server) a single source of truth, and to make each server request - usually an API call - as idempotent as possible.

The backend app will have a single code base in this case, probably all in the same source code repo, but that's not a downside yet. With a clean API design and avoidance of state management within instances, it should be possible to create dozens of instances of the same backend, and with the help of a load balancer, achieve very high levels of scalability. There is nothing wrong with this approach. It will usually work perfectly fine to serve hundreds of thousands of users.

Some positive sides to the monolith approach are:

• It's simple(r) compared to the alternatives. There are a lot of senior developers capable of architecting and creating such a project from scratch.
• The shared codebase can make many operations more efficient in the short term.
• At this scale, complex SQL queries with multiple joins and grouping are usually par for the course.

But an app which grows to serve hundreds of thousands of users usually has some growing concerns:

• The development team will usually grow, and working on the same code base might introduce unnecessary friction
• Redeployments (updates) of the monolith might mean downtime for the whole app
• Upgrades of APIs might be tricky since the whole monolith will eventually contain a lot of legacy code
• The monolith might take too much resources to be instantiated over a certain limit, because it needs to support all the functionalities of the whole app from a single process.

I call this pattern the "enlightened monolith" if it serves a clean API, rather than ad-hoc endpoints needed for a tightly-coupled front-end. This approach is not exactly a 100% fit to wear the name "microservice" but it is probably good enough for 99% of web apps out there.

If the web app server get clogged, just spawn a new one (horizontal scaling). If the database is getting clogged, migrate to a bigger db server (vertical scaling) - this will take modern app a long way in terms of scalability.

Classic microservices

A big step above the monolith is the classic microservices architecture, whose major attribute is that the app functionalities are split into separate processes by some criteria, and the database is designed to take a much higher load. Each microservice is its own process (or a group of undifferentiated processes), serving a set of endpoints.

The criteria by which the functionalities are split into microservices are a major architectural and organisational decision.

One aspect of the criteria is performance oriented, and beings by asking the question: what will be the hotspots of the app? Or, in more detail, to determine which endpoints can be clustered together based on their usage pattern. Presumably, endpoints which serve user profile data might not be as frequently hit as endpoints which serve product information. And endpoints which perform sales operations must be hardened against failure (and possibly against security issues) much more than the others. Often, this criteria can be matched to specific groups of functionalities an app implements. For example, there might be multiple endpoints serving product data, and most of them have a similar usage pattern.

The other important aspect is organisational: which teams, or parts of the teams, will be managing each group of endpoints? While the company or the product are young, it might not be important. Maybe there are some services which can be delegated to junior developers, but some must be handled by more experienced ones, and that's it. But as the company grows, this will begin to change, and more specialised teams will emerge out of necessity. Maybe the product team does not have to know the details of the user profiles.

This allows grouping the implementation of endpoints into a single service process - or a container, or a pod. At this point, it is also beneficial if they are based on the same URL prefix.

In Kubernetes jargon, a "service" is the name for a group of pods with containers created from the same image. In this case, the pods are usually undifferentiated and interchangeable. The number of running pods can depend on load balancing. There might be dozens of product pods, but only a few pods of the user profile service.

A common variant of this architecture is to split it into gateway services (aka business services, external services) and internal services (aka API services). This approach tends to build a library of internal APIs which can be meshed and combined into more complex APIs served by the gateways.

The benefits of the classic microservice approach in either case are:

• Easy to scale horizontally (at least the services themselves - the database needs a special treatment)
• Can be made less sensitive to bugs and code problems - deployment of new code can be staged so that the number of services running new code is gradually increased, rather than switching them over all at once; if there are problems, fewer users will be affected before the rollback
• If an entire app service goes down for whatever reason, only one app feature could be disabled, while the rest of them will keep running (especially effective with a feature database)

The downsides:

• This is a complex setup which needs a dedicated devops team to support it
• Costs more than the monolith approach to develop and maintain
• It's not geared for single request efficiency, but for scalability, especially in the two-tiered approach. It will truly shine only with a large number of requests.

In short: the classic microservice approach scales very well for millions of active users, at the expense of being more complicated to implement than the monolithic approach, and needing a large volume of users to actually show its benefits.

Lambda functions

Finally, lambda functions in the context of code deployment can be seen as the ultimate expression of the microservice architecture. Here, the functionality of the app is not even grouped into processes serving similar endpoints, but there is complete deconstruction of the app into individual, small functions, serving a single end-point. Or not even that - the lambda functions might not even know what transport protocol is used - HTTP, or gRPC, or protobuf, or anything else; they might just look like ordinary functions, and the environment which calls them takes care of the transport.

The diagram is pretty much the same as for the microservice case, only much more granular. Every single endpoint will have its own box, and load balancing will happen at the granularity of a single endpoint. This approach is often labelled "serverless" as we are no longer even concerned where and how the containers with our code are running.

The benefits of this approach are:

• The operating environment (e.g. AWS Lambda, Google Functions as a Service) takes care of scalability.
• App features, migrations, and resiliency is granular to the level of individual functions

But the downsides can be severe:

• This approach leaves the app at the mercy of the service provider
• It can be more costly than the other approaches, depending on the exact use

I'd say than this approach is still reserved for special casees - not many projects can benefit from it.

What about the database?

All of the microservice approaches described previously have excellent horizontal scalability, and that also includes resilience to common problems like device, network and data centre failures. Just create some instances somewhere else.

However, there are two general bottlenecks still present: the ingress load balancing and the database. All cloud services offer reasonably good solutions for ingress load balancing, but database scalability is kind of an ongoing issue.

The ideal solution, and the one I have been happy to employ in a couple of times, is to have completely independent databases for each service. With a two-tiered microservice architecture, the internal services can call each other if they need "foreign" data. In this way, we are introducing lightweight horizontal scalability for databases, and also having the opportunity to configure each database differently, depending on scalability requirements.

Generally, data storage (which goes beyond databases and includes caches, long term storage and backups) is a separate topic which deserves a separate article.