Database Considered Harmful?

Think "Events" (not CRUD)

As you dip your toes into the world of microservices, you start thinking this is great stuff, it will simplify development, help with continuous delivery, and ..., and ..., and ... You will get dizzy with the claimed wonders of microservices. The microservices train is rolling fast.

As soon as you try to build some non-trivial microservices, or just sketch your new microservices architecture, you find that you are entering into, hate to say it, yet another "paradigm shift". And please forgive me for having to use that phrase. Shared databases are NOT a good thing anymore - not from the point of view of a microservice architecture. 

This quickly sends you into thinking about who owns which data, how much duplication can we allow, how to partition, how to distribute databases, replication, read-only databases, and other thorny distributed data and distributed systems architecture issues. Then you step back and say, "hey, this stuff was supposed to make things easier! how did we get here?".

We've gotten very comfortable (and very good) with solving almost all IT problems by designing a database. The largest we can. Put all the data we have in it. Then any application is reduced to read some data, do something useful to the business from it, and possibly make some change on it. UIs are a front to the database. We based system architectures essentially on the enterprise data model or subsets of it.

Problem is: in the brave new world of small independent services, the role of the large database markedly diminishes. We will lose our "God View" of "The System"

What to do? 

Think, at least conceptually, "no database". We don't have CRUD anymore. Data is immutable. We don't store data values, we store events, and maintain event logs. When we need the current data value, we can derive it from some snapshot, and a set of events since the snapshot. 

Well, it is not that simple. There is a lot of details, and please forgive me again for resorting again to the phrase "paradigm shift". The paradigm to compete with database is event sourcing. I say compete, not replace. No technology ever replaces another technology, we still use horses, COBOL and even FORTRAN are still in wide use. Event sourcing is a simpler paradigm, conceptually, than "database". I don't have the space here to even just define it. Many people have done a marvelous job of that. See this, this, and this. Better yet, ask Uncle Google.

<sidenote> On "paradigm shifting": Think back to the days of transition from "Data Processing" to "Information Technology" - if you're old enough to have experienced it. From sequential "master files" to databases. Developers, called "Programmers" back then, were very fearful of "database". They had just started adjusting to moving away from tape, some even had master data on punched cards. A lot of COBOL had to be re-written to accommodate the revolutionary concept of "direct access" to an individual record! Couldn't do that on a master file on tape. IT professionals adjust all the time. Most IT professionals probably had seen dozens of "paradigm shifts". At one point in my career, I had resolved that the next person that says the phrase "paradigm shift" in front of me, I was going to kill on the spot. I must have "killed", in my mind, many writers, marketers, and "evangelists". But, in the end, I had to drink the cool-aid. Does anyone remember the days when there were no "web app"s</sidenote>

At this point, I just wanted to throw out the notion that we have to have a database. We may or may not - not mandatory anymore. Perhaps for historical or analytical purposes, but not to run the business. We need services and event logs. We can use database technology to deliver event storage in a highly performant manner. But conceptually, the notion of needing a database to develop applications is dead