Technical complexity is not the answer (most times)

Disclaimer: Opinions are my own, they do not necessarily reflect those of the company I work for.

Disclaimer #2: I initially wrote this text on May 19th 2020 (at 11:34 pm), but never publicly published it for some reason. In retrospect two years later, I feel my thoughts and this text is as relevant and actual as ever, hence, after minor tweaks, I finally decided to push the publish button.

TL;DR: Reduce technical complexity. Focus on business complexity. Start simple, learn and then redesign. Remember Conway's law. Be ready to fight a hard fight.

?

Why, oh why?

Why do people make things complex? I read a good article the other day about it after Vaughn Vernon tweeted about it (https://fs.blog/2018/01/complexity-bias/). And that article made me think about it in the Software Engineering world. The article focus on many aspects of why humans are biased towards complexity and at one point quotes Edsger W. Dijkstra:

?“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.”

I'm not going to focus on the obvious pressures of the many vendors out there that want to promote their latest framework, tool, software as the best thing ever that will solve all our problems. I want to focus on the Software Engineering professional (Developer, Architect, Manager) and what we should and should not do to provide the best service we can to the companies we work for.

Many reasons can lead Software Engineering professionals to opt for a complex solution. In my experience, the great majority of times this is due to technical decisions being the very first consideration in a project, and afterwards attempting fit the solution into the technical constraint. Or simply put, use technical complexity as the solution. Also, many people will choose whatever they are comfortable working with instead of the right tool (a coding language is a tool) for the job. It always comes to mind the sentence "I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail." -?Abraham Maslow.

Sometimes the argument is that the technology provides agility, however, is well documented that it not only does not, but it also might even create more silos (e.g. Designing Autonomous Teams and Services - Deliver Continuous Business Value through Organizational Alignment?- Scott Millet and Nick Tune - chapter 1 p6-7)

The following topics have a lot of technical stuff, but also some more organizational/team/process topics. Anyway, based on many books read, interaction with some brilliant minds in the industry, and using my brainpower and experience in Software Engineering, I came up with a list of what I observed as recurrent things people usually do that causes problems and what they don't and, if they would, it would be greatly beneficial for the project.

?

What most people do (and they shouldn't)

• Define platforms and or technologies before even knowing how to solve the problem - or what the problem is! As Uncle Bob says "The Database is a detail - Chp.30", "The Web is a detail - Chp.31" and "Frameworks are details - Chp.32" (Clean Architecture, A Craftsman's Guide to Software Structure and Design – Robert C. Martin)
• Micro-services and event-driven architectures as a silver bullet. Very useful in the right context, but they add complexity, require a fully automated CI/CD pipeline and a robust monitoring system. And still many people in the industry will want to say they have a micro-service architecture. Even giants like Uber are reverting their approach to have more like a well-sized-service architecture (https://kalele.io/microservices-and-microservices/). They still have plenty, but the scale they operate at does require it.
• Event-driven architecture that exposes internal domain models and publishes everything on an external bus and call it decoupled - just because you're sending messages on a bus does not make your services decoupled. Are your messages versioned? Do you have idempotent receivers? Are the receivers tolerant to different message versions? A bus and events in themselves do not solve integration problems… oh boy, here comes the distributed monolith.
• Define anemic models everywhere because, well, that's what is on MSDN pages… or in Javadocs, or whatever docs your language of choice provides. OOP is about encapsulating logic and exposing behavior, not about having logic scattered in multiple classes/files.
• Focus on curriculum-driven development, meaning, selecting a tool/solution because it looks good on your CV rather than because it's the right approach.
• Invest a lot of time learning the latest framework, instead of the fundamentals of Software Engineering. You're not doing OOP just because you're writing code in Java/C#.
• Create long inheritance structures and completely forget about composition and/or other patterns, especially if you think about the SOLID acronym. When creating inheritance, most times SOLID principles are not adhered to, and the tricky one is the Liskov substitution principle, which many so-called senior developers and architects can't describe.
• Use the right tool for the problem you are trying to solve. Many times you don't need CQRS, Event-sourcing, Mediators, etc. Sometimes a simple CRUD interface is ok - but be wary of going with the anemic model everywhere.
• Don’t apply the YAGNI mindset enough. How many times did you add code "just in case someone asks for this in the future"?

?

What most people don't do (and they should)

• Keep it simple - if it is simple, it's easier to understand how it works and it's easier to maintain. Another quote: Simplicity is prerequisite for reliability - Edsger W. Dijkstra
• Ask "why we're doing this". It helps bring focus to tasks at hand and to make decisions when facing several options. And then ask some more. And keep asking.
• Learn about the places where you should have "good enough" and the places where you need to invest your time. You should invest your best resources in what's core for your business.
• Expose behavior and focus on behavior- A while ago, Alberto Brandolini had an interesting tweet "Well... you cannot call your service ‘autonomous’ if you can blame other teams for failures or late delivery." (https://twitter.com/ziobrando/status/1254077492052320258). I replied asking why he thinks that happens, also making a few observations about the causes for that lack of service autonomy and his reply was genius: "It’s mostly because people think ‘data’ instead of behavior." Brilliant! He just summarized why many times it’s so hard to discuss things in Software Engineering and my struggles in many meetings over the last years. On top of that, if we think about OOP, that's exactly what OOP is about, encapsulating details and exposing behavior.
• DRY is about behavior, not code - it's ok to have a little bit of what might seem duplicated code if that allows similar behaviors to evolve in different ways.
• One model does not fit all - yes, information about someone who interacts with the system being built can be in classes called, Lead, Subscriber, Passenger, etc., instead of a class Customer that will try to fit all use cases partially but does not fit any appropriately.
• Write self-testing code - POCO/POJO objects with validators and similar are just getting the business logic scattered around multiple classes.?Your class should ensure it has everything it needs when you call its constructor. Yes, that's one of the things they should be used for. Fail fast if you can too in your methods.
• Test your code, especially the part that matters - Code coverage in itself does not warrant good tests are being written. Martin Fowler has an excellent way of thinking about good tests in his article about test coverage (https://martinfowler.com/bliki/TestCoverage.html). There he states that:

"I would say you are doing enough testing if the following is true: You rarely get bugs that escape into production, and You are rarely hesitant to change some code for fear it will cause production bugs."

• The tests are one of the best documentation there is if done right - On top of the previous point.
• If you think event-driven is needed - define first seams/boundaries, then define code modules and publish events inside your monolith/service between modules. Break into separate services if and when needed.
• DevOps is a must - automate everything. Yes, including DB changes. Whatever slows you down, automate. Remember the Theory of Constraints.
• Write a decision log/ADRs - no, 1000 Jira stories won't do the trick of answering "Why did we do it this way?". Neither will the comments on the commits in Git and definitely won't the comments in the code.

But Miguel, I, really, really know I'm going to need something complex, why can't I just start with a complex thing?

Naturally, if you're designing an application and you know for a fact that you have to be able to scale to millions of requests per hour (I'm guessing replacing a legacy app that is very hard to change and has that load already) it's tempting to immediately think of event-driven architecture and micro-services as a good option as they are very powerful tools. It's natural to try to skip intermediate learning steps and go to a final solution that will be complex because it has to cope with the requirements of running event-driven micro-services. However, there's an argument to be made even in that case about the benefits of starting with a simpler approach, learning about the problem, and then redesigning when the time is right. It's funny how these things keep happening, despite the effort of so many brilliant people like Martin Fowler (https://martinfowler.com/bliki/MonolithFirst.html):

"almost all the cases where I’ve heard of a system that was built as a microservice system from scratch, it has ended in serious trouble… you shouldn’t start a new project with microservices, even if you’re sure your application will be big enough to make it worthwhile"

or his take on what he called Design Stamina Hypothesis (https://martinfowler.com/bliki/DesignStaminaHypothesis.html).

?

So what should the Software Engineering community do?

Good question, I'm glad you asked. If we consider all the above topics, they all keep bringing us back to simple things like YAGNI, KISS, and SOLID (or CUPID), and all the so many times forgotten fundamentals practices of Software Engineering. We must also always strive to understand the problem we must solve first and then figure out the right tools to help solve it. We, the members of the Software Engineering community and professionals must make an effort to always advise the least complex option to start with, and then re-access frequently and adjust accordingly. A flexible software system allows quick and safe modification and extension, and those requirements can be achieved with simplicity and focus, not technical complexity.

?

Just my two cents.?

MSM