why your tech architecture needs to adapt to your team team

Many many years back, I was taking an interview for a principal engineer position. As is the norm in all my senior engineering interviews, I asked the candidate to explain the architecture of their system and explain to me the design choices they took. As is common in all systems that evolve over time, there were weaknesses in the architecture - usage of a synchronous API where a queue would be more suited, using Solr for key-value style queries and so on. We discussed the gaps for a few minutes and both us happily passed the judgement - there were problems in the architecture and this should be done differently the next time around. We were both totally utterly wrong !!

So, where did we go wrong in our dissection of the system? We completely missed the context of the team that had built this architecture and made it work for "them". They were able to scale their business on this architecture with a small relatively inexperienced team before they had the funds to hire folks like the principal engineer that I was interviewing.

In order to explain this better, I first need to digress and cover two important aspects.

1) There are many many software engineers in the world (24.3 million as per this study) and they are all different with widely varying skill sets. Some of them are employed but can't really code; have personally seen some of them in coding tests. However, we can exclude them from this discussion as they don't really contribute in building large systems from scratch. However, even among the engineers who can code with a reasonable level of proficiency, you will definitely find two distinct varieties

• The deep-in-one-skill engineer - These are the engineers who have worked on one language, one database and one architecture all their life. Many of them are motivated and hard working; but the friction to learn something completely new is too much for them to overcome. They are relatively easy to find, recruit and retain.
• The jack-of-all-trades engineer - These are the engineers who love to learn about new stuff and try it out; they can talk about REST and GraphQL and gRPC in their sleep. They can tell you why Kafka scales so well even though it is disk-based; they are also the engineers who quickly get bored of building one API after another. They are harder to find and even harder to retain.

2) There are a large variety of businesses in the world and they are solving an even larger variety of business problems. Most of these problems use tech as an enabler; and some of them need tech as a differentiator. These are two very distinct scenarios and every engineering leader should be crystal clear on which of these scenarios he/she is dealing with.

Let us get back to the interview question - why was the so-called "best" architecture for the problem not the best for the team building it? There were two reasons - (a) it was a small inexperienced team that was comfortable with synchronous APIs but was not comfortable with event-based programming and (b) the system was an enabler and not a differentiator for the business.

The bottomline is that the architecture was relevant for the team and the problem in hand; and both me and principal engineer ignored the team context completely.

In the recent past, I had the fortune to work across problems and companies and now I always ask the two questions - what kind of an engineering team do you have? is the platform an enabler or a differentiator? Of these, the first question is the more important one. Even if tech is a differentiator but you don't have the brand and the means to build a rockstar team, adapt your architecture accordingly. Let me clarify here - it does not mean that you build a bad architecture; one that will not scale or be brittle. Instead, you keep the limitations of your team in mind, while defining the architecture.

Let me share some examples here.

An integration platform is a common layer in many large systems. We had built a "well-architected" one at Makemytrip and one of the days, a team from Oracle visited us to pitch their integration platform. While building the platform, we had encountered various performance issues; especially related to parsing large XMLs (yes, those were the days when most vendors we worked with used XML) and had painstakingly solved them at the cost of adding some complexity. So, I asked the Oracle team about their latency numbers for large XML payloads and had a smug smile when they shared their number. They were at least 2x slower than what we used to do; and these were numbers coming from the sales team. However, our platform did not support random XML traversals to achieve this performance, which made our platform slightly complex to configure and use. Even in hindsight, this was the right call; we had the engineers who could handle this complexity and any latency in this path directly carried over to end users. So, this tech differentiation had both business justification and a tech team that could handle the complexity.

The second example I have turned out to be a mistake; so will keep this one anonymous. We were building a service that returns a list of ranked "objects" with various filter options- we needed the response to be fast and the ordering to meet some property. The ordering was a bit complex involving multiple parameters and we needed to support filtering with paginations. We devised a "clever" solution with a complex algorithm that worked - really fast and accurate; but a solution that only 2 and a half people understood - me, because I had conceptualised it, the senior developer who implemented it and the QA who understood some of it. This, in hindsight, shaved off a second from the response time but very limited material impact and made the system complex. Once I and the senior developer left, each and every extensions to this feature led to bugs as frankly, most engineers are not very good at algorithms.

These experiences have made me more circumspect of recommending the "best" architecture. I am not a big fan of low code systems; however, I still recommend them for "young" teams. I strongly recommend platforms like AWS Glue Studio for teams that lack good data engineers, and auto-generated HTML tools for teams lacking good frontend engineers.

The key here is to understand that a tech architecture does not operate by itself; it needs to be managed and enhanced by a tech team and this adaptation is crucial.

Many years back, I was working on hand-coded compilers for key high-performance computing libraries and even though hand-written compilers outperformed off-the-shelf compilers, they are only suited for super computers; everyone else should still use off-the-shelf compilers; because not every one has the skill, nor the need to hack a compiler.