The Growth Spurt

Welcome back to the next edition of my humble newsletter. This month, we are looking at growth challenges—not from the business edge but from the technological perspective. We reflect back on some of the decisions made in the early release cycles and how we arrived at those decisions. Finally, we discuss some of the lessons learned now that we have progressed.

Back in 2021, when we released the system, the architecture we worked on was reasonably straightforward. When we started, we took an easier way into the build. We engaged a software house to deliver our vision and they built our first system architecture. It was well thought out and delivered its intended purpose, bringing this idea of an accessible, app-driven car-sharing service into the market. Were we the first? Definitely not! We were probably the 7th car-sharing service to launch into the market. The only difference I requested was the setup of 3 separate environments for us to manage later on: the Dev, UAT and Production environments. I was effectively a Project Manager and Technical Lead representing the organisation at this stage. As soon as we launched, we hit the ground flying. All the cars out on the road were out on the road. We took pride in seeing our GetGo-branded cars on the road every now and then.

ISSUES! Yes, we ran into them rather quickly. It was quickly apparent that the demand for freedom was beyond our expectations. We soon had hundreds of registrations daily, and location requests were rising quickly. From the system's standpoint, we saw increasing compute and load on our database. This led to multiple load problems, database locks, and sometimes slowness. This raises a few alarm bells.

First, this means growth will be a problem for us. A quick and dirty way to manage this is to increase our computing by providing a higher capacity for the container running the service and increasing the number of containers running a service. There are a few drawbacks to this as well! This method of increments isn’t limitless, and as we increase these limits, it also means we are increasing our costs almost exponentially. We can also replicate this at the database level, increasing the capacity of the database to meet the load demands. The drawbacks were also similar, as mentioned.

Second, the single service structure means that innovation will be slow and cumbersome. We quickly learned that pushing out changes, though simple from the technological perspective, was slower as every new change had to be planned, or there would be much unravelling. This was especially difficult when we had a sudden need for change.

Third, the code became super thick, quickly. As we went through a few minor releases, managing the changes was difficult as the codebase was becoming hard to manage and manoeuvre. It was tougher to debug our issues, and with an increasing user base, this was a sure red flag.

The first instance of this hitting me was when we had our first deadlock incident in our database. It was weird that we had this issue, as we barely had 1000 active concurrent users then. I’ve not seen it happen in such a small scale load from my past experience in the financial sector, where millions of transactions were happening at any time. However, the caveat was that the code was vendor propriety. No chance to peek through there.

Yet the puzzle remains. How do we make this better future-proof this architecture? There was a short-term plan and a long-term plan.

The short-term plan was to go back to basics. Looking back at the code in its current state and finding ways to optimise them. Moving into this mode, there were a few objectives

1. Knowledge growth

The main source code was not built by us. As part of the business's release strategy, we first engaged a vendor to build it. This has pros and cons, but eventually, when the curtain was opened and we were exposed to what was happening beneath the covers, I knew our work was cut out for us. Our developers poured countless hours working through a base understanding of the code. We poured through and listed areas of improvement daily whilst working through system enhancement and building upon the current feature set.

1. Building the process

I had built a new team of developers and some orientation work and team building to be done. Processes had to be built, and I have to drive these processes to enable us to build for the future state. This was necessary to bring us into a cadence of enhancements, changes, and the building of new feature sets.

1. Establishing a team

Basic routines like monitoring, duties and cadence of releases had to be built from scratch. Meetings had to be set up, and other important things like 1-1s, peer reviews, and performance management had to be structured as part of building a team. Most of the team was remote, and navigating cultural needs and norms was a huge task.

These three objectives were a great start for us to embark on, and for me, it was a no-brainer that we had to go back to basics to establish the team’s foundational roots.

As these were happening and contextual understanding was built, I started working out plans for a new future state. Looking back, I was glad that we took this path. It is always easy to be critical and think things can be done better when we are not the originators. However, these systems were also built upon our requirements and what was projected at the point of development. Many assumptions had to be made, and approaches were measured as growth was an unknown factor. There was no point in building a skyscraper when there would be nobody to live in it; hence, we agreed to build a smaller, more manageable house first!