Creating a Culture of Engineering Excellence

As a company evolves from an early stage startup (Seed to Series B) to a late stage startup (Series B onwards), engineering culture needs to shift from "build the right product" to "build the product right" to ensure continued engineering excellence. Most of the time this shift in culture and mindset does not take place due to many reasons. The primary reason being the lack of experience at the leadership level and not knowing that this shift needs to happen. The implication of not making the shift at the right time can be very costly and painful.

“Software and cathedrals are much the same; first we build them, then we pray.” ~ by Anonymous.

If the engineering team fails to "build the product right" achieving scale after product market fit becomes very painful. The following problems start manifesting due to lack of engineering excellence.

• Team Velocity & Timely Delivery: Engineering quality issues, architectural issues, and mounting technical debt leads to longer lead & cycle times. This reduces velocity and increases missed timelines for critical projects. This impacts the ability to scale revenue with new product and capability launches.
• Organisation Scale: Adding more engineers makes the problems worse due to spaghetti code base, brittle architecture and lack of automation. This impacts the ability to scale the organisation to support the greater demand leading to loss of new revenue.
• Customer Experience: Due to service interruptions caused by defects and instability of the production environments, customers generally tend to have a bad experience leading to increased churn.
• Developer Experience: As the on-call volumes increase due to production instability it starts to impact the quality of life of engineers as it sucks up bandwidth, reduces moral and motivation leading to higher than usual attrition
• Budget Bloat: Larger teams are required to support the organisation scale leading to inefficient allocation and utilisation of engineers, which leads to lower ROIC for the company.

These are typical symptoms that your engineering team is struggling to cope with the growth challenges and your product is in trouble due to bad engineering practices and culture. Getting out of this requires significant investments to "build the product right".

"Engineering excellence is the pursuit of being excellent at the craft of engineering, being both efficient as well as effective in delivering delight to customers and in the process deriving pride and satisfaction from the work and the finished product." ~ by Anonymous.

During the "build the right product" phase, engineering is all about prototyping and conducting low cost experiments to validate different hypothesis to provide continuous feedback until you get to the right product.

This phase requires engineers who have a hacking mindset that knows how to take a lot of shortcuts to enable faster experiments at a lower cost. These engineers are excellent at building fast prototypes to test new product hypothesis at a lower cost. Some startups tend to hire low cost and less experienced engineers to reduce expenses. This tends to be an amateur mistake and can cause lots of issues later on. The people you hire for engineering at this stage might not be the people who are going to help you scale. This is an important and painful lesson I have learnt over the years.

Build The Product Right

The next stage of evolution to "build the product right" requires different type of people altogether. Getting the small things right plays a big role on how this phase evolves as the company hits scaled growth.

“Time pressure gradually corrupts an engineer’s standard of quality and perfection. It has a detrimental effect on people as well as products.” ~ by Niklaus Wirth.

Principles for "Build The Product Right"

For this phase you need to get 4 things right. The following 4 principles will help establish a good culture of engineering excellence to help the organisation scale.

• Leadership: At this stage of the journey, startups require experienced tech leadership and a legit CTO who is hands-on and knows the fundamentals well. This is not the time to hire people who have not done this before and promote them to be the leaders. This is the most important thing to get right. Founders of startups need to spend significant amount of time searching for the right tech leadership with the right culture fit. If you get this wrong, there is going to be a lot of pain.
• Engineering Team: Upskilling existing engineers with proper training and an infusion of new experienced senior engineers to your teams is going to be vital. Hiring high caliber engineers while upskilling your current engineers will need to happen in parallel. This is the time you need to ensure you have the right mix of engineers to managers to architects ...etc. The following ratios have helped me established a balanced team.

Engineering Team Setup Ratios (Max Ratios)
------------------------------------------

- Engineer to Manager = 8:1

- Engineer to Product Manager = 15:1

- Engineer to Principle Architect = 40:1

- Junior Engineer to Staff Engineer = 5:1

- Staff Engineer to Principle Engineer = 10:1

- Engineer to TPM = 50:1

- FTE to Contractor = 4:1
        

• Foundation: Most of the times we tend to ignore the small things. Doing the small things well leads to higher quality outcomes. It is really important to do the small things in a world class way. Getting the foundation right is essential for scaling the company. If not, you are building on top of a "house of cards". Longer you continue without fixing the foundation the harder and more costlier it gets to fix. I suggest you get this foundation right from the very start. If not, at least get it right before hitting product market fit.

Getting the small things right - Foundation
-------------------------------------------

- Unit Testing & Acceptance Testing

- Automated Continous Integration, Delivery, and Deployment

- Static & dynamic code analysis

- Code Quality & Security Metrics

- Environments as cattle and NOT pets via Infrastructure as Code (IAC)

- Tech and Architectural Debt Measurement & Reduction

- Blue/Green or Canary Deployments to production

- Feature Flagging & Feature Ramping

- Observability, Alerting & Monitoring - User Journey Level (MTT|D|A|R)

- Automation, Regression, Somke Test Suits & Infrastructure

- User Journey & API Documentation

- Automation of User Journeys and APIs

- High Availbility with DR Production Setup (RPO|RTO)

- Service & Platform Level SLAs, SLOs, SLIs 

- Code Reviews

- Developer Productivity Tools and Resources

- Performance & Scale with Auto Scaling

- Platform Architecture Principles & Governance 

- Hiring & retaining "A" players & getting the wrong people off the bus
        

• Engineering Culture & Mindset: Continuous pursuit of engineering excellence is a journey. It is important to form good engineering habits as an organisation. These need to be embedded in the engineering culture and practiced like a religion. It takes time to form new organisational habits. It is critical that your senior and middle management is well versed in these practices. If not, need to hire the right leadership and get the wrong people off the bus. This is too critical to risk it.

Engineering Habits and Practices
--------------------------------

- Culture of Continous Refactoring

- Domain Driven Design Practice

- Pair Programing Practice

- Test Driven Development Practice

- Strong DevSecOps Practice 

- Quality & Security Conscious Culture

- Culture of Radical Transparency

- Culture of mastery in the pursuit of engineering excellence 

- Culture of learning and growth mindset
        

“Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” ~ by Martin Fowler.

Majority of the startups don't make the switch to "build the product right" on time. Usually this leads to several failed attempts before finally getting it right. The reasons behind the failed attempts could be a lack of understanding of all the moving parts that is needed to get right. Startups that do have the right leadership in place or have high quality engineering talent that has experienced this before are more likely to get this right sooner.

My experience leading several startups and having made lots of mistakes along the way have thought me valuable lessons on the importance of building a culture of engineering excellence. I hope you find this article helpful.

Dhanush Hetti