System Design Workflow

In a few interviews recently, I’ve been asked “What metrics do you like to measure related to production engineering excellence”. This has prompted me to do some research and some thinking. I'm a firm believer in the significance of metrics, but I also understand that mere measurement doesn't inherently enhance product quality. It's a backward-looking approach, offering insights into current quality levels but not providing guidance on enhancing engineering quality. In fact, I've observed that prioritizing metrics retrospectively can trigger a knee-jerk reaction from teams, as they scramble to address suddenly evident, pressing issues. This activity never translates into improving process to consistently improve first-time yield.

This concept reminds me of someone obsessing over their personal financial budget, finding ways to cut costs and cut back, when it would be better to spend time investing in their ability to generate more income.?

With this perspective in mind, I’d like to share a technique I’ve used over the past 8 years that I believe has had a huge impact on the quality of software that my teams have delivered. It starts way “left”, before a single line of code is written. It’s a key concept to empower engineering teams to be inclusive and use the diversity of their team members to “Build the right thing” before they “Build the thing right”. So today, I have recreated my “High Level System Design Review” workflow document template and made it available for you to try with your projects.?

Objectives

This HLSD Review template provides a workflow outline be filled out by the Tech Lead of a project and helps them ensure that:

• The relevant members of the team are included, consulted, and informed
• The "user needs" and "business objectives" of the project are understood and documented
• Emergent properties are considered
• Non-functional requirements are identified
• Technical discussion and decisions are documented

This template serves as a reliable tool for your teams to continuously refine and evaluate your system designs, facilitating and normalizing peer-review as part of your process. Defining the workflow checklists also allows you to evolve and enhance over time.

Using The Template

This template can significantly empower your Tech Lead's approach to design, regardless to their level of experience. I highly recommend that you allow every engineer on your team to lead the design and delivery of a feature or capability as this will ignite energy, commitment, and growth. Conducting normalized design reviews allows everyone to contribute and learn how to build great system designs.

Following the workflow in this template should take less than one day's effort, and the payoff is huge; it steers clear of potential technical debts that could drag on for months or even years. I've seen this first-hand and received glowing feedback from my engineers, junior and senior. Also, every peer review session turns into a brainstorming treasure trove, sparking new ideas, sharing insights, and rallying team engagement. In total, we're talking less than two person-days to get this done, which is a smart trade-off considering the heaps of time it can save in the long run.

Be sure to identify someone to represent each of the "reviewer roles" in the template and invite them to participate in the review. The Product Owner must review and approve of the design. They may not be technically trusted to create the design, but they will know it is "building the right thing" when they see it. The QA Lead must also be invited to participate as they will find that understanding the design is invaluable to their analysis and definition of verification tests.

I’ve provided guidance on how to fill-out the template in italics in the template itself. These prompts should be replaced with your content.?For the most part, I hope it's intuitive and self-explanatory.

Product and Technical Design Principles

The heart of the template lies in the "Product and Technical Design Principles" section. Here, you'll find a series of thought-provoking principles, each typically requiring a 5-10 minute reflection to jot down your insights. More often than not, these musings crystallize into clear non-functional requirements. Make sure to record these in the designated table; it's crucial for future scoping and estimating. Occasionally, you might encounter a principle you believe isn't applicable to your project. In such cases, it's OK to explain why, but remember, you'll still need sign-off agreement from the relevant reviewer.

At past companies, we’ve defined more specific product principles, typically trickling down from the CEO to the Product Management and Engineering teams. Some examples that I've integrated into these templates from past experiences are:

• Mobile First - products must function well on mobile devices.?
• Safety First - prevent the user from causing harm to themselves.?
• Alarm Fatigue - don’t inundate the user with warnings or they’ll tune out and miss real issues.?
• Never Enter Data Twice - pre-fill data that we already know about the user wherever possible.?
• Hide Implementation Details - provide a platform that allows the company to pivot while still providing the same functionality to the users.
• Partner Ready - build components that can be white-labeled and embedded into partner sites

You will want to add your company’s guiding principles in this section. This template should evolve over time to represent the principles your company values. Additionally, I’ve typically made each principle in the template link to a wiki page that provides more context on how the engineering org interprets that principle. As you find root-causes to issues in production, it’s a useful exercise to review and update these principles in the template so as to avoid future similar occurrences. You could bake this step into your RCA process.?

Usage Scenario Sequence Diagrams

Mapping sequence diagrams for each usage scenario is another beneficial step to ensure the design delivers efficiently and effectively and for future understanding / refactoring. I usually place these diagrams within the system design documentation, linking them to this document for easy reference. Use a tool like PlantUML to create these diagrams from text so they can be easily updated. I find it best to keep the diagrams simple, catering to only one scenario per diagram.

These diagrams illustrate the user's interaction with the system and how the system's internal services collaborate to achieve a usage goal. It's key to keep the interaction descriptions at a semantic level, steering clear of technical jargon and specifics like APIs or existing technologies. This approach not only makes the design universally comprehensible but also fosters creative thinking about UI and API design to fulfill system requirements. I've often observed scenarios where pure API principles dominate product design, leading to less than ideal outcomes.

Whenever I've worked through building a sequence diagram, even for apparently simple scenarios, I've always surprised at the edge cases I discover that often lead to fundamental design decisions and changes to my preliminary ideas.

Conclusion

I hope this will be helpful to you. Be bold, try adopting this for a new project, and make it your own. In my experience, even for small features, it is surprising how useful this has been to avoid pitfalls that would have otherwise occurred. I've tended towards adopting this process and keeping the expectations low, rather than raising the bar and having it be neglected. The Pareto Principle can be very effective when combined with a process like this. I'm not advocating that you only apply 20% effort, but applying 20% effort across every aspect of a design process is radically more effective than only performing 20% of a design process.

Now, go out there and create some amazing designs, ensuring quality right from the start!