Can you have both Governance & Freedom in Developer Experience? Part 2

Part of the Engineering Leadership Series

This is the 3rd article in an ongoing series on Developer Experience where I'm initially to exploring how to balance developer enablement, the needs of the individual, with organisation efficiency at scale.? Its aim is to give technical leaders in the organisation some tactical mechanisms they can put in place to drive a better developer experience from an architectural perspective.?

In the previous article , I explored the concepts of an adaptable model of architectural governance for Developer Experience that encourages freedom and innovation in the business problem space and at the team level, while giving more constrained guidelines where they can have the biggest impact.? Typically these are concerns that have to be done well but don’t add directly to the customer experience. e.g. Running Kubernetes better than anyone else is of high business value to AWS, Google or Microsoft but much less so to a retailer or insurance company.? ? These capabilities need to be done as effectively and cheaply as possible to preserve capacity for more business value adding work.

The exceptions to this rule are capabilities which are high risk to the organisation with potentially catastrophic consequences if they go wrong.? e.g. Data and Security. The impact of PII ending up on the dark web because of data mismanagement or cyber breach are extremely high with Board? & Directors possibly being personally liable. In these areas you want to have solid, tested and certified architectural guidance and engineering components on demand to teams so that these constraints don’t becomes tomorrow’s blockers.

This is what I call the Gravity Well Model for architectural governance. The idea is that the greater the mass of a capability in an organisation, the more it distorts the surface teams work on and the slower it moves. The areas that need greater innovation are free to move fast but if they become significant to the organisation (greater mass) then they fall further into the well. As an engineering leader in your organisation it’s your job to find the right balance.

For further reading:

Why Constraints Are Good for Innovation , Harvard Business Review, 2019–11–22

Innovation Starts with Defining the Right Constraints , Harvard Business Review, 2021–04–05

Where to draw the lines

Where that value line lies will be dependant on the strategy and business model of each different organisation. What is always consistent though is that capabilities ABOVE the value line contribute to revenue growth, usually the CEO and CFO’s #1 priority. ?

The capabilities BELOW the value line contribute to reducing costs, currently priority #6 according to Gartner. ? This is where a lot of the non-functional requirements lie in your system (scalability, quality). ? You have to do the capabilities below the line well for the lowest unit cost.?

A mistake many CTOs make is to focus on the cost element over the ability to reduce the costs of technology delivery and maintain long term quality.

“Understanding your value chain is key to your engineering strategy and working out what capabilities to treat as a commodity, product or your key customer feature”

These driving forces are crucial to remember if you are running a DevEx program as the mistake most DevEx teams make is to focus on what is below the value line (tools, code reviews, cloud platforms), without articulating the benefits above the value line.

DX Governance Tactics

A traditional, centralised, architectural approach to standards will only get you so far in an environment that values Engineering Excellence and Developer Experience.? Engineering teams are making decisions every single day that shape the architecture of their systems, from their choices of languages, libraries, CI/CD pipeline architectures to integration styles and API designs. In many organisations I’ve worked with there is an Enterprise architect who works with the business and specifies the high level target architecture but doesn’t have the capacity to get to the engineering level. In that gap exists a great deal of uncertainty and teams make their own decisions. When the lack of alignment in those decisions starts to add unmanaged complexity, blocking stage gates often appears which in turn degrades the developer experience, slowing down learning, feedback cycles and breaking flow.?

As an leader, this is where you need to work with the teams to give them a toolkit to be able to make decisions on their own that are aligned with your strategy.? In the past, I have found a number tactics that work well in this space and are covered in detail in this article.?

In terms of DX, Engineering teams may not work well with “central standards” approach if it is poorly implemented and come across as too top down without actually solving their problems.? If you want to them to follow the architectural strategy and standards then you need to have organisation mechanisms that have continual feedback and pivoting the architectural approach based on results.

In the next few articles, I will detail several mechanisms I have used for Developer Experience ( aka Engineering Excellence) initiatives to set the architectural direction with the engineering teams and achieve a feedback cycle based the ability for leadership to to Observe, Orient, Decide and Act on feedback.

These mechanisms are:

1. Architectural Vision - a clear why for the direction and a “best-guess’ target state.
2. Decision Principles -? Architectural Principles that are more than aspirational
3. Technology Choices - Enumerating the paved road, the basis of a dev platform.
4. Adaptable Architectural Patterns - ideally in code with an InnerSource Operating Model.
5. Community Feedback Mechanisms - for continual improvement.
6. Architectural Decision Records - Team based decisions, stored in code, visible to all.
7. DX Tooling Strategy - How to enable cost efficiencies while meeting the needs to your teams to be effective.

Mechanism 1: Architectural Vision & Strategy

I’ve run Engineering Strategy in organisations with hundreds of teams and you need to be aware that each team would follow their own path based on their maturity, goals and constraints. ? In this context, a one size fits all standards approach is always going to ultimately fail to make a significant impact.

Modern technical leadership focusses more on mission control rather than centralised command and control where teams are given the capabilities, training and guardrails and are evaluated against mission objectives (OKRs). Underpinning all of this is a sense of purpose that binds your teams together and is a motivating force to encourage uplift and continuous improvement.?

One aspect of creating this purpose is a high level architectural vision for your strategy and is a vital communication tool to raise awareness of where teams should be moving to when they make decisions. A strong, clear, compelling and consistent vision is key to an effective engineering organisation.

In a Developer Experience environment, teams are working in a different style where they are thinking in a world of constant change. Their language is often different speaking in relative terms and not absolutes. Budget, headcount are terms of burn rate, cost to learn (experiment).? They aim to maximise learning & value delivery per iteration, not a quota of story points.? They look at value over time and talk about Net Present Value instead of ROI. Time is an equally important dimension as cost.? In this context, an architect needs to think in the cost of change and create an architecture that is designed to adapt to better knowledge with the teams.

However, there are anti-Patterns to watch out for.? Too often strategies are wishful thinking that paint a perfect future state but give teams no path to get there, no next steps and no actionable insights. ?

• Buzzword Bingo - Strategies are by their nature, forward looking and tend to focus on the future of technology. It’s very easy to fall into the trap of the latest Tech Trends report and populate the strategy with buzzwords that don’t have direct business value.? e.g. synergising digital, data and AI!
• No Next Steps - an hourglass strategy is one where there is a big top line and a bottom line of technology but very little in between on how to get to our target state. Your strategy can’t be so far ahead of your teams that it becomes meaningless.
• Death by Powerpoint - As a consultant, I have been guilty of writing strategy packs that where hundreds of pages long.? It was partly driven by wishing to “show value” for the engagement but I know that much of that work went unread. ? The strategy needs to be clear enough for teams to understand easily and remember when they are making technical architecture decisions.
• Missing Linkage to Purpose - You strategy isn’t just about technology. We are social creatures and relationships are the foundation of any team and are crucial for resilience.? Research has shown that monetary reward, incorrectly applied, can actually decrease performance (e.g. annual bonuses).? Having a common, inspiring, social purpose will do more for the performance of your strategy that any annual performance review.

How do you know your strategy is working?

There is a very simple effectiveness test for your strategy:

Go round your development teams and ask any developer what the architectural strategy is and how it helps achieve the business strategy.? If they can tell you what they are doing, why and how it aligns with the strategy then you’ve succeeded in communicating it. If not then it’s? likely that teams are building applications without knowing where/what they should be building.

Your strategy doesn't need to be complicated. Simple and understood is better than perfect and unknown.

Below is a simple high level target state architecture which is often the first image in an architectural strategy for an organisation embarking on a digital transformation.?

Mechanism 2: Engineering Decision Principles

One of the most useful architectural tools I have found for distributed decisions making is a good, clear, actionable set of principles.? Not your usual principles full of aspirational thinking such as “Our strategy unifies cloud & data” or “All our systems will be micro services” but good, clear value statements that allow engineers to make trade-offs without having to call in a Domain Architect to interpret the ancient scrolls of mystic knowledge.

Good Principles drive alignment

My goto reference for the why behind principles is Cloud Strategy by Gregor Hohpe , who lays out the ideals for principles and how they work with architecture strategy. Principles are a useful mechanism to translate your strategy into a toolkit for teams to make decisions day-to-day.?

These decisions will be in turn be the living reality of your architecture as teams decide what frameworks to use, how they integrate, what vendors to work with and what clouds to deploy on.

Defining these principles should be mildly challenging as they will cause debate and disagreement between priorities for your organisation. This is no bad thing as it means your are bringing out mismatched expectations and conflict at the early stages when they are much easier to fix and not after you’ve spent millions on a failed project.

While engineering principals help ground a group in shared commitments, not everyone will agree with every decision made about how products should be designed and built. ? This is why it's important to develop robust processes that people can trust there is a logic & fairness in the decision, even if they don't agree with the end decision.? They can at least trust the process that drove the decision and understand that a decision isn’t a mandated override from the top but one based on logic and reason.

The suggested checklist for principles include:?

1. Make sure your principles are not defaulting to wishful thinking.
2. If the opposite of a principal is nonsense, it’s likely a poor choice.
3. Principles that mention products (AI) or architectures (micro-services) may in fact be decisions pretending to be principles, a form of reverse principle engineering.?
4. Principles should stand the test of time. Is this likely to still be true in 3-5 years? If not then it may be a fad hiding as an aspirational principle.
5. Principles that involve product names or specific technologies are likely technical or commercial decisions that have been reverse engineered into principles.? e.g. "we will use OpenAI to maximise productivity and customer satisfaction" is at best wishful thinking and more likely a commercial decision based on a commercial relationship.
6. Principles should be few and memorable, enough to hold in your head so that they can be used in any conversation.? If you have 15 principles, you won’t remember them all and you will start to fill the gaps with assumptions. Amazon’s 16 Leaderships principles is an example that has IMHO too many principles to be simple, clear and effective.?

In order to turn this into reality I create a series of statements around tensions in the tech organisation and put them on sliders with the opposite positions at either end. I then run a series of workshops with stakeholders across the transformative agenda (advocates and laggards) to map where we are and where we want to be across these statements.

e.g. for "Build differentiators, buy commodity" example principle below I would have the two ends being "We build every system ourselves so we understand it and customize" and "We buy standard capability rather then build" and then plot where the organisation is and where the different personas wish to be. It quickly throws up contrasting positions and the gaps in expectations across teams. That is then used to run further workshops to thrash out the principle behind this and how far you go towards either end.

The Structure of a Principle

It helps to have a consistent structure to your principles so that they are easy to navigate and understand. I usually use this format:

1. Short Header
2. Explaining sentences(s) - Give some detail to what comes next
3. Intent - Why this principle is important to us
4. Examples / Implications

An example of a few architecture flavour principles might be:

• Build differentiators, buy commodity. Our limited engineering capacity should focus on building market differential and buy/rent undifferentiated capabilities. Intent: the more we build, the more we have to maintain. Our most limited resource is the time of our colleagues. Use it wisely and build great customer experiences.
• Build Value Through Services not Infrastructure. Focus on the service and the value being provided, not the infrastructure being built. Build services so that engineers can focus on operational performance and not maintaining a collection of machines.
• Design systems to evolve continually. System designs must be adaptable to enable ongoing small enhancements. Intent: We can't predict the future and the shifting requirements so our systems need to assume change.
• End-to-end ownership: Our engineers have complete ownership over their projects, all the way from inception to production and beyond. Intent: We want teams to be laser focussed on customer outcomes and the systems that underpin that. With ownership comes accountability.

Other examples of principles

• GitLab Development Principles

The Cloud Strategy ?book also references a paper (Impact of Principles of Enterprise Engineering ) that goes into this concepts in more detail if you want to dive deeper.

Another useful reference is Engineering Values and Architectural Principles

On Principles versus Values

Principles and values are foundational concepts that guide our behavior and decision-making, but they are distinctly different.

Principles are fundamental truths or statements that serve as the foundation for a system of reasoning. They are often universal and can be applied across various situations and groups. Principles also remain consistent over time and across different contexts.

Values are deeply held beliefs or ideals that individuals or groups consider important. They are subjective and often influenced by personal experience, peer groups or cultural influence. Values can vary group to group. e.g. Architectural values will be different to engineering values.

• Principles guide actions and decisions in a consistent manner.
• Values shape attitudes and influence behaviors and choices.

As an example for a company they may follow the principle of continuous improvement, striving to enhance processes and products consistently. They same may also value sustainability, innovation, and customer satisfaction, which reflect their priorities and cultural ethos.

In summary, while principles are universal guidelines that provide a consistent framework for actions and decisions, values are personal or organizational beliefs that influence motivations and priorities. Both are crucial in shaping behavior and achieving goals, but they operate in different domains and serve different purposes.

Example: Engineering Values from HubSpot

Mechanism 3: Managing Complexity with Tech Stacks

What started as one mechanism in a list has turned into an article in it's own right as it's a very rich topic. To respect your reading time, I'm pushing this to another publication next week. See you then.