The Rise of Developer Experience

Part of the Engineering Leadership Series

Over the last year there has been a lot written about developer experience with Gartner putting it as one of it’s top technology trends in 2023, tech companies like Microsoft, Atlassian, RedHat and Amazon are building their DX capabilities and newer companies like Humanitec, Linearb and getDx are creating a marketplace presence.?

Personally, I’ve been working in the field for the last 7 years, helping teams improve their Developer Experience (DX) to achieve their goals by improving their working environment with tools, better processes, platform engineering, education and simpler ways of working and I’d wanted to start to share some of that experience.

If you are not familiar with Developer Experience and what it is, I’ll help walk you through where it came from and why it’s so important in 2024. ?

So what is Developer Experience (DX) and why is it so important??

With increasingly volatile macro conditions, organisations need to be more nimble and innovate faster.? From pandemics to market instability and the cost of living crisis, your technology capability must be delivered much more reliably, faster and cheaper than it was a few years ago.? In a recent survey by Forrester, 87% of DevOps leaders agree increasing developer productivity is a priority in 2024.

Unfortunately few companies have traditionally paid much attention to how they build software with low alignment between teams.? With shared engineering services, some companies have a centralised team providing a mandated service that doesn’t aways scale or meet the needs of teams.? Usually no-one is responsible for the end-to-end flow of features through the product development process and the hand-offs between teams and the daily lived experience of engineering is fun of friction, inefficiency and lost information.

DX turns this on its head and it’s the developer at the centre of focus for the development process with supporting teams focussing on satisfying their needs, usually via a combination of self-service and easy to consume guardrails. A pull, rather than push model. Beyond tools and processes, how a team is run and managed also has a big impact on the daily lived experience of team is often comes down to training and skills of the Engineer Manager.?

“Developer Experience is the lived experience of an engineer building and running systems, focussing on people and what they need to be effective* & safe.”

*I deliberately chose effective over efficient but that is a topic for another article

In practice, DX is a combination of design thinking, digital customer experience techniques and software engineering to provide developers with the ability to innovate quickly, with reduced cognitive load and get rapid feedback on their changes.? Some companies will have a wider Employee Experience (EX) team in their People team and will have some overlap with DX (onboarding, HR systems, management training) but DX is very targeted at technology teams and goes much deeper into the friction points and how teams work.? A DX team may also collaborate with the EX team to engage in external stakeholders and be part of both a talent attraction and retention strategy.?

Why this is important is that most technology teams are working significantly below their true potential.? Business Leaders feel that they can get 10x times performance out of their teams than they are getting. This leads to team burnout and shifting priorities with 40% of digital innovation funding wasted.? If that wasn’t enough wasted effort, only 8% of what is planned by agile teams gets delivered.?

Software Engineering Top Trend

Gartner believes that?"developer experience?extends beyond developer tools and technologies. “The tools used in day-to-day work certainly play a role in improving the quality of developer workflows. However, developer experience also depends on non-technology factors. These include having dedicated time for deep, creative, meaningful work, as well as personal freedom to try new things?without the fear of failure,”?

InnerSource is the second pillar of modern developer enablement and is a topic which I will come to later in this series and why it’s the secret sauce to a successful digital transformation.? I’m also a Director at the InnerSource Foundation and it’s very exciting to see this being recognised as a key enabler for teams.?

McKinsey is also working in the field of Developer Experience with a report that states a better DX can lead to extensive benefits for organisations, such as improved employee attraction and retention, productivity and security. They recently republished an article around developer metrics that received significant feedback, having missed the core principles of measuring developer productivity. Software engineering is unfortunately a space with many opinions and scarce research and evidence to back up the said opinions.? ?

Deloitte are also starting to publish their emerging views on Developer Experience and it is one of their 2024 tech Trends.?

Even Amazon, famed for its two pizza teams and “API Mandate” that declared teams only talk through APIs has been struggling with a drop in developer productivity and is investing heavily and building its own DX team.?

Embracing DX is not merely a short term trend; it's a strategic imperative for any organization looking to thrive in the ever-evolving world of software development.?

In an era where the supply of easy funding has disappeared and companies are being forced to look at their cash flow, or pivoting their investment to AI initiatives, Developer Experience should be the #1 priority of the CIO to drive the performance of her company.

Lessons from Digital Transformation

For me, DX has its origin around 2010 in the world of DevOps and Digital transformation where it borrowed from systems thinking of DevOps and the Design thinking of Digital.? Every successful digital transformation places the customer at the centre of its why and works out from there, reshaping technology and processes to work for the customer and not, what usually happens, making the customer bend to companies policies and clunky systems.?

Around 2015/2016, I was combining digital CX techniques with the principles of Lean Software Development (Mary and Tom Poppendieck, 2003) and Product Development Flow (Donald G. Reinertsen, 2009)? to build a developer-centric model of DX that focussed on flow by reducing friction with effective tooling, optimising for fast feedback with good CI, building guardrails to allow for decentralise decision making and reducing cognitive load with a clear “source of truth” of knowledge. After 25 years being a developer working in 18+ companies across 3 countries, I had a good idea of what stopped teams being effective in many different situations.? It wasn’t an academic approach and was subjective based on experience but it was good enough to be effective.?

During the last few years, the field has come on immensely and moved to much firmer ground with the academic rigour of studies being produced by university and corporates with specific call outs to Nicole Forsgren, GitHub, Margaret-Anne Storey, University of Victoria (and et al.) for their papers on SPACE metric framework and Developer Productivity and Andre N. Meyer, Earl T. Barr, Christian Bird for their studies on how developers work.

So What is a Good Developer Experience?

"If you organize around the consumer, the rest of it will follow." – Eric Schmidt

A good developer experience focuses on what teams need to achieve high software delivery performance will make it much easier for teams to deploy, learn from their mistakes safely and increase their resilience.

One of the quickest way to understand if you have a good developer experience is to ask your developers! ?

Indications of the quality of your DX:

• Am I able to onboard quickly and easily to a new team?
• Do I have timely access to systems and tools??
• Do I have to open services tickets and wait on other teams or can I access what I need for my job quickly??
• Do I have clear guidelines on how I am expected to work??
• Does my team have a clear mission, business KPIs and a realistic backlog??
• Can I understand the code base quickly & easily?
• Do I get fast, effective feedback on my code changes??
• Can I use Open Source software easily or are there significant barriers to bringing in emerging tech??
• Can I find shared code & libraries (InnerSource) and get support for them??
• Do I have clear guidelines on security standards and what can go to production??
• Is there a clear place to find up to date information on process??
• Do I have easy access to the best tools and which ones are mandatory and which ones are free choice??
• Are there feedback loops from production/operations teams??
• Do we have a capacity for improvement in every sprint??

If you can answer yes to most of these then it is indicative of a good DX where you teams have the potential to be effective. ? If not, then it would be worth running a listening session or survey with your development teams and identifying the highest priority items that they feel they need. Software delivery is a complex socio-technical system that is hard to measure which means that qualitative data is often good enough to get started. What important is that you have a baseline of data to measure progress against.?

Benefits of a Good Developer Experience

The benefits of building an environment designed for efficiency covered most?

• Improved Workforce Happiness.
• Reduced Cost of Change.
• Reduced Cost of Maintenance.
• More Systems Resilience.
• Higher Functional Quality Outcomes.
• Higher Non-Functional Outcomes (security, availability).
• Reduced Cost for Compliance.
• Increase ability to respond to customer demands.

According to the survey by Forrester around internal developer platforms & DX, survey respondents are observing significant benefits of a DX program, specifically:

• 74% see improved productivity.
• 77% can shorten change lead time.
• 85% positively impact revenue growth.?
• 82% reporting increased customer satisfaction.?
• 81% see improved developer recruitment and retention.

The last point on talent retention is so important because it can cost $50,000-$100,000 to replace an engineer in a team. LinkedIn even has a calculator that shows how the cost of replacing a tech leader can be 250% of their annual salary.? Typical attrition rates for a tech team are around 12-15% in normal conditions.? If you had a team of 100 engineers, that’s a spend of $750,000-$1.5m just to stand still and maintain your current tech capabilities. That cost is paid in fees, lost IP and time, time that your organisation could have used to move forward with new value add features instead of frantically working to stay still. Any shift in this number alone is usually enough to pay for any DX program.?

When making the case for a DX program, the above the line (revenue) will always have more persuasion than below the line reduction in costs, unless your company is under pressure to cut.

While the benefits of a DX program sound attractive, what does this mean for the bottom line??

Let’s take a mid-sized corporate as an example which has a cohort of 1000 engineers, depending on which roles are included in that definition, with a blended rate of $850/day between on and off-shore.? That’s a burn rate of $850k per day (~$190-200m/year @ 225 working days) in costs for those engineers before even considering cloud, software and support team costs.? It’s not unreasonable to get a 20-30% increase in feature delivery when working with a team end-to-end to improve the DX.? That 20-30% boost can be used to outperform your competition, be put back into continuous improvement or taken out of run costs by running a smaller workforce with saving of ~$45-50m. In my experience, there is never a shortage of work to fill the new capacity create by DX.?

What Does DX mean for my teams?

Firstly, it doesn’t mean you have to increase headcount or budget.? It’s a reshaping of your current budgets & resource profile to a new paradigm.?

At the core of a DX team is the concept that the developer is a customer and has a choice on which services to use. This is a significant shift in mind shift for a traditional services based team. E.g. could platforms, service management, AppSec. Even if a team doesn’t realistically allow teams a choice of what capabilities they use, it’s imperative that they treat the application teams like they do have a choice.? This is often very hard to do in large organisations who split responsibility into silos with their own mandates.? For instance, the security of an application lies ultimately with the CISO even if it’s an application team deploying insecure code. In this context, an AppSec team is not naturally incentivised to enable productivity in an application team and will be focussed on ensuring that no exploitable code makes it to production.? Similarly a cloud platform owner will typically be judged on their performance by the cost and compliance of the cloud platform, not the ease of use of efficiency of teams using the service. A Service Management team will, by default, focus on service stability over feature delivery.? These behaviours are a natural reaction to the size of an organisation and the need to have a single owner for a responsibility.? These problems are not new and I’ve been solving them since before the days of DevOps with the idea of shared goals.? DX takes a new approach to solving this problem.

The shift for DX is there needs to be a central team to own the accountability of the lived experience and co-ordinate between each of the functional sites that faces into the application development teams.? The team typically consists of a Technical Lead, Product Owner, Technical Writer, L2 Support and some developers to deploy a simple developer portal as the entry point to knowledge and services from other teams.? The main purpose of this team is to act of the glue between other services (cloud, k8s, security, monitoring)?

It doesn’t need to be a big investment but each supporting silo needs to be aligned on the need and benefits of the DX team otherwise they will struggle to make any significant difference if priorities are not aligned.

The other approach to effective DX is Engineering Operations (EngOps) where there is one large team providing technology as a platform to applications teams under a Head of Engineering such as a container runtime, API gateway, pipelines for CI, developer portal, database capabilities, messaging, training, monitoring, observability, test data management, application security, shared components (InnerSource), Service Management integration and automated compliance reporting, to name a few possible capabilities needed by app teams.

While centralising an engineering function may seem like an idea from the past, it can work well if the EngOps team has a strong product focus, practices platform engineering and uses design thinking to put the developer at the centre of the service, not the usual governance roles. ? This technique is a bigger change to make for your teams but it’s able to move faster and with a good product owner, make more of a positive impact.? The key here is to ensure that the team minimises backlog coupling between platform and application teams and allow teams to self-service as much as possible and be able to make contributions to the platform to set their need.?

Evan Bottcher’s (MYOB, ex-Thoughtworks) definition of a platform fits well

Into this space:

“A?digital platform?is a foundation of self-service APIs, tools, services, knowledge and support which are arranged as a compelling internal product. Autonomous delivery teams can make use of the platform to deliver product features at a higher pace, with reduced co-ordination.“

An EngOps team is all about the business of running technology for technology teams and as it has the prime focus on developer productivity, it is able to face into multiple stakeholders such as Architects, Security Architects, AppSec, Finance, HR, Integration, workplace engineering and Operations.? In addition to this, a strategy I have found useful is to bring the development teams along the journey with the EngOps team

When rolling out a DX program, one anti-pattern to be aware of the that engineer managers? don’t always understand the nature of software engineering and can consider working in tools, shared libraries or non-direct delivery work as developers “tinkering, not delivering”. I’ve seen this behaviour in multiple organisations and while there is can be a tendency for development teams to work on great tech without having a clear business value realisation plan, it’s important that Engineer Managers understand the benefits of DX and have KPIs which support the work. If they are measured on point delivery per sprint, they are not likely to invest in the long term improvement unless executive leaders create the right incentive structures.? ?

DX, Metrics and Productivity

Developer Metrics are difficult to get right with many opinions in the field and a strong desire for managers to “manage” their teams, not always in ways that helps the team. ? However, for every dollar spent on DX, you need some form of reliable proxy data to show the benefits you put forward in the original business case. A good place to start is the DORA metrics.

DORA (DevOps Research & Assessment) has become the default word on high performing teams in software delivery thanks to the great work by Nichole Forsgren and team. Our industry has fundamentally be moved to the next level by the academic rigour she brought to the field of software engineering, filling an area that had little data and many opinions.? DORA has established that companies with a high rating on the DORA metrics are also highly likely to be outperforming their peers.??

Many companies I have worked with are using the 4 key DORA metrics as a way to benchmark their teams.? My advice when doing this exercise is to start with a few simple teams to understand how you are doing to define the metrics. For example, one of the metrics is deployment frequency but it’s not always clear what a deployment is for every team.? If you are doing blue/green deployments or feature flagging, is it deployed when the code is in production or when it’s in use? Releases often also bundle multiple changes which can also skew the Change Lead Time metric. Remember they are lagging indicators of progress and are most effective for benchmarking.

If you are looking for metrics to measure developer productivity, there is an entire field of research about how teams write software that can inform how you measure teams. The general guidelines for developer metrics are:

• Don’t measure the individual as it creates an unsafe environment.? Software delivery is a team sport. ?
• Measure a system or process, possibly a team of its large enough to not isolate individuals. ?
• Any single metric you chose to measure performance will be over optimised at the cost of all else. ?
• Measure outcome over outputs.? Eg service quality over number of pull requests or story points.
• Engage teams to define metrics. If they are useful to teams, they will use them and help you improve them.

If you measure no other metric, measure the quality of the code review. It’s the single largest indicator on the quality of your code.

Conclusion?

I hope this has given you a high level introduction to Developer Experience and how you might start thinking about it for your company.? It’s an area that I’m deeply passionate about and it’s part of my personal mission to improve the daily life of every developer.? ?

Hopefully you will be including it in your technology strategy for 2024 and sharing your story as inspiration for others. If it’s not on your radar yet, then it’s probably being implemented by one of your competitors.

In future articles, I’ll cover other areas of engineering strategy and DX in more detail.? I’m always open to feedback and suggestions and if you feel there are improvements in any of the above, please do get in touch.

See also

• Can you have both freedom & governance in a good Developer Experience?