Assuring Quality in Agile-at-Scale: 5 Anti-Patterns

Disclaimer. This article is based on my own experience and stems from my reflections on several large-scale agile transformations I have contributed to, performing a number of roles within and outside the testing space. It does not relate to any specific project or employer, and presents my personal opinions only.

In their journey towards business agility, many companies are scaling up agile frameworks to serve and integrate wider and wider parts of the organisation.

Scaled-agile methodologies have now been experimented on the ground for years, enjoy a vast literature, and have become a de-facto standard – irrespective of which one is your favourite. Yet, I feel there is still an aura of magic around agile-at-scale that mystifies some key elements of the delivery cycle, and above all the management of quality.

In my personal experience as a test manager in large delivery programmes, and from the vantage point of owning Quality Assurance for a large portfolio, I have observed a number of common pitfalls in the implementation of a QA function for scaled-agile initiatives.

I found that these thoughts resonate with many, yet they are not clearly articulated in the literature, where they would deserve big warnings. Since I cannot read every book or article on agile, I have decided to come clean and seek the feedback of this community.

Whilst in the following I use SAFe terminology, the concepts are general, and the focus is not the frameworks, but their average implementation. Finally, I use the phrase test/QA manager for the responsible of the test strategy and process, irrespective of people management responsibilities.

Here we go…

1. QA Managers are needed more than ever and at all levels, yet no framework prescribes this role.

With the digital revolution, testing is more vital than ever, and the test strategy has to sit with an SME. Moreover, scaling up requires some level of standardisation, and that cannot come from self-organising community of practices: thought leadership, and accountability for decisions, are required. At scale, the 11th principle of the agile manifesto does not hold.

Quality must be managed actively at all levels of the organisation, but this does not mean that we need to go back to the old-school setup of a test team with a dedicated test manager. Managing teams and devising a test strategy are distinct responsibilities: it all boils down to have the checks and balances in place to ensure that the strategy is embraced even when removing the people management element.

These checks and balances mainly take two shapes:

1. selected quality-related metrics and KPIs agreed with the project leadership and regularly reviewed;
2. clear accountability for the overall architecture of the QA function, together with the empowerment to have it implemented (directly and through delegation to the delivery teams).

Whilst metrics and KPIs are found in many frameworks for agile-at-scale, there tend to be no clarity on the accountability for test processes, architecture, enablers, etc. “Quality is a joint responsibility” is a great motto and I subscribe to it, but it doesn’t take away the need for an experienced owner to drive and consolidate all the input in a coherent strategy.

Don’t stick to the letter of the frameworks (“there is no test manager role in SAFe”), but take them as frameworks and follow their spirit (in SAFe, for instance, test mangers can easily be characterised as architects at train and solution level).

2. The System Team is not a parking lot for all that doesn’t fit a regular delivery team.

Where we have got test managers, they are often treated as second-class citizens and relegated into a System Team, together with many other critical delivery functions. But there is no value in imposing a Scrum Master to a bunch of professionals who perform different jobs just for the sake of reducing the number of “managers”. Actually, expecting this poor SM to be on top of release planning, test management, environment strategies, and so on, is unfair to the individual, demotivating to the team, and ?dangerous to the project.

RTEs/STEs and the like: if quality is a top-priority of your delivery agenda, then your QA manager must have a sit at your leadership table. Why? This will drive consistency in the train and ensure that delivery teams are functioning correctly from the quality angle, preventing the following anti-pattern.

3. Pushing testers to scrum teams is not enough.

A classic move on the journey to achieve agility is to disband centralised test teams and distribute testers to scrum teams (and test managers to system teams).

Result: test managers (if any survives) try to push the quality strategy by virtue of influence, for they do not have a test team any more. If they do not have a seat at the table (anti-pattern 2), that is, if the scrum teams are not mandated to work within the boundaries of an agreed test strategy, the immediate consequence we observe is waterfalling the sprint by postponing test-related activities, soon followed by the creation of a specific type of technical debt: quality debt! The same happens without adhering to the overall strategy for developing, releasing, etc., but those seem more difficult to get wrong: we can demo untested software, but we cannot demo undeveloped software…

So, what should we do? Push testers to scrum teams, yes, but also push quality into the teams' Definition of Done, so that the coverage, execution and automation of testing become primary concerns – and then stick to the DoD (KPIs & metrics, as mentioned in anti-pattern 1).

The main side effect of pushing quality into the DoD is to avoid waterfalling the sprint: before starting working on a new user story, a developer must help automate missing test cases, and a business analyst execute some manual test cases that is not worthwhile automating. T-shaped skills are not a matter of vocation or personal interest, they are a requirement for a functioning scrum!

Another desirable side-effect is to promote quality considerations as first-class citizens of the delivery discourse of POs/PMs and SMs/RTEs, for they cannot show progress without fulfilling the DoD.

In other words: if quality is not part of your DoD, due to the increase in technical debt the business?value delivered over time is doomed to decrease, sprint by sprint, PI by PI, and the time-to-market will increase (or worse, defects will be discovered in production and make it to the news).

4. Developers (and everybody else!) can and should test. But do not forget the pyramid.

This one is actually discussed here by Tricentis founder Wolgang Platz, but it is worthwhile highlighting.

The focus on automation and on the test pyramid has a tendency to promote unit testing above other test levels. In combination with the lack of a test strategy (which is the result of anti-patterns 1 and 2), this leads to delegating all testing work to developers, believing we are actually doing the right thing just because we are “shifting left”.

However, integration and end-to-end testing are critical at scale: the actual value comes from integrating deliveries across a wide landscape of applications, and only at a given point in the release cycle that integration becomes possible and can be tested in full. Moreover, functional testing is closer to simulate the user mindset in production than unit testing is.

5. If the delivery plan does not follow E2E increments, then the whole delivery is waterfall.

Testers are often asked to decrease the length of the test cycle, especially when approaching a planned release date. Of course, automation plays a role here, but have we delivered the software to enable early and incremental testing?

The whole delivery plan needs to be shaped so that the target E2E business flows are delivered in increments. This is a key factor in reducing the length of the test cycle, for defects are uncovered and fixed earlier, and the robustness of our tests (manual or automated) and the confidence with our chosen coverage mature over time.

Otherwise, expect a long and painful end-to-end test phase at the end of the delivery cycle.

Summary

Popular frameworks for agile-at-scale do not focus on the strategy, architecture and process of testing beyond the team unit. This often leads to failing the implementation of a quality assurance function. I have pointed out common pitfalls observed in real life, and suggested how to tackle them. And no, your project is not special ;-)