The Software Quality & Software Testing paradigms of Agile

Challenges in Agile Transformation for Software Quality and Testing

Many companies going through Agile transformation face challenges in adopting Agile models for software quality and testing. While the business impact of these elements is undeniable, their linkage to business values and Agile methodology applications is not always clear. This white paper summarizes some of my experiences working with leaders and executives of software product companies in their continuous improvement efforts toward business execution excellence.

Before diving into detailed discussions and problem statements, let's first define Software Quality and Software Testing, assuming the Agile definition from the Agile Manifesto, for clarity and alignment throughout this paper.

Software Quality

ISO-25010 defines the following eight product quality characteristics and 31 sub-characteristics:

1. Functional Suitability - Functional completeness, functional correctness, functional appropriateness.
2. Performance Efficiency - Time behavior, resource utilization, capacity.
3. Compatibility - Coexistence, interoperability.
4. Usability - Appropriateness recognizability, learnability, operability, user error protection, user interface aesthetics, accessibility.
5. Reliability - Maturity, availability, fault tolerance, recoverability.
6. Security - Confidentiality, integrity, non-repudiation, accountability, authenticity.
7. Maintainability - Modularity, reusability, analyzability, modifiability, testability.
8. Portability - Adaptability, installability, replaceability.

Software Testing

Software testing is part of the software quality validation and verification (V&V) activities throughout the SDLC process. There is no single definition for software testing, and rarely do we see any mapping of V&V activities to software quality characteristics. Software testing is also heavily dependent on the selected SDLC methodology. In this paper, we use software testing phases, terminologies, and categories based on industry best practices.

1. Unit Tests – Tests of a software unit (module, class, function).
2. Domain Logic Tests – Tests of business logic (also referred to as functional tests).
3. Integration Tests – Tests external dependencies and interfaces of a software unit.
4. Workflow Tests - Tests a series of business activities involving a few business logics and inter-component interfaces.
5. End-to-End Tests - Tests a complete system (including its external dependencies), also referred to as Regression Tests - End-to-End business functionality continuity tests.
6. UI Tests - Tests the system user interface.

There are many more types, models, and artifacts for software testing, but the above shortlist serves our further discussion.

Problem Statement

a) Software Quality is not explicitly mentioned in the Agile Manifesto or Agile principles. Does it have business value? If so, what is it and how is it measured?

b) Software testing V&V activities don't produce or add anything to the product. Does it have business value, or is it a waste in the process?

Paradigm Differences

Part of the reason for the different paradigms is the need to create explicit definitions for software quality and software testing business values. There is a lack of direct, simple linkage between quality, testing, and business values. The challenges of measuring software quality and the business efficiency of different testing phases leave space for varying interpretations, definitions, models, and methodologies in different organizations.

Software Quality Paradigms

To resolve the lack of explicit software quality business value definition in Agile terminology, a new "software iron triangle" has been introduced by a few Agile gurus. As the original iron triangle included scope, cost, and schedule as its vertices, the new "Agile triangle" has value (extrinsic quality), quality (implicit quality), and constraints (scope, cost, schedule) as its new vertices. However, the triangle representation is academic and may not convince any business leader to take action.

One common paradigm is that managers (business) focus on functional quality (scope/features) and tend to ignore other software quality characteristics. Non-functional requirements are often left to engineering teams, assuming these should be part of product engineering practices. This neglects the cost of these requirements.

As a result, organizations focus on functional features, creating a debt in all other quality characteristics. These characteristics are difficult to validate and verify, their impacts are longer-term, and they are harder and more expensive to fix. A backfire is inevitable, usually after a large-scale product failure in production, leading to major business value loss and high recovery costs.

Another paradigm is the "Bad Code" paradigm. Bad software engineering practices, caused by unprofessional engineering, fold under tight schedules and commitments. This leads to bad software code that grows into a mess, reducing productivity, requiring rework, redesign, and bug fixes, and increasing business loss. Software engineers need to be as passionate about "clean code" as managers are about deadlines and commitments. Continuous efficiency is achieved through good software engineering.

Software Testing Paradigms

On the testing front, testing activities are often perceived as non-productive. Organizations tend to reduce investments in testing, influenced by the following observations:

• Developers dislike testing (coding is seen as art, testing as pain).
• Managers often don't understand testing (seeing it as non-productive).
• Pressure for scope increases, leaving less time for testing phases.
• Test efforts are not included in estimations.

There is a lack of planning and investment in creating proper automation infrastructure and testing strategies as part of Agile transformation plans. Organizations focus on software development and product business value definitions, leaving the software testing transformation aside.

As a result, there are large variations in Agile testing transformations, methodologies, and implementations. Some companies end up leaving large parts of their software testing to their customers, causing organizational chaos around testing in the early phases of transformation.

Testing Strategies and Investments

We often see the "ice cream cone" pattern, where minimal time is spent on unit tests and most on end-to-end GUI tests, resulting in:

• High relative costs to fix software errors in later phases.
• Complex, fragile end-to-end tests with high maintenance costs and low ROI.
• Delayed feedback from end-to-end tests impacting development velocity and increasing business loss.

The "100% Testing" Paradigm

No software is 100% bug-free. Some managers and engineers get comfortable with having bugs, citing examples from successful companies like Google, Microsoft, and Apple. However, balancing the cost and effort in testing to achieve acceptable quality is crucial.

The Optimization Problem

The challenge is optimizing investments in SDLC activities to achieve the highest business value with the highest quality, lowest cost, and minimum time to satisfy customers. This requires iterative solutions, stakeholder interactions, continuous monitoring, and optimizations.

Recommendations

Here are some best practices to optimize delivery models and improve business results:

1. Improve alignment and understanding among stakeholders regarding the linkage of software quality, testing, and business value.
2. Ensure transparency and agreement on delivery process optimization efforts.
3. Handle non-functional quality requirements at the same level as functional requirements.
4. Invest in "inverting the test pyramid" – focus on unit tests, training, automation, and mapping quality characteristics to testing stages.
5. Adopt non-testing techniques for improving software design and coding quality.
6. Enhance quality-related retrospectives to focus on improving quality and reducing business value-related issues.
7. Implement quality effectiveness metrics to prioritize high-severity quality areas and reduce waste.

Summary

Software quality and testing have implicit business values with significant impacts. Becoming an Agile company involves delivering business value to customers with high velocity and quality. By improving software design, coding, and testing optimization techniques, companies can achieve execution acceleration and business excellence.

• V&V - Validation & Verification: Validation answers "are we building the right system?" and verification answers "are we building the system right?"

** Prevention is better than a cure - Examples: $17 billion for Samsung's battery management system bug, $18 million for NASA's Mariner 1 software bug, $60 million for AT&T's long-distance call software bug, $10 million for NYSE's 2015 downtime, and $300,000 per hour on downtime according to Gartner. Related Articles Agile Top 10 - Things you should know about Agile transition How to tune your Products SDLC to your business goals