The Agile tester

Ever wondered what makes a Tester an 'Agile' Tester? Acting as a professional test engineer within an Agile Development environment for many years, I recently took the 'ISTQB Certified Agile Tester' certification examination with the International Software Quality Institute (iSQI), I will try to explain what it takes to be one. First of all I will give a short introduction to agile development and then put more focus on the role of the Agile Tester.

What is 'Agile' Software Development?

In 2001, the most widely used lightweight software development methodologies were grouped in the Agile Software Development Manifesto. The Agile Manifesto contains four statements of values:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

The Agile Manifesto argues that although the concepts on the right have value, those on the left have greater value. 

In an Agile development project, it is important to involve everyone with the knowledge and skills necessary to ensure project success in one team. The team should be relatively small and share the same workspace. This is referred to as the 'whole-team approach' and promotes more effective and efficient team dynamics.

Agile projects have short iterations enabling the project team to receive early and continuous feedback on product quality throughout the development lifecycle.

There are several Agile approaches, each of which implements the values and principles of the Agile Manifesto in different ways. To me the most used are: Extreme Programming (XP), Scrum, and Kanban. I will not go into the detail of explaining the differences between these 3 appraoches, but most approaches practice collaborative user story creation, retrospectives, continuous integration, and planning for each iteration as well as for overall release. 

Typical testing activities

Since test activities are related to development activities, they will vary in different lifecycles. Testers must understand that testing in an Agile development environment differs from testing in any other environment. The ability to adapt to the context of a given project is a key success factor for testers. 

One of the main differences with traditional lifecycles is that Agile lifecycles work with iterations. At the beginning of the project, there is a release planning, followed by a sequence of iterations. These iterations are highly dynamic, with development, integration, and testing activities taking place throughout each iteration, and with considerable parallelism and overlap. Testing activities occur throughout the iteration, not as a final activity.

Testers, developers, and stakeholders all have a role in testing, as with traditional lifecycles. Developers perform unit tests as they develop features from the user stories. Testers then test those features. Stakeholders also test the stories during implementation. Stakeholders might use written test cases, but they also might simply experiment with and use the feature in order to provide fast feedback to the development team. Testers may also serve as testing and quality coaches within the team, sharing testing knowledge and supporting quality assurance work within the team. This promotes a sense of collective ownership of quality of the product. 

 Test automation at all levels is favored in Agile teams, and this means that testers spend time creating, executing, monitoring, and maintaining automated tests and results. This leads to a tendency for Agile teams to favor testers with a strong technical and test automation background. Because of the high rate of test automation, a big chunk of the manual testing in Agile projects tends to be done using experience-based and defect-based testing techniques.  

 A core Agile principle is that change may occur throughout the project. Changes to existing features have testing implications, especially regression testing implications. The use of automated testing is one way of managing the amount of test effort associated with change. However, it’s important that the rate of change not exceed the project team’s ability to deal with the risks associated with those changes.

The daily stand-up meeting includes all members of the Agile team including testers. 

The testing work products

Agile testers use typically the fillowing categories of work products:

• Business-oriented work products that describe what is needed and how to use it. Typicaly these include user stories and acceptance criteria. User stories are the Agile form of requirements and should define a feature
• Development work products that describe how the system and typicall include the actual source code.
• Test work products that describe how the system is tested, that actually test the system, or that present test results. Typicaly these include automated tests, as well as documents such as test plans, manual tests, defect reports, and test results. The data is captured as lightweight as possible.

In an Agile project, it is a common practice to avoid producing vast amounts of documentation. Instead, focus is more on having working software, together with automated tests that demonstrate conformance to requirements. In a successful Agile project, a balance is found between efficiency and providing sufficient documentation to support business, testing and development.

Teams may use other metrics similar to those captured in traditional development methodologies, such as test pass/fail rates, defect discovery rates, confirmation and regression test results, defect density, defects found and fixed, requirements coverage and code coverage to improve the product quality.

To provide an instant, detailed visual representation of the whole team’s current status, including the status of testing, teams may use Agile task boards. A testing column and testing tasks can be used for user stories to achieve a 'done' status.

Agile testing methods and test levels

Agile testers aim to introduce testing practices early. Testers in Agile projects play a key role in guiding the use of these testing practices throughout the lifecycle.  

Test-driven development, acceptance test-driven development, and behavior-driven development are three complementary techniques in use among Agile teams to carry out testing across the various test levels (tests are defined before the code is written).

A software system may be tested at different levels. Typical test levels are, from the base of the pyramid to the top, unit, integration, system, and acceptance. The test pyramid emphasizes having a large number of tests at the lower levels (bottom of the pyramid) and, as development moves to the upper levels, the number of tests decreases (top of the pyramid).

• Unit test level, typically done by the developer
• Integration and system test level, which starts once the first user story is ready for such testing. This can involve executing functional tests, as well as non-functional tests for performance, reliability, usability, and other relevant test types.
• Acceptance or UI test level. 

Testing quadrants align the test levels with the appropriate test types in the Agile methodology. The testing quadrants model, and its variants, helps to ensure that all important test types and test levels are included in the development lifecycle. This model also provides a way to differentiate and describe the types of tests to all stakeholders, including developers, testers, and business representatives. 

Testing and Configuration Management

Within Agile projects, developers use tools for static analysis, unit testing, and code coverage. Automated build and test frameworks allow the continuous integration of new software with the system. These automated tests can also include functional tests at the integration and system levels. The goal of the automated tests is to confirm that the build is functional, and if any automated test fails, the team should fix the underlying defect asap. For this to work, test reports have to be avaiable as soon as the tests have run.

Organizational choices in an Agile team

Agile organizations may encounter some test-related organizational risks:

• Testers work so closely to developers that they lose the appropriate tester mindset.
• Testers become tolerant of or silent about inefficient, ineffective, or low-quality practices within the team.
• Testers cannot keep pace with the incoming changes in time-constrained iterations.

In general, independent testers are often more effective at finding defects. So in Agile teams there is a risk of loss of independence and objective evaluation if the tester is per definition part of the team. A good way to tackle this problem, is to have separate test team where testers are assigned to Agile teams on a long-term basis, at the beginning of the project, allowing them to maintain their independence while gaining a good understanding of the product and strong relationships with other team members. In addition, the independent test team can work on long-term activities, such as developing automated test tools, carrying out non-functional testing and tests that might not fit well within a sprint (e.g., system integration testing).

Managing regression risk

Regression testing occurs throughout the iteration, re-running the preferably automated tests via a continuous integration framework.

Change in welcomed in agile development, and as a concequence manual and automated tests must be kept up to date to deal effectively with regression risk. In an Agile project, as each iteration completes, the product grows. Therefore, the scope of testing also increases. Agile testers have to validate the code changes made in the current iteration, but at the same time also need to verify no regression has been introduced on features that were developed and tested in previous iterations. The risk of introducing regression in Agile development is high due to a high code churn. In order to maintain velocity, it is critical that teams invest in test automation at all test levels as early as possible. in addition, the use of test automation, at all test levels, allows Agile teams to provide rapid feedback on product quality

It is also important that all test assets such as automated tests, manual test cases and other testing entities are kept up-to-date with each iteration. Make sure to use version control, to ensure preserving the historic information of the test assets. 

Automated tests can be run continuously against the system. And an initial subset of automated tests (smoketest) can be run to cover critical system functionality, so teams don’t waste time testing an unstable build.

The role and skills of a tester

In an Agile team, testers must closely collaborate with all other team members and with business stakeholders. This has a number of implications in terms of the skills a tester must have and the activities they perform within an Agile team.

A tester in an Agile team should, apart from the skill every tester needs, be competent in test automation, test-driven development, black-box testing and experience-based testing. As Agile methodologies depend heavily on collaboration and interaction between the team members, testers in an Agile team should have good interpersonal skills. 

Testers in Agile teams should be positive and solution-oriented, display critical and quality-oriented thinking, actively acquire information, effectively define testable acceptance criteria, respond to change quickly, plan and organize their own work 

The role of a tester in an Agile team is to generate and provide feedback on all quality levels. An agile tester is actively collaborating with developers and business stakeholders to clarify requirements, especially in terms of testability, consistency, and completenes. He participates proactively in team retrospectives, suggesting and implementing improvements.

 Within an Agile team, each team member is responsible for product quality and plays a role in performing test-related tasks.  

 Assessing Quality Risks

A typical objective of testing in all projects, Agile or traditional, is to reduce the risk of product quality problems to an acceptable level prior to release. Testers in Agile projects can use the same types of techniques used in traditional projects to identify risks, assess the associated level of risk, estimate the effort required to reduce those risks sufficiently, and then mitigate those risks through test design, implementation, and execution. However, given the short iterations and rate of change in Agile projects, some adaptations of those techniques are required.

In Agile projects, quality risk analysis takes place at two places.  

• Release planning: business representatives who know the features in the release provide a high-level overview of the risks, and the whole team, including the tester, may assist in the risk identification and assessment.
• Iteration planning: the whole team identifies and assesses the quality risks.

An iteration starts with iteration planning, which culminates in estimated tasks on a task board. These tasks can be prioritized in part based on the level of quality risk associated with them. Tasks associated with higher risks should start earlier and involve more testing effort. Tasks associated with lower risks should start later and involve less testing effort.

Quality risk analysis during iteration planning:

1. Gather the Agile team members together, including the tester(s)
2. List all the backlog items for the current iteration
3. Identify the quality risks associated with each item
4. Assess each identified risk -> categorizing the risk and determining its level of risk based on the impact and the likelihood of defects
5. Determine the extent of testing proportional to the level of risk.
6. Select the appropriate test technique(s) to mitigate each risk.

The tester then designs, implements, and executes tests to mitigate the risks.

Agile testing techniques

Many of the test techniques and testing levels that apply to traditional projects can also be applied to Agile projects. However, for Agile projects, there are some specific considerations and variances in test techniques, terminologies, and documentation that should be considered.

Agile projects outline initial requirements as user stories in a prioritized backlog at the start of the project. The user stories serve as an important test basis. During each iteration, developers create code which implements the functions and features described in the user stories, and this code is verified and validated via acceptance testing.

• Acceptance Test Driven Development (ATDD): ATDD focuses on involving team members with different perspectives such as the customer, developer, and tester. Three Amigos meetings are held to formulate acceptance tests incorporating perspectives of the customer, development, and testing. The customer is focused on the problem that is to be solved, the development is focused on how the problem will be solved whereas the testing is focused on what could go wrong. The first step is to specify the user story, the next step is to create the tests. The tests are examples that describe the specific characteristics of the user story and these examples will help the team implement the user story correctly. Since examples and tests are the same, these terms are often used interchangeably. Typically, the first tests are the positive tests, confirming the correct behavior without exception or error conditions, comprising the sequence of activities executed if everything goes as expected. After the positive path tests are done, the team should write negative path tests and cover non-functional attributes as well. Tests are expressed in a way that every stakeholder is able to understand.
• Behavior Driven Development (BDD): BDD improves communication amongst project stakeholders so that all members correctly understand each feature before the development process starts. There is continuous example-based communication between developers, testers, and business analysts. The examples are called Scenarios which are written in a special format called Gherkin Given/When/Then syntax. The scenarios hold information on how a given feature should behave in different situations with different input parameters. These are called “Executable Specifications” as it comprises of both specification and inputs to the automated tests.
• Functional and Non-Functional Black Box Test Design: In Agile testing, creating tests and programming activities occur concurrently. Just as the developers are programming based on the user stories and acceptance criteria, so are the testers creating tests based on user stories and their acceptance criteria. Testers can apply traditional black box test design techniques such as equivalence partitioning, boundary value analysis, decision tables, and state transition testing to create these tests.
• Exploratory Testing and Agile Testing: Exploratory testing is important in Agile projects due to the limited time available for test analysis and the limited details of the user stories. In exploratory testing, test design and test execution occur at the same time. During test execution, the tester uses creativity, intuition, cognition, and skill to find possible problems with the product. The tester also needs to have good knowledge and understanding of the software under test, the business domain, how the software is used, and how to determine when the system fails.

Tools in Agile Projects

• Task Management and Tracking Tools: In some cases, Agile teams use physical story boards (e.g., whiteboard) to manage and track user stories, tests, and other tasks throughout each sprint. Other teams will use application lifecycle management and task management software. This tio visualy represent the current state of each user story, the iteration, and the release, allowing all stakeholders to quickly check status.
• Communication and Information Sharing Tools: In addition to e-mail, documents, and verbal communication, Agile teams often use wikis to build and share an online knowledge base on various aspects of the project. These tools should be used to complement and extend, not replace, face-to-face communication in Agile teams. 
• Software Build and Distribution Tools: Daily build and deployment of software is a key practice in Agile teams. This requires the use of continuous integration tools and build distribution tools.
• Configuration Management Tools: These tools may be used not only to store source code and automated tests, but manual tests and other test work products are often stored in the same repository as the product source code. This provides traceability between which versions of the software were tested with which particular versions of the tests, and allows for rapid change without losing historical information.
• Test Design, Implementation, and Execution Tools: Use of tools such as mind maps have become more popular to quickly design and define tests for a new feature.
• Exploratory test tools: Tools that capture and log activities performed on an application during an exploratory test session are beneficial to the tester and developer, as they record the actions taken. This is useful when a defect is found, as the actions taken before the failure occurred have been captured and can be used to report the defect to the developers.
• Cloud Computing and Virtualization Tools: Virtualization allows a single physical resource (server) to operate as many separate, smaller resources. When virtual machines or cloud instances are used, teams have a greater number of servers available to them for development and testing. This can help to avoid delays associated with waiting for physical servers. Creating a new server or restoring a server is more efficient with snapshot capabilities built into most virtualization tools.

Conclusion

A tester on an Agile project will work differently than one working on a traditional project. Testers must understand the values and principles that underpin Agile projects, and how testers are an integral part of a whole-team approach together with developers and business representatives. The members in an Agile project communicate with each other early and frequently, which helps with removing defects early and developing a quality product.

Principles of Agile Testing

• Testing is continuous: Agile team tests continuously because it is the only way to ensure continuous progress of the product.
• Continuous feedback: Agile testing provides feedback on an ongoing basis and this is how your product meets the business needs.
• Tests performed by the whole team: The developers and the business analysts also join in testing the product.
• Decrease time of feedback response: The business team is involved in each iteration in agile testing & continuous feedback shortens the time of feedback response.
• Simplified & clean code: All the defects which are raised by the agile team are fixed within the same iteration and it helps in keeping the code clean and simplified.
• Test Driven: In agile methods, testing is performed at the time of implementation whereas, in the traditional process, the testing is performed after implementation.
• Every iteration has its own testing phase. The regression tests can be run every time new functions or logic are released.
• Testers and developers work closely together.

References

• ISTQB Foundation Level Agile Tester Overview, Version 1.0
• ISTQB Foundation Level Syllabus, Version 2011
• ISTQB Foundation Level Extension Syllabus Agile Tester 2014