How Does Behavior-Driven Development (BDD) Work In an Agile Setting? (Part 4)

In the prior post in this series, I explained that BDD tests are really integration tests, and how BDD tools can be used for most of the kinds of tests that are needed. In this fourth post on BDD, I will explain why your testing effort should not focus on the UI level—contrary to common practice among those who use BDD.

Always Define API Level Tests

Many application components have an exposed API, such as a REST or gRPC API. Such an API enables other components to access the component. A common design approach is to build an application as an assembly of components that interact via APIs.

There should always be regression tests that are maintained against those APIs. These APIs are not at a user story level, and do this is not behavioral testing. However, behavioral tools are very useful for such API tests. It is common practice to use unit testing tools for API tests, but I have found that test written that way are less visible and therefore less maintainable that if BDD tools are used, with the API test scenarios written in gherkin.

UI Level Tests Will Slow Down Your Whole Pipeline

In contrast to API level tests, BDD tests are typically at a user interface level: they simulate what a user would do. To accomplish this, they use special tools that enable them to control a browser, simulating clicks and so on. Such UI level run much more slowly than API level tests. If your tests are focused at the UI level, as BDD tests normally are, you will find that your test suite starts taking hours to run, and after six months or development it might be taking six or more hours. You can speed that up by running subsets of the tests in parallel, or by simulating a browser with tools like PhantomJS, but there are severe drawbacks to these approaches. The fundamental problem is that your testing is concentrated at the UI level, which is inefficient, and is actually pretty brittle as well, since layout changes to the UI require changes to the tests, even if the behavior has not changed.

A better approach is to architect your application so that there is a “business API layer” just under the UI. The UI should not contain business logic: all business functionality should be delegated by the UI to business objects. If that is the case, your tests can then be written against those business objects, instead of against the UI. If you do that, you will find again that the test specs cannot be written by the product owner—there will be too many nuances pertaining to the business objects. Thus, your analysts will need to write the test specs, and the analysts will need to be fairly technical, or they will need to work with test programmers to help them to get the specs right. That’s the reality of building complex applications today.

To be clear: You should still write UI level test; but they should be cursory. Your main functional testing should be at a business object layer. It is also important to perform exploratory tests against the UI, but that is best done in parallel with your Agile process—not as part of it. In other words, don’t hold up acceptance of a story because exploratory tests have not been done. Perhaps hold up deployment of that release until they are done, but don’t block stories on exploratory UI tests. Exploratory testing is something that should be going on all the time, against the release under development and also the current production release, and not tied to particular stories.

Are BDD Tests Acceptance Tests?

The phrase Acceptance Test-Driven Development (ATDD) is essentially a synonym for BDD. However, as I have explained, BDD tools can be used for many kinds of tests other than acceptance tests.

What are acceptance tests anyway? They can be anything: what makes them acceptance tests is that they are performed by the person who asked for the application—the Product Owner—or they are performed by someone the PO delegates to. But they can be functional, plus performance, plus security, whatever. In fact, they should be all those things.

One thing about acceptance tests though, is that they are almost always against the entire system. They are not story level. They are things like, “Make sure that a user can get such-and-such kind of job done”. At least, they should be, if they are done right. This corresponds to our “Use case level” tests, #3 in the list of test types in the prior post. Thus, to serve as acceptance tests, those tests should be written by someone other than the development team. Don’t be put off by the term “use case level”: we are not developing to use cases—we are merely acknowledging that there is a level of test scenario above the story level.

Now immediately you might think that I am bypassing the Agile approach of getting acceptance along the way, with continuous demonstration and PO acceptance of stories. Absolutely not: those are essential. But end-to-end type tests are really important. We don’t want them all coming at the end though, like in a waterfall project: we want them being created all along, and the team should have access to them so that the team can run them. This is an Agile version of what Quality Assurance can do: they can work alongside the team, and write these full use case level tests, adding those to the repo so that the team can run them. These tests are above the story level, so they are not driven by stories or by the sprint cycle: they are developed in parallel. They are not part of sprint story acceptance, but the PO should be seeing the results of them on a regular basis, and if some of them fail, or reveal shortcomings in the application, then new stories should be written to cover those gaps.

When Defects Are Found Downstream

One question you might have is, what happens when a defect is found as a result of an end-to-end test that is at a use case level, and not at a story acceptance criteria level? Or a defect is found during whole-system testing such as performance testing? The answer is that it depends:

1. If the defect can be traced to a particular story, and that story has not been completed:

You must ask yourself, is it a-lot of work to fix it and add some additional acceptance criteria to test for the problem? If not, then tell the PO and suggest that you just add the additional acceptance criteria and test scenarios and fix the story. If it is a-lot of extra work, then that should be explained to the PO and a new story created to cover the defect: the existing story is not changed and can be completed using its original acceptance criteria.

2. If the defect can be traced to a particular story, but that story has been completed:

Then the defect must be defined as a new story. Defects do not ever get applied to a completed story. Stories are not pieces of code that can have defects: stories are tasks that have acceptance criteria. If you find a defect that does not apply to a story that is currently being worked, then the defect must be made into one or more new stories.

3. If the defect cannot be traced to a particular story: it is a design defect or a gap in the application’s stories:

Then the defect should be made into a new story.

If you do a-lot of upstream testing at the story level, downstream defects will be very infrequent, but they will occur sometimes, and you need a process for dealing with them. The whole point of “shift-left” testing is to make downstream defects infrequent—not to pretend that they cannot exist.

Sharing Test Code Among Tests

If you have end-to-end tests that perform entire use-case level scenarios, you will soon realize that those use cases can be composed by performing the tests for stories in sequence. For example, a use case for purchasing a catalog item might be composable by performing a test from the login story, followed by a test from a catalog perusal story, followed by an item selection story, followed by a shopping cart transaction story. You won’t be able to just “call” those stories—they are not written that way—but you could potentially use much of the testing code, if it is written in a callable manner. The question is, should you re-use your story level testing code to assemble other kinds of tests?

The story level tests probably already share code: it is common practice to write reusable modules for, say, logging in, or making the entries on a given page. That is called the “page object model” approach. But should you share code across between different categories of test, such as story level tests and use case level end-to-end tests? It depends. Doing so risks that an error of assumptions in your story level tests is propagated to your use case level tests. However, you might feel that the risk is small since the use case level tests have received so much scrutiny. Then again, you might end up sharing a small, little used testing module—one that has not received a-lot of scrutiny. So it is a risk. You will have to decide this as there is no standard answer.

In the fifth and final part of this series, I’ll explore some of the process issues for BDD.