So you skipped automated testing to move fast and break things. Now what?

Well you probably got what you asked for, broken things! Here is some Founder and CEO advice on how to recover from this all too common mistake.

Have you introduced a new product to the market and now have prospects trying it, or even better yet, have some paying customers? In an attempt to move fast, you decided to forego automated testing. As I discussed in “Why skipping automated testing doesn’t make you faster”, it is with the best of intentions that this mistake is made. But now you’re paying the price.

Are you afraid of making any changes to your product? Do you need antacids every time you do a demo? Do you dread opening your email or messaging app in the morning and reading compaints about your product? Is missing product deadlines delaying the closing of client deals? Are you losing opportunities or clients?

Now what?

It’s time to implement automated tests and get control of the development process (and be able to sleep at night). But where to start? My experience is that you want to start with high level ‘acceptance tests’ that validate the core value proposition. I call this:

driving the app down the center of the road

There are many types of tests, including: performance, usability, integration, unit and acceptance testing. They are all important, but I am going to focus on the last two: unit and acceptance.

Using a mechanical analogy, if we were testing a new vehicle, separate test stands for the engine and transmission would be ‘unit tests’. They test the component in isolation and are easier to control the inputs and measure the outputs. They also perform faster so you can get immediate feedback. They are especially useful for testing edge cases — all those exceptions that have to be dealt with.

Continuing the analogy, the ‘acceptance test’ is the track test for the vehicle. It is more of a ‘black box’ test, where the inputs are the human interface — in this case the driver’s seat. It validates that all the components are working together to get the expected results.

If you have already built your app and now realize that you need to add automated testing to regain control, I recommend a two pronged approach:

1. develop a first pass automated acceptance test
2. ensure that all future feature additions, fixes or changes have both unit tests and updated acceptance tests

The first pass automated acceptance test

Start with automated acceptance tests that focus on the core value proposition, i.e your pitch. The best place to start is the standard demo you give for your product. The demo you give to a new prospect or a potential investor. Developing these into automated acceptance tests now ensures that your demo will run, even after the latest changes to the app.

Be careful though. Automated tests done wrong can slow down development. Unfortunately, it is too easy to write brittle and interdependent tests that break on nearly every application change. With these type of tests, you can quickly end up spending more time fixing tests than developing functionality. But this is because the approach and methodology is wrong.

The biggest mistake is to make the test interdependent. I will give you an example of the classic mistake.

Lets say you have an app to record purchase receipts. The following test steps are an example of what not to do:

1. register a new user into the app
2. edit information about that new user (from 1)
3. add a new receipt for now edited new user (from 2)
4. edit the new receipt for the edited new user (from 3)

This may test the functions of adding a user, editing a user, adding a receipt and editing a receipt, but it is fragile. If the user registration in step 1 fails, allof the subsequent tests will fail. Furthermore, slight changes to the new user data could ripple through the other tests and cause failures.

The better approach is to pre-seed your test system with data and isolate each test. Re-doing the above tests properly would be:

1. register a new user into the app
2. edit information on an existing user in the test database
3. add a new receipt to an existing user — not the same user as in step 2
4. edit an existing receipt

You get the idea. Isolate the tests by using pre-seeded data for known and stable starting conditions for each test. And reset back to that condition after each test (rollback the transaction or reload the data).

Think of each independent test as a given-when-then scenario:

• given a known starting situation
• when some actions are taken or events occur
• then an expected response should occur

A good test database is also a good demo database, and vice versa. The more realistic the data, the better! (aside: It’s a good idea to have a mechanism to reset your demo database as well)

With the right approach, you can cover off the core features in this first pass. The first pass automated acceptance test should validate the expected behaviour if the app is ‘driven down the center of the road’ — that is, if the app is used as expected to get the most benefit.

This will not catch all bugs, but it should give you confidence that it is fundamentally working. My experience is that as long as the fundamentals are working, most people will give you some slack for the edge case bugs — but only for a time.

Now, moving forward.

Unit and acceptance test going forward

I recommend that you then implement automated unit and acceptance tests on all future additions, fixes and changes to the system. Whenever developers open up a part of the system, they should take the opportunity to add unit tests while they have their head wrapped around the functionality. As I have said in my previous post, its not the time it takes to write the code that is the bottleneck, it’s the thinking and design time that is the real effort.

As the test coverage grows, those pesky edge case bugs will be caught and you will gain more and more confidence in your app.

It will now be much easier and safer to bring new developers on board and to increase productivity and velocity. Major changes in both functionality and architecture can be made with confidence.

Any new features should automatically include acceptance criteria in the planning phase. Now your developers will really know when they are ‘done’ a feature — when the automated tests prove the acceptance criteria.

Now you can:

move fast and NOT break things