The Origins and Future of Tests as Specifications
So, I got a question from a colleague that I'm re-posted here for broader discussion. Slawomir Zborowski wrote:
"Recently some of our colleagues came up with obvious idea - to merge acceptance tests with specification. It sounds great but I guess they aren't the first that try to do that.
What is your opinion on this topic? Do you believe it is possible to achieve such a goal?
Do you know any team that succeeded with that? Or maybe there are existing open-source tools in the field? I've tried to find something but I failed. I also found some white papers related to such a task, but they are quite old (<2010) and it looks like they're just white papers, with no implementation.
My personal opinion (based on hunches frankly speaking) is that such a thing will be simple at the very beginning but after a while will snowball into huge and complex system that is hard to control. My understanding is that documentation/specs always require a mix of natural language and math expressions. The math part could be possibly easily transformable between different forms. The overwhelming part is NL part. I think it is clearly visible in frameworks like Robot.
So this was only what I think. I will appreciate your opinion :)"
I responded:
"Well, this idea certainly is something that comes up repeatedly. You can trace it back, in some sense, to ideas related to correctness proofs for code, which would try to prove that code satisfied the requirements. After all, people reasoned, code is just logic, and therefore one should be able to do proofs with it. For various reasons, this concept never made it very far.
The idea of 'test then code' stretches back to Beizer (at least), when he wrote about 'the threat of a test being enough' to convince developer's to write code to handle certain situations. Gelperin talks about 'test then code' as well, in his work on the V model.
In the Agile world, this idea of 'tests as specifications' shows up in ATDD, SBE, and BDD. Those are useful ideas, and, when coupled with effective automation, do support higher quality.
However, any such tests are, ultimately, a form of verification, demonstrating that stated requirements are met. Verification is useful, but you must also have validation, demonstrating fitness for use. A good way of thinking about this is a variation on the testing quadrants that I'm attaching here as an image."
So, what do you think? What interesting lessons about the future can we learn from the past concepts related to tests as specifications?
Product Line Manager at Nokia
7 年Tests as specifications? Sounds like bass ackwards logic: you need a premise and facts before you can draw a conclusion through logical tests using both things to be sure your statement and end result(s) match up.
Expressed how and at what unit of granularity? This idea has struck me many times, and as much as I always felt that as a tester I was the one stuck with "completing the specification" when I invented a bunch of tests I also knew there was no point trying to get anybody to stop and wait until we got all those behaviors identified. There is a good degree where the implied requirements that expand into a thousand fold or greater expression of the specified behaviors are a combination of "of course that should work... why mention it?" and "um... dunno, let's see what happens when we try it" Truth is, so many times the result of my test forced a redesign once people saw a concrete example of the true result of the product as designed. Not once did anybody re-write a specification. There wasn't time. This is pretty much the case for every tester I have worked with. I doubt this dynamic will ever change, the costs of the process are unwieldy to try to cover that completely in spec.
CEO at Qualizens S.A.C.
7 年I have been using ATDD and BDD for the recent years, doing manual verification and automating it. The main benefits i have found on doing ATDD is that it is a way to be more clear on the specifications with no assumptions because we work with examples in BDD, making questions each time and in natural language. When we worked a long time before with test case traceable to requirements, the team didn't make so many questions for better understanding that using now examples in BDD. Then the developers use the BDD examples to start developing and we are having better results that before in terms of a making a software product better aligned to business objectives and expectations.
The future of automated quality management
7 年I think BDD is a fad. It doesn't give transparency into how the product is driven, and the "specs" aren't really executable anyway. OTOH, I'm not sure about the value that Gherkin gives in terms of a standard language to loosely describe SUT behavior.
Philanthropist and Student
7 年The phrase "necessary but not sufficient" comes to mind in using the tests as specification. I'm remembering my time in military aerospace, where we had very thorough and formal 2-way traceability between the specifications and the tests. Even though we didn't use the tests as specs, the 2 artifacts had a very clear relationship and those tests were still not enough to thoroughly validate fitness for use.