How to expose fake UI test automation in fake Agile companies?

Over the last decade, many software projects have claimed to do Agile/DevOps, with some degrees of UI Test Automation and Continuous Testing. The reality is, most were faking it. Michael Feathers, a renowned author and agile expert, wrote an insightful post (2009), in which, he said: "everyone does" (faking UI test automation). My view is: "nearly everyone fakes it". The situation has not changed since, only with more lies.

Some might wonder, how could that be? "Our company has a dedicated DevOps team, dedicated agile coaches"; Or "our project was out-sourced to an agile software service company, they even showed us good-looking charts, test reports", ..., etc.

First of all, the chance of finding a real Agile company (doing real UI test automation) is very slim. Don't just take my word for it. Check out this article on the most recent World Quality Report: 5 Reasons Enterprise Test Automation Is So Challenging. "on Global 2000 enterprises, a dismal 8% have end-to-end functional tests". Please note, this is just "have". It is far from comprehensive, not guaranteed high-pass rates, not run daily, not if-pass-all-deploy-to-production. It would be safe to say, only < 1% of that 8% can achieve the goal.

Does it really mean that those so-called Agile Consulting are lying? Statistically, yes. Here I will provide an alternative perspective. Assuming the budget for your application development is $1 million. If the agile software company really has the knowledge and process to do the real UI test automation and do it properly, the quote will be reduced to $200,000. Yes, 5 times cheaper. Why? because UI test automation will greatly increase development efficiency, typically 5x to 10x. Of course, few have this knowledge.

Why do you want to do Agile? Certainly not for all sorts of charts, right? You want to be able to release high-quality software frequently. Yes, that's real Agile, end-2-end automated regression testing makes "Release early, release often" possible, together with many other benefits. I have developed feature-rich ClinicWise, WhenWise, SiteWise web apps (plus international award-winning BuildWise server app, TestWise desktop app) mostly in my spare time, and I can still maintain all of them, thanks to the Continuous Testing: executing automated end-2-end Selenium/Appium tests regularly.

Agile software delivery companies charge clients for the development time. If they do real Continuous Testing (running UI tests multiple times a day), this is a big IF, it will conflict with its commercial interests.

How fake agile software companies fake their UI test automation capabilities?

1. Showcase a perfect demo project

Once I was helping a friend (a PM) on advising an out-sourced project. The vendor was a small service delivery company who claimed that they had a good process of CI/CD with UI test automation. I spotted the problems immediately with their showcase project. The tests were too simple, and the scripts were not flexible. Not long after, we have seen many rudimentary bugs made to the test server, passing their so-called 'robust' CI/CD process.

2. Record test scripts afterwards

The majority of automated UI functional testing efforts are on maintenance. Creating test scripts only took about 10% of your efforts. Test scripting maintenance, when the applications keep changing, requires high-level skills, efficiency and dedication. The reward for that effort is great. check out this Facebook's presentation), However, many never had experienced it. Consequently, they regard test script maintenance as a burden. Hence, a common trick to fake test automation is, after the software is nearly ready, to create a set of automated tests with a recording tool (such as Cypress recorder).

Those automated scripts have little value. They

• did not provide valuable feedback when a feature is implemented
• did not detect regression errors (early detection can greatly reduce development/debugging efforts)
• have no use after delivery, as the scripts were not refined continuously

Another common trick is 'swapping the concept' by using API (some with the fancier name: microservice) testing as 'end-2-end testing'. This is absolutely wrong. As the World Quality report 2018-19 shows: “end-user satisfaction” is the top objective of quality assurance and software testing strategy.

Alert keywords: UFT, Cypress, Cucumber, SpecFlow, Jenkins, Bamboo. When these frameworks/tools are used in the context of UI Test Automation and Continuous Testing, be aware. I have worked in this field for 14 years. I have witnessed that every project that has used one of these technologies failed. According to Michael Feather's observations (11 years ago), this is not news. 

Now here comes the simple way to expose those lies. In the service contract, try to add this clause in:

THE END-2-END FUNCTIONAL UI TESTING MUST BE CONDUCTED AT LEAST TWICE A DAY. THE TEST RESULTS MUST BE TRANSPARENT.

90+% of USER STORIES MUST HAVE END-2-END AUTOMATED TESTS.

Reports like the ones below:

for each CT Build, full test results (and test script content) are displayed. Fixing the test failures is the team's top priority (refer to Kent Beck's famous Agile book: Extreme Programming Explained: Embrace Change).

Those 'agile' companies shall not object to the above, as that's what they advertised. But they will be very upset, you put them into an objectively-measurable cause. They would find it hard to say NO. Any excuses would put you into a much better position to negotiate.

Lastly, one real-life case. This is a government project out-sourced to a medium-sized software company. The PM saw how we did Continuous Testing (we located on the same floor), he then asked the company to add automated tests as a part of delivery (like my project). The company's answer: "It will cost extra, double current the contract amount". I visited this company later, they did not have the test automation capability at all.