Testing Strategy – Finding Nemo err Glasses

Thanks to COVID-19, I am locked down in Chennai with my sister and brother-in-law. Day before yesterday evening around 8pm my b-in-law was watching “Mahabharath” and after sometime he said he could not find his glasses. All of us tried to search below the tables, in his bedroom, below pillows, below the cot, all the cupboards and could not find it. We gave up and decided to look for it next morning.

After b-in-law’s morning Pooja we were having breakfast and discussing where would be the missing glasses. I was asking where all he went after watching TV. He said he went to change his pants. After breakfast we narrowed our search near the clothes rack. There you go he found his glasses next to the rack beside an orange basket. None of us had searched that small gap instead we were searching all over the house since none of us would act or visualize like him. Though this was just an act of forgetfulness, I think there is a learning applicable to real world.

Testing is sometimes similar to the above scenario. Complexity is much larger and finding a bug is like casting the “fishing” net wide and deep within available resources constraints (to catch small and big fishes and some fishes do escape). Interviewing user is a key part of requirement gathering phase!

Part of the issue is the Gap between understanding of Real Users’ actions and scenarios and what the system designer captures as scope. This is where many systems start to fail. Typically in any organization those with Business Understanding do not have Technology knowledge and those who have Technology expertise do not have Business knowledge. You can notice that in-spite of 100s of years of technological breakthrough still simple Accounting systems fail in many cases. 

I started my business in 1991 with developing Financial Account Software. Then, I did not even know what was Debit / Credit. Thanks to my then room-mate Sanjeev Garg (a Chartered Accountant and now founder and MD of Indicaa group), I learnt Accounting principles and developed a product which matured in 1998. I was able to develop a complete Trade Finance software covering Import, Export, Loans, Deposits, Placement, Borrowing, Forex, Interest calculation all integrated to my Financial Accounting base and got it approved by Hong Kong Monetary Authority by Dec 1999 with one year full parallel run to meet the Y2k deadline.

I had no idea about Import or other functions. I used to sit next to the one of the users and observe what she was doing in their current system (which was mini-computer based solution without any documentation). With that knowledge from observing users, Interaction with another consultant friend Anil Kumar Muthiah (also was a Chartered Accountant) for brainstorming the design, and training a small team of 20 developers in India (who had neither banking knowledge nor coding knowledge (at that time I used a RAD tool named Magic Software from Israel), the full system was developed within six months and tested and corrected live through iterative cycles in another six months.

The topic here diverges from testing, my intention here is NOT political but just want to point out the huge impact of flaws in System Design. If you look at many of the accounting scandals repeatedly in-spite of best accounting organizations doing Audit. Sometimes I wonder even the scandals are designed by individuals (so called experts) in such organizations taking advantage of all the Accounting and Tax loop holes.

The Modus Operandi are all similar:

Shell Companies are established in different countries (using “binami” untraceable unrelated individuals) and invoices are “legally” billed to boost revenues systematically creating big numbers over a period of time. Loans are taken from banks showing boosted “Revenues” , Companies are established in “Low Tax” countries without much stringent audit controls and or transparency and one fine morning the owner vanishes announcing bankruptcy with Bank not having sufficient collateral (with financial assets moved somewhere else) to recover or unable to even prove there were accounting frauds. I am sure there will be many more explained in Panama Papers.

Governments and Public are always invariably at loss, with certain individuals unethically earning exorbitant income and stashing away the loot (common man even does not even know how many zeros in those accumulated wealth) and all the financial institutions and the audit organizations involved getting away with smaller or no fines. Some of those fraudsters start businesses again repeating the similar modus operandi in different names in different countries.

Similar things happen with Natural Resources ownership (Land, Mines and many more fall within this categories). In many countries this is quite normal for few wealthy families to own the entire natural resources (copper, diamond, or others) economy.

Wall-street and other investment bankers also play a big role in creating such anomalies. Most recent being explained in the documentary “the China Hustle” released in 2017. Though the firms involved were China based companies listed in NYSE. The Reverse China Merger Fraud was master-minded by some American investment banks with audit reports signed-off of by affiliates of global audit firms in China. Collectively individual US investors have lost billions. Similar cases have happened before in some of the Indian companies listed in NYSE/NASDAQ as early as 2003-2005 as well as most recently in 2018.

Slowly some of these System Design Flaws are being addressed with Data Transparency, AI, ML and international collaboration between Stock Exchanges and other initiatives. Typically System Design Flaws are very expensive as they involve structural changes at many levels and involves many parties and in many countries.

Coming back to Software Testing, typically testing team (whether internal or external) are limited by this. They can only test what they know and documented as base system. It is quite hard for testing team to visualize the real business user scenarios unless they have prior exposure to such systems from other project experience.

Though this may be arguable, my personal preference is to expose Developers and Testers to work with different end-users to observe how they currently work in real life projects and document as many actions as possible. Obviously it may not be practical to observe many projects and uncover all variations, but at-least 2 or 3 is bare minimum essential. It may also be feasible to video record such sessions to play back and see closely all the interactions, scenarios, key board actions. However in reality this may never happen.

Unlike Waterfall methodologies, in Agile Methodologies, Product is developed in iterations using Fail Fast approach and with continuous release, user scenarios or challenges faced in real life use are quickly addressed. Though this is a bit frustrating for the first few users, for organizations with limited resources to Design, Test, Develop, they can continuously improve the functionalities and features. However there is a threshold when to release such solutions to limit the frustrations. If released too early, users will give up and if released too late also the users may avoid using completely. Balancing is essential.

Off course best strategy in my opinion is to involve all Key Users, Designers, Testers and Developers, as early as possible to collaboratively capture as many features/functions and consciously, decide on the release priorities. With limited resources it may not be feasible to complete all deliverable and Prioritization is key here.

This is where the Industrial Engineering / Toyota Production System concept of Plan-Do-Check-Act, PDCA cycle is very relevant for any System Design and/or other Projects as well as the RAID Log (the Project planning tool) to address the Risks, Assumptions, Issues and Dependencies. Both requiring regular (weekly) reviewing and updating priorities. Establishing Trust and Transparency is key. Looking forward to hearing your comments.