Testing Miracles

I started this note by writing on another subject entirely - The riveting topic of the way we focus on skill rather than talent when seeking out new staff and how we might redress this. However, while outlining that note a small bolt of lightning struck and this topic insisted I raise its profile. I have reflected in a somewhat light hearted, but hopefully still thought provoking way and offered a few suggestions for breaking the mould. 

Let’s get started: 

Self-Justification and claimed expert status bit coming up.

I have worked in the IT industry for well over 30 years and although I have been an architect, designer, project manager at times, but most of my experience comes from the discipline of test engineering (I do love it).  During that time I have worked in many industries and many projects in various stages of undress, and mostly in roles that have had specific focus (i.e. performance test this, automate that) and a more general involvement - manage the testing of these ’n’ phases. 

During a period of reflection I realised that there are a few topics that have come into sharp focus each and every time I have had an initial conversation with the relevant CIO / CTO / Head of department / Project manager/ Team leader upon introduction to the project/programme/department.

The gist of the conversations is: This is a very challenging department/project/role and we are not very mature in this area and our timescale is tight and our budget (especially for QA/testing) is weak and our resource pool is poor and, and , and, and, and. At some point the classic KPI topic joins the conversation - always an amusing introduction in these circumstances from my perspective.

This represents an attempt (genuine I have always assumed) to set the scene and perhaps limit my expectations and perhaps my aspirations. For my part I am often left to ask focussed questions on the precise nature of the limitations to try and work out exactly how much trouble really exists. 

The problem with all of this is: my brain has in reality already slumped in a small heap in the corner crying and asking itself why I am here - they don’t need me, they need Jesus Christ or possibly God herself!

When I first started to take a more directional role in projects, we (this is not my invention) had a rule of thumb 40:20:40 for planning purposes - 40% thinking; 20% doing; 40% checking. Nearly 40 years later I have not seen this rule (despite all the fantastic ideas and methods now available for misuse and abuse) broken - it’s still valid - check your spend (not your budget, but your actual spend when the job is done and then tell me I am wrong - and please remember using in service support desk to fix your development problems is cheating and more expensive).

So you can imagine how delighted I am each time I join yet another project where it’s not in a good state and they only have ￡3 as a testing budget and have already committed ￡1.50 on the celebration party.

In many cases the debate actually starts from the perspective of not understanding that testing is needed. Really? Yes really! I have attempted to work with several Heads of Development Departments who think testing is an optional add on and not really required. True, I assure you!

 I am often left wondering why I was invited to join the conversation in the first place - I am obviously aware of the concept of budget, but really! Is that all we can discuss? I am not sure if it is realised that sometimes the best you can do is to acknowledge that you will overspend and seek additional funding in order to make the planning/management activities sensible and achievable. If this fails - stop! Stopping and checking the business value is still in positive balance is a great way to avoid the use of blind optimism as your basic model for success and go on to overspend by an astronomical amount (or roughly the same as the average government run IT project).

 Brain Surgery or simple stuff - slight change of direction:

 When you go to see a brain surgeon you start with the presumption that he is the expert and knows his stuff. He/she will perform some examinations, work out what the situation is, and provide a prognosis and a diagnosis. You don’t tend to challenge it because that’s why you went to see them - they know their stuff.

But with the topic of testing and despite the fact that I am invited in to discuss the topic because I am supposed to be an expert, I find that I am in discussion with people who are also experts despite having no experience in the domain at all (because testing is easy right?).

Wrong! I It would be much more constructive and much less expensive for all concerned if I was not given the required answer at the same time as having the problem outlined.

Still in use today:

I cannot recall all the lunatic ideas I have witnessed for reducing the cost of testing and still bringing the project in on time - there are simply too many but what follows is a summary of the most frequent in my experience.

Theories for reducing the need for expensive testing activities include:

• We have spent more time and effort in design so there will be fewer defects
• We have unit tested everything and have a CI environment in the cloud (which is of course shared and/or not available 24*7)
• We have been very careful; our developers are very good
• It’s all plug and play
• Our contracts with our suppliers dictate that testing is their responsibility - we will not pay for defects.
• We don’t have the time to test it, so our development model is optimised to minimise the need.

Project practice/measures to reduce the testing budget include: 

• Sharing the main test environment between security testing; performance testing and UAT in both time and space.
• Attempting to perform development and test activities in the same environment.
• Eating all the time allocated for testing with debugging the software and adding new/late features and often starting the activity after it is supposed to finish - leaving no available time - so ship anyway.
• Employing low skilled, low grade resources late in an attempt to reduce spend. That one always works well.
• Using developers to test their own work. My personal favourite. I have an anecdote which I will share another time - explains all!
• Deciding not to test (well not until we get to production anyway).

None of these work. I repeat - They do not work!

I can explain in detail if you like - but I you will need to pay me.

That is, they don’t work if your objective was to deliver all the functionality on time and in budget and be able to (with all sincerity) make confident statements borne out by post-delivery performance that your product works as intended.

If you are delivering in another planetary system then ok, they may work but nobody will understand you.

Next time try these - Here are some things you need to do to deliver an adequately tested product/project cost effectively (No charge for this):

1. Determine your prevailing standards and feed them into your planning - so do this first.
2. Do pick a project control and development control method and apply them with rigour.
3. Do hold all disciplines of the project (analysis/design/implementation etc.) to account on timescale, budget and quality. Any debt here is never recovered.
4. Do hire at least one test professional at the beginning of your project to keep you sane and in reality mode.
5. Do assume that you will spend 40% of your budget on testing or quality related activities - if you can’t afford this - you probably cannot afford to do the project

Things to do if you find yourself running a project in a testing mess and are facing a perspective saviour in conversation about miracle execution: 

• If he/she smiles and says no problem and is prepared to sign up to almost any condition you are imposing - End the conversation immediately. This person is certainly not Jesus Christ and miracles don’t happen. They are just another politically astute career engineer who will only ever tell you what they think you want to hear. Or they are an idiot. Either way you don’t need them - that’s how you arrived here in the first place.
• If they challenge you about the logic of some of the sentences and attempt to push and stretch the boundaries of your initial scene setting - consider continuing
• If they offer possible models for testing delivery and explain the pitfalls and benefits (especially the impact on spend and including the classic JFDI approach) then consider continuing
• If they advise directly within the conversation that at least one of the things you are doing is nuts and needs to stop now then this might just be the person you need. Consider hiring. 

If you think I am wrong in any of this, that’s absolutely fine - please tell me where you last saw Jesus so I can go and find him for my next testing miracle.

Post Script - Its nearly Christmas and some good friends have suggested that it might not be good for business to suggest your potential customer is an idiot. I agree completely. I think it might be better to focus on the prime characteristics required from good test practitioners, which must include honesty and integrity as well as talent, tact and discretion. Business will run more efficiently if it doesn't waste time and effort travelling the wrong road. Lets make a difference in 2017 please