The ExcElephant in the Room

First of all apologies for the pun – it’s terrible.

Now it has been well documented in the news how through the (mis)use of Excel in their data processing solution Public Health England (PHE) underreported around 16,000 COVID-19 cases. PHE has taken a bit of a pounding in the press and been ripped to shreds on LinkedIn by data professionals.

The use of Excel as an enterprise reporting tool or within in a data ecosystem has been the bane of my working career since the day I started my first job back in 2008 and here are some of the reasons why:

• I have seen lives lost to the upkeep of VBA macros written by someone who has disappeared into the ether without leaving documentation.
• I have had to manually move multiple excel files to prep for my ETLs because “that’s the way we’ve always done it”.
• I have seen critical reports with no auditability of changes, no data quality checks and no documentation/meta data of how the data is transformed within the 70-tab Excel rats’ nest.
• These reports have contained assumptions or hard coded logic from years ago that has led to incorrect information being reported for who knows how long.
• I have been on the receiving end of having no other choice but to accept csv/xls/xlsx as a data source for parts of my warehouse solutions that change shape more often than my stress ball when reading articles about Excel data clangers.
• “We don’t need a data warehouse or BI solution, we use Excel” arrgghhhhh

The list could go on and on and I think most data professionals will have their own list, but Excel isn’t the problem here, the problem lies elsewhere:

• In the lack of data literacy/data awareness of those who rely on Excel for much more than it was ever built for
• With data professionals who know better but don’t challenge or speak up against its misuse/over reliance
• With senior leaders who do not listen to those data professionals that do speak up.

Let’s be honest, if you have worked in the data field you will have used Excel for something you ideally wouldn’t have done the term ‘tactical solution’ springs to mind. I am guilty of this too, which is why you won’t catch me joining in with the PHE bashing.

We do this with the intention of putting a fully productionised solution in place once the dust has settled but as we know you don’t always get the time to go and fill the cracks you merely papered over during the delivery of your MVP. 

There are a number of high-profile excel based data ‘mishaps’ such as:

• In 2003, TransAlta lost $24 million due to Excel copy-and-paste error
• In 2005, Kodak suffered an $11 million severance error due to Excel typo
• In 2008, Barclays Capital was forced to spend millions on worthless contracts due to Excel reformatting error
• In 2010, MI5 bugged the wrong phones due to a spreadsheet formatting error
• In 2011, AstraZeneca accidentally released confidential information due to an Excel templating error
• In 2012, JP Morgan was hit with $6 billion trading loss due to Excel copy-and-paste error
• In 2012, London Olympics were publicly embarrassed by a simple Excel typo
• With PHE Track and Trace 2020 becoming the latest addition. 

I try and find the positive in these situations and to quote Sun Tzu, “in the midst of chaos, there is also opportunity”. So here is that opportunity.

These events become the aces up our sleeve …

When the time comes to replace that 'tactical solution', are you given the time and resource to do so? When you speak up against any future Excel based shortcuts in your data ecosystem, are you taken seriously? When you see departments and services that are in Excel reporting hell and you speak up and offer a solution that treats the data with the respect it deserves does the conversation go further than “how much will it cost?”

If the answer is no then pull that ace from out of your sleeve and slam it on the table for all to see.

Next time I challenge the business on the overreliance and misuse of Excel then I will be citing all of the above like a lawyer fighting for his client’s life. 

Keep fighting the good fight!