Cow Eggs are Bigger than Chicken Eggs

Avoiding disastrous data-based decisions

This was the response Chat GPT provided when asked by Nicholas Davis “are chicken eggs bigger than cow eggs?” and posted the results on his LinkedIn feed earlier this year.

Taken at face value it’s a reasonable question and a reasonable response, and Chat GPT proffered all sorts of additional nuggets of information about the relative characteristics of chicken and cow eggs to make the response seem extremely plausible and useful.

We call laugh at this because even a five-year-old child probably knows this is nonsense, but the very serious warning here is evident: Chat GPT is not “artificial intelligence” and not a substitute for human critical thinking (Chat GPT is a large language model (LLM) which is a type of machine learning (ML) using natural language processing (NLP) – and if you don’t know the difference between these you had better educate yourself!).

In this case the warning on the packing label isn’t needed, but what if we were trying to find evidence supporting a business decision about a new technology investment, or to help find a solution to an internal process failure??

We are deluged with data and time poor, so its easy to assume that “what we see” is the situation or problem, especially when it’s from a source we believe is credible, rather than recognising this as a visible symptom of a deeper ‘why’.? Data to support evidence-based decisions is critical, but the adage “garbage in, garbage out” holds true now more than ever.? Mistakes happen when we stop being critical thinkers and accept what we are told at face value, without question.

We cannot outsource our intelligence

Analytical thinking and creative thinking remain the most important skills for workers in 2023…. reflecting the increasing importance of complex problem-solving in the workplace[i]

These skills are not just for technical experts in your organisation, as leaders we must resist the temptation to accept data or ‘facts’ as presented and strengthen our own critical and analytical capabilities.? Seek another perspective, never assume you know or can foresee everything, and be comfortable with a degree of ambiguity and complexity in the situation. ?

A regenerative organisation uses data insights to illuminate issues and opportunities, as well as to inform and evolve the board and executive's strategic decision-making frame of reference. When evaluating the worth of business data and analytics, we must ask:

1. How does the metric inform decisions or actions?
2. Is it a lagging or leading indicator of the situation?
3. Does it tell you ‘what’ is going on inside the organisation but not provide any indication of ‘why’?
4. Are there data gaps that might help us understand likely causes, not just see the effects?

However, its often easier to rely on flashy dashboards and ‘expert’ consultants to tell us what the problem is.? Over the past few decades multinational corporations, government and non-governmental organisations (NGOs) have increasingly relied on consultants to construct a picture of the world for them, tell them how to change it, and then apply the 'solution' to the problem as they have defined it.? Organisations have gradually allowed internal critical thinking and analytical capabilities to be eroded[i] at a time where we most need them to cut through the hype, data deluge and slick research reports to make sense of our world and inform sensible decisions.

Some statistics are more equal than others

Dr Craig Knight called out the danger of overreliance on the truths, trends and insights that characterise most business research in a recent article; a scathingly well-informed and robust review of a global employee survey that concluded that most management research is at best “dreadful” (I’d thoroughly recommend reading his full post, not only is it pretty funny, it’s highly instructive and revealing as to the dangers lurking in the shallow waters of pithy insights, attention-grabbing headline numbers and quotable statements of ‘fact’).?

My take on the report in question (again just using this as an example) was that while it was a solid sample responding to (mostly) well-worded questions that have been refined over many years of use, the analysis falls far short of where an organisation of this size and reputation could have taken it.? Add to that, like most consultant-generated research, the authors haven’t referenced third party supporting or contradicting sources (of which there are many), or indeed any other substantive evidence, preferring to assume their data exists in a perfect vacuum of absolute truths. ?

For example, take the statement from the report “Employees value the office to focus on their work, meet their colleagues, and access critical tools and people. This signals a shift from the office as an important place to rebuild social capital during the pandemic, to a place where employees can simply get their work done alone and with their colleagues.[ii]”?

Radical thought hey, that the office is now a place to get work done!? This calls for a complete rethink of workplace strategy and a redesign of our offices, aided of course by the architectural and consulting firm who authored the report.

Looks can be deceiving

I am not suggesting this report is ‘wrong’, although it is biased in perspective, and we don’t know how statistically significant the results are because they appear to be based primarily on percentage results of equally weighted single item answers rather than recognise the complexity, ambiguity, and interrelatedness of work performance characteristics with slightly more advanced analysis techniques.

The report makes a great bedtime read but like too much management research lull’s leaders into a false sense of security – that the experts out there have the answers, the keys to unlock and solve our wicked problems. ?

However, it’s not terribly useful as a contributor to informed executive decision-making because it doesn't get to the heart of its core theme - why some organisations appear to be more successful in supporting employee performance in a ‘hybrid’ workplace than others.?Identifying and addressing a problem effectively requires us to understand what’s causing it - digging in to the ‘why’ - because if we don’t know with a reasonable degree of probability what’s going on, our solutions will be ineffective at best or incompetent at worst.

Most people don’t have a degree in statistics or are fortunate like me to have a PhD.? What I learned though in the process of doing the PhD was that not all statistics are created equal, and not all patterns imply a meaningful relationship. ?I was able to create some beautiful graphs plotting organisational performance against various attributes and characteristics that had highly satisfying statistical metrics – except that like the cow egg, they were completely nonsensical.? Garbage in, garbage out.? It turned out that there’s no neat causal explanation, but I was able to discover some important influential relationships that affected performance (at least in my sample) and had to be satisfied with that.?

Channel your inner child – ask questions

In a complex, ambiguous, and uncertain business environment, getting a handle on ‘why’ is no easy task, and like measuring productivity in knowledge workers, is pretty much impossible.? We don’t need an understanding of statistics or a doctorate to bring a healthy dose of questioning to the world around us.? A three-year-old can be highly irritating in her inability to accept the world around her at face value - for every final response to “but why is it like that?” we get yet another, more insistent “but why?”

We abrogate our responsibility as decision makers if we do not take the time to question, and to think critically about what we know and don’t know about the business environment.?

Asking a few simple questions can make all the difference:

1. What is the source of the data and is it representative of what it claims to be, for example Gen Z, Singapore office workers, healthcare industry employee??
2. Is it comparable to the context or challenge under consideration?
3. What’s the motivation of the data or research owner?? Do they have a vested interest in a certain worldview on the situation?
4. Is there more than one source of data we can use? Do conflicting interpretations exist?
5. How was the data analysed, is it simple averages and absolute before/after changes, or does it leverage advanced analytics or modelling to deal with complexity??
6. Are averages masking a huge variation or skew in data, for example because of some important outliers?
7. Are comparisons over time or between categories statistically significant i.e., the difference is non-random?

Asking a few of these questions as you read a report, hear a consultant presentation, or debate a strategic recommendation with your peers will help you avoid the trap of absolutes and facilitate nuanced decisions that are adaptive to changes in the environment. ?

After all, being a leader is not about painting a big cow egg by the numbers, it’s about seeing the patterns, potential and risks in the numbers and creating what you can from it.

[i] “The Future of Jobs Report 2023”. World Economic Forum.

[i] “The Big Con: How the Consulting Industry Weakens our Businesses, Infantilizes our Governments and Warps our Economies.”? Mariana Mazzucato, Rosie Collington, 2023.

[ii] “Global Workplace Survey Comparison 2023”, Gensler, p.10.