Under the Aqueduct : exploring problems of collective decision-making

It has been now more than a decade that I have been underwhelming MBA students in the first weeks after they arrive at my school with a workshop entitled “The Aqueduct Challenge.” The challenge is simple enough: given the existence of the Roman aqueduct in Segovia (pictured above), what can we conclude was the population of the city?

I originally conceived the exercise as an offbeat estimation problem, of the kind then much in vogue at top-shelf employers – McKinsey, Google, Microsoft, etc… – like “how many hairdressers are there in Singapore?” (And, incidentally, this is a poor strategy for candidate screening, as I’ll discuss briefly at the end of this post.) The idea was to use an example from the Roman period (since I am a historian) to exercise critical-thought and analytical tool-sets needed to come up with some kind of an answer. Students, working in groups, apply their critical thinking skills to this issue of Roman-era Segovian demographics for an hour, after which we have a lively discussion dissecting all the answers from each group.?

Which were all invariably wrong.

For the first maybe five years of running the exercise I didn’t think too much about this 100% failure rate in getting the right answer. But after more than a hundred iterations of the exercise (and thousands of wholly incorrect answers), it dawned on me that this was statistically anomalous.?

It seems I had inadvertently created the world’s worst estimation question.

Ordinarily, the way to resolve estimation questions is to establish a simple framework and then to fill in the missing variables by making assumptions. For example, to answer how many hairdressers there are in Singapore, we can (1) start by estimating how often an individual visits a hairdresser per year, (2) work out a total number of visits per year using a population estimation of Singapore, and (3) estimate how many clients a hairdresser can see in a day. Doing simple arithmetic, we can work out how many hairdressers are needed to meet annual demand, and then add in a few tweaks to look clever (e.g. skew by age & gender, types of hairdresser visits, or account for hairdresser downtime/vacation days) and come up with a credible answer. Et voilà.?

In general, for estimation questions such as these, I would predict that the more people who collaborate on answering the question, the more accurate it will become. This is because at every point where we have to fill in a blank, such as how often an individual visits the hairdresser per year, the more eyeballs on the problem, the more accurate the estimation.?

For example, let’s suppose I ask “estimate the possible market size of people who will embrace the metaverse, grateful that tech has finally solved the misery of interacting with people in the real-world now that we can hide behind an eternally-cheery jack-in-the-box pikachu avatar (now with legs!).” If the one person I ask is the CEO of Meta then the answer will be: everyone. Or if the person I ask is a particularly ornery professor of humanities at IE Business School I know, I will get another answer: nobody. Put both those answers together and we’re certain to arrive at a closer approximation of the truth (which is?six).

Back to my aqueduct problem. As any student who can vaguely recall suffering through the experience can attest to, the parameters are very straightforward and the number of variables you need to take into account very small. So, as a group exercise, it ought to follow this rule: greater accuracy based on more inputs to generate the output. So, since no group ever gets the question right, it would seem quite useless, not just as a means for evaluating the kind of “critical thinking” logic needed to solve such problems, but indeed as an exercise in exploring critical thinking skills generally. It’s just a mean-spirited exercise designed to waste two hours of someone’s life they are not getting back. And, worse: are paying for. Not nice.

What was the problem?

I have recently started a substack for my writing, which offers a better venue than LinkedIn, and I invite you to read the rest of this article there. I would be honoured (but in no way humbled!), if you would subscribe.