Why Nick Bostrom got it wrong

Critiquing Nick Bostrum’s thinking feels a little bit like checking Einstein’s sums. Bostrum’s an AI rockstar; his homepage carries recommendations from the likes of Bill Gates; I have heard nothing but praise for his work from people whose opinions I deeply respect; and I love the fact that (in homage to Alan Turing’s seminal paper?) he used a Disney movie as inspiration for a his career-defining AI hypothesis.

So how come his latest working paper, The Vulnerable World Hypothesis, gets so much wrong? I have a hunch I know why. But first let’s look at what.

The urn theory

Let’s start with Bostrom’s definition of our vulnerability:

One way of looking at human creativity is as a process of pulling balls out of a giant urn. The balls represent possible ideas, discoveries, technological inventions. Over the course of history, we have extracted a great many balls—mostly white (beneficial) but also various shades of grey (moderately harmful ones and mixed blessings)…What we haven’t extracted, so far, is a black ball—a technology that invariably or by default destroys the civilization that invents it.

I wonder, is a lottery draw really the best analogy for creativity? Hollywood movies and that story about Archimedes might suggest that technological breakthroughs come in a flash like lightning bolts, but in practice they are the culmination of longer, collaborative processes. Newton and Leibnitz developed the Calculus independently and broadly concurrently not because of random statistical coincidence (and probably not by sneaking a peak at each other’s sums, despite mutual accusations of same) but because they were part of the same intellectual culture in Europe. They read each other’s earlier work, they were exposed to similar ideas and inspirations and conversations, they were both travellers on a community-shared journey of trial and error, argument and testing.

Neither of them plucked the Calculus fully-formed from an urn.

Creativity: black or white?

There’s also something unbalanced about Bostrom’s white, grey and black balls theory. The coloured balls represent good, middling and bad outcomes, but only the bad – Black Ball – is civilisation-destroying. If an inevitably destructive technology is possible, why not an inevitably productive one? Something that saves humanity and the planet together? Sort of like in that Dan Brown novel?

In reality, all technological leaps forward have been inherently Grey – they carry the potential for bad as well as good outcomes. The same is true for natural disasters. The eruption of Toba, or whatever annihilated over 90% of humanity 70,000-ish years ago, was a Bad Thing, let’s not kid ourselves, but the bottleneck in the human gene pool that resulted – everyone alive today is descended from the small (probably single-digit thousands) group of humans who survived – may well have favoured characteristics that have done us proud since: our ability to communicate, cooperate, adapt and – not such a source of pride this one, but still – wipe out rival species.

But it is, of course, possible that as the small print says, past performance may not be an accurate guide to future performance. Irredeemably Black Balls – killer technologies on the Toba scale but without the side-benefits – may emerge. What might they look like?

Black Balls revealed

Bostrom spends quite a few pages outlining some potential Black Ball scenarios: easy-bad tech (think nukes that are as simple to put together as a Billy bookcase); bad tech that’s too tempting not to deploy in a first-strike-wins manoeuvre (nukes again! Bostrom is so last-century sometimes); embedded bad tech that would require a complete reboot of our economy and society to eliminate (like carbon-based energy, only worse); and something he calls “surprising strangelets” which I think means the sort of “who’d have guessed?” unexpected disasters we all encounter, especially at this festive time of year.

It’s an interesting taxonomy, not least because it bears little resemblance to the tech-related problems humanity has encountered so far. Let’s look at an example Bostrom himself quotes: Rapa Nui, or Easter Island as he Euro-centrically calls it. Bostrom categorises this as an almost-Black Ball – a culture-destroying event not quite big enough to rank as civilisation-destroying – but I bet it didn’t feel that minor to those involved, and on a micro-scale it fits his Black Ball scenario. What went wrong?

Bostrom follows the Jared Diamond thesis that the Islanders sealed their own doom by cutting down all the trees to use as rollers for moving stone blocks to make those statues, only to find that without trees the soil eroded and crops failed, so vast numbers of people starved and the last survivors, maddened by hunger pangs, used the waking hours freed up by the absence of tilling, weeding, etc to fight each other to death.

There’s only one problem with that theory: it’s almost certainly bunkum.

True, Easter Island was rapidly deforested once humans arrived. But it wasn’t art but an anagram of that word that most likely killed the trees: the Polynesian rat, a sapling-munching rodent that arrived with the settlers in their canoes. As for the collapse in the human population, there is no evidence for that occurring before the Europeans came in 1722…around the same time that Easter Islander DNA starts to show up in South American DNA, likely a result of South American slave raids which are estimated to have removed around half the native population, the rest being all-but wiped out by European germs, destruction of property and enforced migration.

So it was not a technological Black Ball that depopulated Easter Island, but the combination of migration-tech side-effects, natural processes and human exploitation of other humans for economic gain.

Why does this complexity matter? Because along with complexity come multiple opportunities for redemptive action. Exactly what Bostrom’s Black Ball theory does not permit.

But, of course, this time might be different. Let’s look at why Bostrom believes our world is so vulnerable to Black Ball killer apps, and what we can do to reduce that vulnerability

(Semi-)anarchy in the UK

The heart of Bostrom’s argument is that to combat the threat of Black Ball tech, we need to fix our civilisation’s “semi-anarchic default condition”.

Semi-anarchic default just about sums up my life. But our entire civilisation? Surely anarchy is its opposite, not an inherent component?

Bostrom’s semi-anarchy is based around three factors: our limited capacity for preventative policing; our limited capacity for global governance; and our diverse motivations. By “preventive policing” he means we need to boost surveillance and interception of potential Black Ballers; by “global governance” he means we need to get serious about worldwide coordination, especially on security; and by “diverse motivations” he means we need to reduce the broad distribution of human goals and preferences, eliminating incentives for individuals or groups to destroy civilisation.

I think he is wrong about two out of the three. And too pessimistic about the one he got right.

Where he has a strong case is global governance. Nations don’t always play well together, and right now it seems like multi-national forums and alliances are weakening, not getting stronger. But there is room for (cautious) optimism. Alliances form, and work well, when there is a focused and serious threat to be countered. They have changed as times change. Today we find ourselves in a transition period. There is no reason why, once again, we won’t adapt our institutions and alliances to fit our needs. Indeed, we are already starting to do so.

As for Bostrom’s other two vulnerabilities and attendant solutions, I think he’s not only wrong but dangerously wrong.

It’s true that we need to get better at preventative policing, but that doesn’t mean what Bostrom thinks it means. What has worked in practice is not Minority Report-style surveillance and interception, but a broader public health model – treating illegal actions like epidemics by blocking transmission, preventing future spread and changing group norms.

Big Brother tactics are not only morally dubious, they don’t work.

Managing motivation is an even dicier area. Human desires and preferences have a natural distribution, as with any other data set. Bostrom makes what I think is a false distinction between self-interest (good) and potentially-destructive (bad) drivers of behaviour. In practice, we are all motivated by self-interest, which is based more on beliefs and preferences than on actuarial tables. Martyrdom in a religious war might not top most people’s to-do lists, but an Islamic jihadist might go to his death convinced – now that the raisin theory has been demolished – that among the seven blessings awaiting him in the Qu’uranic Paradise are 72 virgins with non-drooping breasts (sorry, female martyrs, you only get one man apiece, pectoral pertness not specified).

Bostrom’s suggestions for shifting motivation towards the positive range from fluffy (move nations “in a peace-loving direction”) to potentially discriminatory (screening unsuitable candidates out of any path towards technological advancement – is he not aware how biased hiring is?) to just plain ineffective (he proposes motivating behaviour change through economic gain, ignoring the research consensus that money is an excellent demotivator but a useless and often counter-productive motivator). His final proposals (bottom of page 21) sound like something out of 1984: restricting information-sharing, establishing “some kind of surveillance and enforcement mechanism”. He even sketches out plans for a PanOpticon and a Freedom Tag. Finally he proposes that:

A state that refuses to implement the requisite safeguards—perhaps on grounds that it values personal freedom too highly or accords citizens a constitutionally inscribed right to privacy—would be a delinquent member of the international community.

 Let’s not take that argument any further.

 But, semi-anarchic or not, Bostrom is right about one thing: our world sure feels vulnerable. And I think I know why.

Real-world vulnerability…and how to fix it

What makes us feel vulnerable? In a word: complexity.

In our interconnected, highly technical twenty-first century, most of us no longer understand how stuff works. From British politicians claiming that the Brexit trade negotiations would be the easiest deal to strike in history, to the shocking revelations of security holes in much new tech (Internet of Things, I’m looking at you), we are increasingly learning that our world is not simple, that we have failed to predict problems, that solutions are slow and complicated and unsatisfactory and do not fit the standard models.

Hence the fear of an unstoppable Black Ball technology (I can’t be the only one to be reminded of Skynet, by which I mean the Terminator robot command system, not the hokey Belgian internet portal). It’s a metaphor for existential fear of losing control.

Luckily, I think there is a way to make our world less vulnerable. And it goes back to Bostrom’s urn analogy, or rather to his source for that metaphor.

The Greeks had a word for it

The first time the urn metaphor for fate comes up is in Homer’s Iliad – rapidly becoming the go-to source for my blog posts! By the final book Achilles, the rage-maddened Greek protagonist, has killed Hector, his decent-bloke opposite number on the Trojan side, and – to everyone’s horror and condemnation – has desecrated his body and refused to release it for burial. Priam, Hector’s father, takes the unprecedented step of approaching his son’s killer, unarmed and alone, to ask him to reconsider. Achilles is reminded of his own father, the mortal who married a goddess and ended up a shrivelled husk, unable to die (his wife had asked Zeus to make him immortal, but forgot to add in eternal youth). Both men weep together, and Achilles agrees to give up Hector’s body. But first he makes a speech about how life works:

Two urns stand on Zeus’ threshold, each bearing different gifts: one holds evils, one blessings. If Zeus the Thunderer gifts a man from both urns, he will find life a mix of trouble and joy. But when Zeus reaches only into the urn of sorrows, he wrecks a man’s life, and desperation will drive him to wander alone through the glowing world, cast out by both gods and mortals. Iliad 24. 527-33

The urns in the Iliad are a way of explaining the arbitrary cruelty of life, but Achilles does not end his speech there. He asks Priam to sit and eat with him – a reestablishment of normal hospitality – and promises to deliver up his son’s body for burial.

He makes a moral, a human, choice. And it defeats the evils that come from the urn of sorrows.

In real life we had Colonel Stanislav Petrov. In 1983, while serving as duty officer for the Soviet nuclear early-warning system, he picked up a signal that the US had launched a nuclear missile. A few minutes later, the system indicated five more missiles had been launched. Petrov insisted that the system was malfunctioning, that such an attack was inconceivable. He refused to order a counter-attack without further evidence…which in the end was not forthcoming. The supposed attack was actually a rare alignment of sunlight glancing off high-altitude clouds. Light, not fight. But that was only discovered months later.

In the crucial moment, Petrov’s concern for the broader context, his reluctance to launch a Black Ball, his human dithering if you like, saved the world.

What Bostrom misses…and why

It’s the humanity of a Colonel Petrov, of an Achilles, that Bostrom’s analysis misses.

His world is one of surveillance, of punishment, of absolute clarity and decisive action. Life doesn’t work like that. Great policing is about winning hearts and minds, not peeping into people’s souls or controlling their thoughts and aspirations. International cooperation is about aligning self-interest, not about arbitrary rules. The way we have saved the world so far is by letting humans be human. In the midst of bleakness, comes generosity and fellow-feeling. In a military command structure, a man pauses to think, to waver.

I’ve a hunch that one of the reasons Bostrom misses this human element is that the community who commented on his paper were less than diverse. Fewer than 8% of those he thanks in his introduction are female. The same percentage have non-European last names – and one member of both minority groups is Bostrom’s executive assistant. Inputs from people with broader backgrounds might have broadened his world-view, made his paper better.

I hope the Future of Humanity Institute is more representative of our world than Bostrom’s paper evidences. The press release for the recent gift of ￡13.3million featured a photo showing a reasonably diverse range of researchers. But the Future of Humanity Institute’s website showcases only the picture of three white, middle-aged men I borrowed to illustrate this article.

If Bostrom’s world is vulnerable, perhaps that’s because it over-represents white men – a group that feels increasingly vulnerable, and eager for punitive solutions?

Just a thought.