Superintelligence, Rationality, and the Race to Save the World
What the world needs, urgently, is more rationality. It needs a greater number of people to be aware of the mistakes that are, too often, made due to flaws and biases in our thinking processes. It needs a community that can highlight the most important principles of rationality – a community that can help more and more people to learn, step-by-step, better methods of applied rationality. And, critically, the world needs a greater appreciation of a set of existential risks that threaten grave consequences for the future of humanity – risks that include misconfigured artificial superintelligence.
These statements express views held by a community known sometimes as “Less Wrong” (the name of the website on which many of the key ideas were developed), and sometimes, more simply, as “the rationalists”. That last term is frequently used in a new book by science writer Tom Chivers – a book that provides an accessible summary of the Less Wrong community. As well as being accessible, the summary is friendly, fair-minded, and (occasionally) critical.
The subtitle of Chivers’ book is straightforward enough: “Superintelligence, Rationality, and the Race to Save the World”. The race is between, on the one hand, the rapid development of technology with additional capabilities, and on the other hand, the development of suitable safety frameworks to ensure that this technology allows humanity to flourish rather than destroying us.
The title of the book takes a bit more explaining: “The AI Does Not Hate You”.
This phrase is a reference to a statement by one of the leading thinkers of the community in question, Eliezer Yudkowsky:
The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else.
In other words, the existential risk posed by artificial superintelligence isn’t that it will somehow acquire the human characteristic of hatred, but that it will end up following a trajectory which is misaligned with the best interests of humanity – a trajectory that sees humans as a kind of irrelevance.
To be clear, I share this worry. I’ve given my reasons many times on my personal blog, and I wrote up my own analysis at some length in chapter 9, “Towards abundant intelligence”, in my most recent book, “Sustainable superabundance”. My ideas have been shaped and improved by many things I’ve learned over the years from members of the Less Wrong community. Indeed, my presentations about the future of AI generally include several quotations from Yudkowsky.
However, these ideas often cause a kind of… embarrassment. Various writers on AI have poured scorn on them. Artificial superintelligence won’t arrive any time soon, they assert. Or if it does, it will be easy to keep under human control. Or if it transcends human control, there’s no reason to be alarmed, because its intelligence will automatically ensure that it behaves impeccably towards humans. And so on.
These critics often have a second string to their analysis. Not only do they argue for being relaxed about the idea of existential risks from superintelligence. They also argue that people who do worry about these risks – people like Yudkowsky, or Oxford University’s Nick Bostrom, or Stephen Hawking, or Elon Musk – are somehow personally defective. (“They’re egotistical”, runs one complaint. “There’s no need to pay any attention to these people”, the critics continue, “since they’re just philosophers, or mathematicians, or physicists, or business people, etc, rather than being a real AI expert”.)
At an extreme, this set of criticisms expresses itself in the idea that the Less Wrong community is a “cult”. A related objection is that a focus on humanity’s potential extinction is a distraction from much more pressing real-world issues of the present-day and near-term future – issues such as AI algorithms being biased, or AI algorithms stirring up dangerous social divisions, or increasing economic inequality, or disrupting employment, or making weapons more dangerous.
It’s in this context that the book by Chivers arrives. It tackles head-on the controversies around the Less Wrong community – controversies over its ideas, methods, aims, and the lifestyles and personalities of many of its leading figures. It does this carefully and (for the most part) engagingly.
As the book proceeds, Chivers gives voice to the various conflicting ideas he finds in himself regarding the core ideas of the Less Wrong community. My own judgement is that his assessments are fair. He makes it clear that, despite its “weird” angles, the community deserves more attention – much more attention – for its core ideas, and for the methods of rationality that it advocates.
It’s a cerebral book, but with considerable wit. And there are some touching stories in it (especially – spoiler alert – towards the end).
The book provides the very useful service of providing short introductions to many topics on which the Less Wrong community has written voluminously. On many occasions over the years, I’ve clicked into Less Wrong material, found it to be interesting, but also… long. Oh-so-long. And I got distracted long before I reached the punchline. In contrast, the book by Chivers is divided up into digestible short chunks, with a strong sense of momentum throughout.
As for the content of the book, probably about 50% was material that I already knew well, and which gave me no surprise. About 30% was material with which I was less familiar, and which filled in gaps in my previous understanding. That leaves perhaps 20% of the content which was pretty new to me.
I can’t say that the book has made me change my mind about any topic. However, it has made me want to find out more about the courses offered by CFAR (the Center For Applied Rationality), which features during various episodes Chivers recounts. And I’m already thinking of ways in which I’ll update my various slidesets, on account of the ideas covered in the book.
In summary, I would recommend this book to anyone who has heard about Less Wrong, Eliezer Yudkowsky, Nick Bostrom, or others in the extended rationalist community, and who is unsure what to think about the ideas they champion. This book will give you plenty of help in deciding how seriously you should take these ideas. You’ll find good reasons to counter the voices of those critics who seek (for whatever reasons) to belittle the Less Wrong community. And if you end up more worried than before about the existential risks posed by artificial superintelligence, that’s no bad thing!
PS1: For a 10 minute audio interview in which Tom Chivers talks about his book, visit this Monocle page.
PS2: If you want to see what the Less Wrong community members think about this book, visit this thread on the Less Wrong site.
David, Hi. A fascinating introduction to lesswrong.com, as I did not know of it, or the book, which due to limited bandwidth, I am unlikely to read. I use to think there were always P +1 versions of the truth, where, ‘the truth’ is something to act upon and P = the number of people involved. Your piece got me initially thinking there were now P + M + 1 versions, where M = the number of machine interpretations. However, given the dynamic interaction of all the different perceivers in any complex system, there are likely to be infinitely more, with, through cognitive dissonance, both humans and different algorithmic subroutines, working with multiple conflicting models at the exact same time. This brings me to three additional considerations. Firstly, the conflict between the ‘less wrongers’, which is the camp I would belong to and their detractors, seems to simply be an extension of the precautionary versus ‘proactionary’ debate. This dangers of the proactionary approach are advancing exponentially, because powerful tech. is becoming ever more accessible and there is a Z-axis to any assessed risk, complexity, which, over time, will increase probability and impact. A cursory glance of lesswrong.com’s website indicates it differentiates basic rationality from human rationality. This is key, because, from a human perspective, humans are more than just atoms. At a lower level, we can also see how ‘rational’ interpretations of what is reasonable, will differ based upon the cultural leanings of the jury and participants in any trial. In this respect, at the very root of a truly human rationality, I would hope to find the ultimate purpose being to maximise net present human happiness – truly global utilitarianism. N.B., this is not promoting communism, but better governance-based capitalism, with a healthy dose of oversight and state sponsored innovation, shaped by environmental, social and runaway-tech. concerns. However, I suspect proactionaries are more likely underpinned by some variation of Rand’s Objectivism, which in my view, is nothing more than a cultish ploy to promote self-interest and, in so doing, the 'gloves-off', capitalist ‘machine’ that has emerged and taken increasing hold over the past several decades. Might it be that the first non-human super intelligence is already among us? Not a quantum computer, or contrived network of machines, but a distributed, steampunk-like combination of businesses, organisations and individuals – networked global capitalism itself – that took decades to emerge and now results in many of us taking decisions which are against our better interests? There may even be another super-intelligence – alien and so advanced, we have no knowledge of it - observing and monitoring our ‘progress’. From its elevated position, it may have even concluded that our modern capitalist machine, encompassing, great chunks of activity across the OECD, Russia, China, Tax Havens and elsewhere, had become self-aware. However, it might also have observed how it has gone the way of many others, and is now teetering on a knife edge between salvation and state change to a Kardashev Type 1, 'super-organism' of happy, 'autonomous' humans, or an Alzheimer’s type decline into oblivion, as biodiversity and human well-being - key elements of its distributed 'brain'- start collapsing. So yes, leswrong seem right and we should also aim to maximise net present GLOBAL human happiness.?