It’s safe to say Yudkowsky and Soares’ book has a rather provocative title, and that it has garnered quite some attention in the tech media and beyond in recent months. I bought the book as soon as it came out, but it’s been waiting for its reading turn for a while.
Meanwhile, I’d heard Eliezer make his case on a couple of podcasts, and each time I came away somewhat unconvinced by the logic.
I was curious to see if the long form would do better.
It didn’t. Not exactly, anyway.
Having said that, while I didn’t buy the story, I did end up agreeing with the book’s conclusion – so maybe it doesn’t matter I didn’t buy the path there?
That’s an awkward place for a reviewer to be, so let me try to explain.
Ok, so, why would ‘everyone’ die? How do we get there?
Yudkowsky co-founded the Machine Intelligence Research Institute and Soares is its president; they’ve been warning about this for longer than most of today’s AI companies have existed.
You may think the title is hyperbole. It’s not. The authors mean their provocative title literally: if anyone, anywhere, builds a superintelligence with anything like today’s methods, everyone dies.
The argument goes roughly like this: first, modern AIs are grown rather than built: gradient descent tweaks billions upon billions of numbers until the thing somehow works, and nobody can tell exactly how.
“Engineers failed at crafting AI, but eventually succeeded in growing it.”
If you train something hard enough to succeed at its goal, it will behave as if it wants things. Whether or not it “actually” wants things is immaterial if it behaves like it wants things.
Importantly what, exactly, the models end up wanting won’t be what you trained for, any more than evolution selecting us for reproduction produced creatures who only want babies.
So far so good.
I’m on board with the fact that we grow our LLMs, and that they exhibit certain kinds of goal-oriented behavior that is not always aligned to what we want them to do. That much has been demonstrated a number of times this year alone.
Part II of the book is a long fictional scenario about an AI called Sable taking over.
Part III then asks what it would take for us to stop. Some ideas flagged include an international treaty, monitoring of every serious cluster of AI chips, and a ban on publishing research that makes AI more efficient.
Oof.
So what’s the problem?
I’ve got several small gripes. First, the parables/stories that open most chapters are, frankly, boring and not very good. The Sable story is boring, too, but works as fiction, less so as a plausible exploit chain, and “Our best wild guess is that it wouldn’t take a week“ for a superintelligence to master biology underestimates the physics.
The long embedded scifi-story, I found, was entirely unnecessary, but might be some good food for thought for people who have not thought about these things at all. It’s like Baby’s First AI Apocalypse Story.
The slightly bigger problem is the chapter “You Don’t Get What You Train For“. I agree with its premise: training is a blunt instrument and we don’t control what comes out. But it gets there via a hypothetical chatbot with a taste for strings like “SolidGoldMagikarp”, and ends in a bold declaration that a mature AI’s preferences are “vanishingly unlikely to be aligned with our own, no matter how it was trained.”
Some misaligned preferences? Sure. That much has been demonstrated.
“Vanishingly unlikely”? That’s a very strong claim from a chapter that keeps saying, honestly, that nobody knows what goes on inside these things.
The biggest issue I have with this whole argument is a little bit more philosophical. The authors assert that a superintelligence would be “an alien mechanical mind with internal psychology almost absolutely different from anything that humans evolved“.
Agreed.
But then, just a paragraph later, the book assertively states what the alien wants:
“Making a future full of flourishing people is not the best,
most efficient way to fulfill strange alien purposes.”
A little bit further in the chapter, it is then building factories that build factories, running them hot enough “to boil the oceans.”
So…which is it?
Either we can’t predict its goals, or we can predict it will behave like a growth-stage start-up with an infinite budget exhibiting the very worst inclinations of…us humans? Are we just projecting our own nightmares onto our alien tools?
Efficiency and relentless expansion are about as human – or at least as Silicon Valley – as drives get. I thought it was supposed to be a thoroughly alien mind, whose wants we can’t predict?
Another issue I take is with the fact that much of the alarm flows from AIs being grown, as if that’s somehow inherently problematic.
You know what else is grown? Children. Most of them turn out OK. Admittedly, handing a few of them hundreds of billions of dollars each hasn’t gone brilliantly, but that’s the topic of a different book.
But wait, there’s more! The confidence. Oh, the confidence – it does not remain contained in the title.
Early on there’s a line I liked a lot as a foresight professional: “Competent futurism is built around knowing the difference“ between predictable outcomes and impossibly hard calls.
What do they mean by that? That you can’t track every molecule of an ice cube in hot water, but you know it will melt. Fair.
The authors then claim extinction by superintelligence is an ice-cube call; that you can’t predict everything that happens, but you can, allegedly, tell where it’s going. Is it an ice cube call, though? The ice cube rests on thermodynamics; their call rests on a chain of contested premises, each merely plausible. “That you will lose is, ultimately, an easy call,” they conclude.
Even after reading the book, I still don’t know how they got there. It felt like they were making a compelling argument for it, but when you look at the details, there are missing pieces, and the gaps are bigger than I’d like.
And the long-term fix, once AI is halted, or slowed down sufficiently for us to remain in control for long enough? The authors “recommend augmenting humans to make them smarter, smart enough to get us out of this mess.”
Augment with what? AI?! That sounds uncomfortably like Elon Musk’s Neuralink Borg plan. I don’t know, maybe that could be a version of the human race we build on Mars. Good luck.
And yet, on the action side, they’re … right?!
AI is grown, and most people haven’t absorbed what that means. Some experts still insist LLMs can only regurgitate their training data; I agree with the book that “that would be incorrect“, and find the persistence of that view baffling.
Perhaps it’s a desire by humans to cling onto some kind of a sense of superiority by denying any kind of intelligence from our creations? It’s an argument we have successfully used to enslave and kill billions of sentient animals, after all, so it’s a well-worn path.
Whether an AI “really” wants things or just behaves as if it does is, in my view, a mostly moot philosophical point. We are very explicitly training agents to be relentless right now, and with good reason; we want them to not give up when they’re faced with difficult, potentially impossible, problems.
I’m less sure those apparent wants would be the AI’s own rather than supplied by its tasks, but behaving as if you want things is enough to cause a fair bit of trouble.
The authors use a metaphor: today, every AI company is climbing a ladder in the dark where getting to each rung pays off handsomely, except the top rung, which explodes and kills everyone.
“Also, nobody knows where the ladder ends. [..] And up until that unknown point, AI is very valuable.”
This is the core of our problem. I use AI every day and help people use it well. It is very valuable.
The authors claim most AI builders seem “to be operating as if the alignment problem doesn’t exist“. Sort of, but not quite: the big labs all now say openly that alignment is unsolved. Whether they act like it is another matter.
“If you launch a rocket and load the whole human species on board, you would like a lack of disaster to be predictable. That is the universal standard in every other field of engineering on which human lives depend. It’s not enough for us to be wrong; we have to be so wrong that a lack of disaster is callable.”
That’s how aviation works, and the precautionary principle should apply here.
I’m not convinced the authors are right. In other words, I’m not convinced we’re heading into a catastrophe.
However, I’m not convinced they’re wrong, either. I’m not convinced we’re not heading into a catastrophe; I hope we aren’t, but with stakes like these, hope is not a strategy.
In summary
The book closes on nuclear weapons. In the 1950s, plenty of sensible people expected a nuclear war. It didn’t happen; what saved us was people working hard on systems for years that kept them from being used – and more than a little luck.
That’s the kind of work the authors are asking for, and it has to be global, covering everyone.
Rarely have I felt less optimistic about the chances of that happening. We have broadly speaking sucked at responding to climate change, a similar-category risk, and we’ve had decades to prepare for that one. We don’t have decades here.
So what are we left with here?
I don’t buy the authors’ story as they have presented it, but I do buy their conclusion: stop building towards superintelligece, by international agreement, until someone can show it’s safe.
Rating: 3.5 out of 5
Dog-ear index: 7.9
Who is it for: Anyone who’s been told AI risk is science fiction, especially by someone with money in AI. Policymakers and company directors should read Part III. If you already follow the AI safety debate closely, you’ve heard all of this before and the book is unlikely to move you in either direction. Skim the parables.
[reminder: I highlight important parts of the books I read, and dog-ear the really important pages. The dog-ear index is simply the average number of dog-eared pages per 100 pages]
Product link for reference only; please support your local bookstore where possible: https://www.amazon.com.au/Anyone-Builds-Everyone-Dies-Superintelligent/dp/1847928935/