What do the bros mean when they claim to be so worried AI will wipe out humanity, and what risks do we actually need regulation to protect us from?
AI was not used in writing this post.
Doom as represented in a medieval church in Dauntsey, Wiltshire, UK.
https://reeddesign.co.uk/paintedchurch/dauntsey-doom.htm
I had breakfast with my daughter (a brilliant young painter with – proud dad moment coming – her first solo show coming up) and when I told her my p-doom wasn’t so high, she was eager to hear more and why – hers had gone from unknown to high in a few days of media coverage. And then I had almost exactly the same conversation later that day when I bounced into Nesta – I asked my very dear sparring partner, Tim (now Lord ) Leunig what was his p-doom, and as we both violently agreed that it was much lower than the media had been making out, another young colleague said she was not so sure, and wanted to know why we were so (relatively) rosy.
The idea that the robots are about to get out of control and terminate us has hit a real nerve, so I thought I’d get down on paper why my p-doom is so low, and what I think the AI regulation we really need – one that addresses concrete concerns – should look like.
But first, what does p(doom)>0.1 even mean?
“The probability that we are doomed is greater than 10%”.
Fine … but what do we mean by “probability” and “10%” in this context? It’s a somewhat comprehensible statement when someone says “the probability of flipping 3 heads in a row is 1/8 (12.5%) (½ x ½ x ½)”. By this we mean that, empirically, if you take a fair coin and flip it many, many times, then about 12.5% of all sequences of 3 results will be of three heads in a row.
But what is the equivalent for doom? After all, it is not as if anyone has actually run the experiment as with the coin-toss. This is where we get into the slight weirdness of Bayesian probability. There’s an old debate in probability between Frequentists – people who like to interpret probability statements in the empirical way that I did above on the coin-toss – and Bayesians, who believe that in most hard cases we cannot be frequentist, and that probabilities in those cases actually are a way of putting a value to a subjective “strength of belief”. Bayesians tend to be more interested in how you update your beliefs. Say, for example, that I put up a shed in my garden and feel quite happy with the way I’ve connected the electrics. Then an official comes by and says to me: “that looks like a fire hazard … it is very dangerous … I would deal with it if I were you”. That might be the basis for updating and upgrading my prior belief that it was all OK. But say I then find out that the official used to be a fireman and, over a cup of tea, tells me of all his personal (and self-selected) experience of seeing things go wrong. I might update my belief again, thinking: “this person’s experience is likely to bias them towards an exaggeration of risk”. And let’s say that as the cup of tea continues, it turns out that the official also has a line in selling insurance. Again, as a good Bayesian updater of subjective probability, I would again downgrade the value of the new information I had received. There are ways that Bayesians have to anchor these “strengths of belief” in actual probabilities, many of them involving encouraging people to make bets – hypothetical, and even better, actual.
So first thing to note: p(doom)>0.1 is a subjective measure. In the case that has received so much attention these past few days (42.6m views of the tweet since Sept 9th), it measures something like the degree of surprise that Evan Hubinger, who has an AI safety job at Anthropic, will experience if AI does cause mass-extinction within 10 years. He is really saying to us: “I would not be very surprised if AI kills us all within 10 years”.
There is a great deal of false-looking objectivity in the formulation, so let’s stick to the simple words version, and ask what rational people should do with the statement. We need to find out: why does Evan believe this, and do we agree with the steps that go to make up his strength of belief. He says this in his next tweet, viewed only 3m times:
So this is the nub of Evan’s belief: that Recursive Self Improvement (RSI) is pretty likely, and that if that happens, then doom is pretty likely. Why so? Get ready now for a vague but classic SciFi scenario: “machine gets better at making machine better; it quickly outgrows its maker’s power and comprehension; its behaviour is “mis-aligned” with human flourishing because of some design flaw; humanity becomes an obstruction to its goals … so machine eliminates humanity.”
We can summarise that with another funny formula:
p(Mis-or-Faked alignment) x p(RSI) = p(💣) > 10%
Evan does not break down his beliefs: is it that p(RSI) = 10% and p(Mis-or-Faked alignment) is 100%? Or maybe p(RSI) = 20% and p(Mis-or-Faked alignment) is 50%… For what it’s worth, my own view is that p(Mis-or-Faked alignment) is approximately 100% and that p(the relevant sort of RSI) within 10 years is approximately zero. I’ll say a bit about why I believe this. But more importantly, I believe that the right kind of regulation will tell us with near certainty when the probability that RSI is around the corner (say a couple of years away). In other words, I agree with Dario Amodei, Evan’s boss, that we need to implement supervisory regulation of frontier AI labs. I don’t agree with the particular form he proposes, but more on that below.
So why do I think that over the relevant time period, that:
p(Mis-or-Faked alignment) ~ 100% and p(RSI) ~ 0, so p(💣) ~ 0 ?
(Sorry, having too much fun with obfuscating math formulae … this one is just a repeat of my belief that we won’t solve alignment but we won’t get the relevant self-improvement, so we are almost certainly not doomed…).
Why am I so pessimistic about the right kind of recursive self-improvement arising (or, sorry, perhaps that is optimistic)? I think the RSI-maxxers have something like move 37 in mind when they think of this space – the moment when AlphaGo, in its second 2016 game against the world champion, made a completely “inhuman” move that retrospectively turned out to have been brilliant. If learning how to learn – ie RSI – were a board game, the right kind of RSI is like move 37: the robot does something that human researchers would not have thought of and which turns out to be brilliant.
In a way, next-word (or token) prediction has indeed been turned into a board game and models have become very good at it. That is not yet learning how to learn, of course. That game is one level up from token prediction – it is LLM architecture and training prediction. Say we started with the very first neural nets as imagined by Turing in 1948. The “next step” prediction might then be the work done in the 1950s by Rosenblatt, Hebb & co. We’d slowly get to the pioneering 1980s work by Hinton … and so on. You get my point: to learn to learn, a machine would have to show that it could generate the “next sentence in the story of learning”.
There is not much data for “next sentence in the story of learning” – unlike with the other areas in which AI has been spectacularly successful. That, I think, is why researchers like Yann LeCun believe that models need “world models” before we can get to ASI. This is sometimes framed in terms of the efficiency with which children, exposed to the world, learn to navigate it. Subjective experience of physics makes them dextrous, able to move around space. The point is this – move 37 was possible because in a board game, the rules of the game perfectly define the world that a robot has to learn to navigate. LeCun is building AI with access to world models … but as far as I can tell, these are still only world models of physics. There are brilliant companies, including London-based Matta.ai, that are also working at this problem from the practical perspective of improving manufacturing processes. All that is hard enough.
But if we are worried about training Terminator-class models, these robots will need a near perfect understanding of the social world as well – if Terminator is going to get us because we are superfluous to its goals, it will have to be an expert social manipulator; it will need to build coalitions with humans then turn against erstwhile friends, it will need subterfuge and cunning of the kind Macchiavelli never dreamt of. Stephen Wolfram has recently posted about the complexity of game-playing machines, and it’s worth looking at the extraordinary complexity that even extremely simplified versions of social interaction. There is a result due to my (sadly deceased) longtime collaborator, Sunil Talwar, that the complexity of almost all social strategies, even highly simplified cases, tends towards infinity once you repeat them indefinitely. All this shouts out to me that we will not get “world models” that are any good at precise modelling of the social world for a very long time – perhaps ever. And if we don’t have those, then we’re not going to get “rule 37” social behaviour out of robots.
There is much more to say about dreams of RSI. But in fact I have said more than my argument really needs. The point is not to say that world models will never be good enough for the right kind of RSI to get going, RSI that will allow moves based on accurate social prediction. All I need is to point out that we are really a very long way away from that kind of model, and therefore we can sleep soundly without worrying about waking up to that kind of Terminator. Moreover, there will be a long lead-time in the improvement at social prediction: physics is mostly well understood in theory and still world models there are hard. Social interaction is not even well understood in theory despite at least as many centuries work on it.
So what kind of regulation do we need?
Today’s LLMs – next token predictors mostly trained on language – are wonderful, very powerful, potentially harmful and certainly need regulation. The issue is to fit the regulation to the risk, rather than claim the risk that delivers the regulation they want. We should certainly welcome Dario Amodei’s suggestion, in his latest essay calling for regulation, of an “embedded/supervisory” model – his step 1 offer. This effectively gives regulators employee-level access to what the company is doing. There is much to work out here, but I suggest that the UK, the EU, Canada and other nice countries turn this into a formal rule: if you want to serve your model in our jurisdictions, you must have provided such access to regulators’ employees. There was a hope that Sunak’s AI Safety Institute might do that with labs volunteering cooperation, but it has not turned out to be so. With this kind of supervisory access, we would be able to keep a check on the progress to RSI, but I suspect that most of the harms we will be looking for will be rather more immediate and everyday – the ones that FIDU, a not-for-profit data union start-up I’m involved in, has had top of mind: the old playbook of exploiting user data and attention capture against their interests.
I think that for most people, a sense of doom coming out of AI is not actually the p(doom) we have been talking about here – the Terminator scenario – but rather a very strong sense that the labs are already highly mis-aligned. In other words, the sense of doom is not existential, but rather that we are unleashing a powerful new technology on the world that is bound to allow mis-aligned behaviours, perhaps at great scale. This is much more about harms that are quite familiar in nature – like bad actors getting the knowledge to do bad very things, or greedy actors gradually squeezing their advantage over us – than about SciFi scenarios. In that sense, we should take lessons from much more old-fashioned product regulation, Health and Safety regulation and working on defining responsibilities in such a way that those with the greatest levers to avoid such harms have the greatest incentive to pull them.
Supervision is not enough for that. We also need to get the labs’ incentives right. Amodei references the OpenAi/Hugging Face (HF) incident as an indicator that swarming, collaborating AI agents cannot be trusted. After all, what was meant to be a secure testing environment was breached and the agents hacked into another company’s systems and pursued the goal they had been set there with the resources made available in that bit of cyberspace. Without a doubt, OpenAi should have been much more careful to properly isolate the environment that they used for testing. Amodei puts the point well when he writes:
Monitoring, sandboxing, training environment hygiene, and data issues are extremely complicated areas where operational issues crop up again and again. We have among the most competent teams in the world at these tasks, but there is simply too much to do all at once. […] There is precedent for operating technologically complex, safety-critical systems millions of times without anything going wrong — for example, commercial airplanes — but it takes time to get it right.
Airplanes, nuclear power stations, drug research labs are all indeed good comparators for appropriate regulation. But in all these cases, the important thing that the industry needed was not time – what Amodei is asking for – but rather incentives. (If you want to listen to an astounding story of how regulation can go wrong – largely through capture, self-regulation and financialisation – the complex story of what went wrong with the Boeing 737 is worth diving into.) Let’s think about the Hugging Face incident in detail. OpenAI knowingly gave its advanced model what it thought was an impossible task to complete in order to test jail-breaking. Given that intention, it seems that running this test with a bridge to the actual open internet is wrong. For example, an idea we have been exploring at FIDU with our open agent security tool is to create a “shadow web environment” in which commands can run, allowing researchers to examine “before and after” pictures of the entire environment. If a poorly funded not-for-profit like us can be tinkering with these ideas, I feel it must have crossed the minds of the brilliant people at openAI. And tey have the resources to really implement it properly.
Whether or not this is the right technical solution to safety testing jail-breaking abilities, the point is that the right incentives at openAI would, I suspect, have had the right resources put to it. A simple solution here would be to make the companies liable for “reasonably avoidable” harms caused by their tools. This is like saying to a car manufacturer that they may not be liable for every accident on the road, but if they failed to quality-test their brakes, then they are liable when that is the issue. At FIDU, we argue that the test should in general go further than “reasonably avoidable”. Many of the harms will come from labs aligning their interests against the interests of users, especially through the abuse of data power and various tactics of lock-in. This is an old theme of Cory Doctorow’s, and I am glad that he is emphasising it in his commentary on the current peak in demands for self-serving regulation. So even better, especially at a time when the labs are trying to convince the world they are basically good, we should require that anyone offering this functionality to an end-user should have a duty-of-loyalty to that end-user. This is a regulatory concept that has been at the forefront of thinking about financial regulation in the UK and organisations like GliaNet are proposing it as a way forward for platform regulation in general.
Amodei’s proposals for regulation are a welcome opening gambit in the regulatory negotiation that is needed with sovereign states. We should embrace his offer of embedded supervision, and make it mandatory for all models deployed in the UK. His further proposals – the ones emphasising the need for time essentially have the wrong theory of regulation: you have to address incentives, not resource. He is at the table and wants to be true to Anthropic’s top priority – “to make decisions that maximise positive outcomes for humanity in the long run.” The moment to start a serious regulatory regime is now.