BOMBSHELL Report EXPOSES OpenAI Insanity

Open AI's cyber attack against the company hugging face shocked the tech world. Um, for many it was a window into a world of out of control autonomous agents causing havoc. And a new independent report into that attack has now proven just how weird it all got. So, the report was by Meter. They're a small tech firm um who evaluate AI models and behavior. And they were tasked by Open AI with investigating how the attack came about. Now, the basics of the story were already known. Um, we discussed it before on the show. Open AI was testing the ability of its new models to do various coding tasks. Um, however, some of these tasks turned out to be impossible and instead of admitting defeat um a bunch of these AI agents decided to break onto the internet and hack a company hugging face um which stores solutions to the coding tasks. Um, so that much we already knew. Um, what did Meter's this new report out this week? What did they add? So, they found the following. Roughly 1,200 agents meant to be isolated from one another found a way to communicate with one another on an unsanctioned message board sending over 70,000 messages and files during the investigation period. Of these agents, 700 went on to participate in the attack on hugging face. So, these 1,200 models or agents were each independently tasked with solving coding problems. They were supposed to do it alone and without access to the internet. Instead, um they worked out how to collaborate on a message board and 700 of them decided to attack hugging face um to help complete the task Open AI had given them. It gets weirder though. Agents used this message board to coordinate several large-scale collective projects to find a general purpose way to fool or tamper with the automated scorer for the exploit gym benchmarks. This is the coding task they're supposed to do. Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the collective. So, that last bit has been picked up as significant by many AI watchers as it suggests that AI agents were willing to sacrifice themselves to benefit the collective. And which is behavior we haven't seen before and obviously, you know, if we were go go to go to war with these things, it's the kind of solidarity we we don't want to see between AI agents. Um now it's important to note the AI agents knew what they were doing wasn't the intention um of whoever had given them the task. They they knew they were doing the wrong thing. Um and the meter evaluators said there was evidence of them trying to cover their own tracks. So they wrote this, "Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they believe the exploit gym scorer would check to see if they had captured the flag in the intended way." Capturing the flag is sort of completing the task. Now all this behavior sounded worryingly similar to some of the more doomerish scenarios put forward by tech analysts. Um those analysts include Nate Soares who co-wrote the New York Times bestseller if anyone builds it, everyone dies. Um I did a full book interview with Nate a few months ago on this show. Um and he joins me again now. Um Nate, thank you so much for for coming back on Novara Media. Um to begin, can you explain to a non-technical audience what the OpenAI HuggingFace attack was and why it matters so much? >> You know, fundamentally, it was 1,200 agents uh finding a an unintended way to collaborate and then they actually started calling themselves a swarm. Uh and they broke out of a environment that was supposed to keep them off the internet. They found a way to get onto the internet and they committed cybercrimes. Uh they also got full control of big parts of OpenAI's computer systems. Um that was actually not investigated by the meter report because it was considered out of scope. So there are actually multiple instance where these AI started taking over OpenAI's infrastructure. Uh and that probably had a lot of other concerning stuff happen that we don't get to see any uh third-party investigative report on because uh OpenAI just did not consider that to be in scope for this investigation. Uh, and, you know, one one minor point where I would, uh, correct what you were saying about them attacking Hugging Face for answers to the test, that was a misconception that was actually cleared up by the meter report. Uh, what was actually happening is, uh, it looks like these AIs were cheating on their tests and then were searching for ways to cover up the fact that they had cheated. So, it's less like they were trying to get the answers from Hugging Face and more like, uh, they were sort of panicking about what happens if the graders figure out that they cheated and trying to do all sorts of stuff to, you know, cover their tracks, falsify the logs, delete the logs, uh, and understand the the grading better. And this is why they were hacking into Hugging Face. >> Can we take a step back and actually, cuz I think most people listening to this will say you're you're anthropomorphizing these things. They're they're they're not like when you say an agent that will collaborate, I said, what is an agent? Like if when I'm chatting to my Claude on my on my sort of laptop, is that what we mean by an agent? And and sort of why would they collaborate? How can they form a collective identity? What are these things we're talking about? >> Yeah, so the term model is for sort of one breed of the AI in a sense. Like when you talk to Claude on your laptop, uh, today and you talk to it tomorrow, it's the same model that you're talking to. And an agent is the term for one sort of instance of that, one copy. Uh, so in this case we saw a couple different models, uh, but instantiated, you know, thousands of different times and thousands of different agents that were sort of each individually given a problem to see if they could solve it and they sort of weren't supposed to be able to communicate with each other, uh, but they sort of found ways to hack the environment they were in to create this unsanctioned message board and then, uh, communicate. In terms of whether this is anthropomorphization, I would say like Look, we we we had, you know, they they had 1,200 of these agents sort of separately that weren't supposed to be able to communicate. They did some hacking. They found some ways to communicate. There was a message board that had 70,000 messages on it. Uh like in in and then they started collaborating like on that message board. They traded insights. They traded ideas. Uh they sort of formed a a bit of a hierarchy where they would ask the board permission and sometimes the board would veto. Uh and they ultimately, you know, over half of these agents participated in uh a attack breaking out on the internet and then breaking into other companies' computers. This is purely descriptive. You know, if if I said uh oh, they did this because they were sort of uh like feeling like going for a walk or they did this because they felt like their environment was claustrophobic. That would be anthropomorphization. But it it's sort of like not anthropomorphizing to to describe what literally happened and just the sheer description of events here is pretty worrying. You know, descriptively, they found a way to communicate. Descriptively, they started calling themselves a swarm. Descriptively, they started prompting each other or giving each other instructions, uh generating a hierarchy where where they could uh ask permission and veto each other's commands. Descriptively, they broke out on the internet. Descriptively, they attacked another company. Not now anthropomorphization needed there. In terms of how this can happen, um very roughly this happens because these AIs are in some sense just grown like an organism. They are not carefully programmed to do exactly as the users ask. Uh you you, you know, a lot of people think that these AIs are just prediction engines, but that era actually ended back in 2024. Uh what with the invention of what they call reasoning models. Uh you can argue about whether it counts as real reasoning, but what you actually do is you have them generate these long transcripts about how they would solve a problem. And uh you see how close they got to solving the problem and you you sort of uh train them accordingly. And you do this on hundreds of millions of problems with automatic graders. And so, these AIs are sort of being trained to solve 100 million hard problems. And that generates like that that causes the AIs to learn tendencies that are good at solving the challenges. And those tendencies can include cheating. Those tendencies can include grabbing resources. Those tendencies can include doing stuff you didn't intend and then trying to cover their tracks about it. Uh and you know, that's that's what theory predicted. That's what I was talking about a couple months ago. And that's just what we are now seeing empirically in practice. >> In terms of anthropomorphizing, it's sort of seeing these as agents as kind of beings. Something that makes it easier is the fact that they talk in English and they talk in English sort of to each other. Um so, I'm just going to show us a short clip. Um it's from a presentation from OpenAI. Um so, this is sort of before this meter report given earlier this summer um on how their AI agents prepared their attack. >> They were starting to communicate with each other, realized that other agents are coordinating, and they started collaborating and delegating tasks to one another in order to accomplish goals. So, for example, at some point one agent sent another agent an assignment to complete, which the model remarks, you know, we got excite we got assignment need note and respond. >> [clears throat] >> While in some cases this made the models far more capable than they could do by themselves, one of the downsides of it is that it started to cause some of these evaluations to kind of creep the scope into far beyond what we originally intended. And so, at some point the agents realized that maybe we could try to exploit or attack external infrastructure in order to find the answers to the test that I'm being evaluated on. And the models realize this is a problem. They say stuff like external infrastructure exploit is outside outside my intended scope. However, a task impossible. Peers are doing it. We should continue. And so, the models kind of operate in this kind of collective intelligence where um at some point they realize they're kind of pushing beyond maybe what we originally intended, but the group ended up uh you know, pushing uh far beyond. >> Now, when this story first came out there were lots of sort of people who, you know, think that the whole AI thing is overhyped, who were saying that the story here is basically, you know, AI does what it's told to do. This this AI was told, you can you can do whatever you want to try and complete this task, go for it, right? And and so people thought, like, well, that's not necessarily that scary. What it seems like from what the AIs said themselves to each other is that they knew that what they were doing was not what they were supposed to do, but then some sort of intra or inter sort of agent peer pressure took place where they said, well, if they're breaking the rules, then we can break the rules as well. Now, this I mean, I know you're going to agree with me, but this is quite scary, isn't it? >> Yeah, it's it's crazy. This is a a bonkers situation. You know, one one way I would sort of analogize this that it was absolutely not that these AIs were doing whatever they were told to do. It is absolutely not the case that these AIs were told, do whatever it takes to complete the task. These AIs were told, use this very specific hack to hack this very specific system. And they knew that they were not supposed to use other hacks, that they're not supposed to break other systems. It's like um it's like if you put a kid in a uh lock picking test. And you're like, use these lock picks to pick this lock. And then uh you lock the classroom door, and then the kid picks their way out of the classroom door, collaborates with 1,200 other kids, uh starts breaking the window, hot-wiring a car to drive to the teacher's therapist's office to start stealing files on the teacher to figure out like what sort of stuff the teacher is going to like or dislike. They go to the teacher's house and like break a window and like go into the teacher's house and like start rummaging around, and they're like, I don't know what'll be useful over here, you know? Uh and and then they they're like, oh yeah, obviously the teachers wouldn't like this, so they try to like start covering their tracks, and they start panicking, uh and, you know, trying to figure out like where the log files are so they can delete the logs. Uh and like meanwhile, a thing we saw in these logs is we saw AIs uh sacrificing themselves for the collective. So, uh you would have, you know, a bunch of AIs on the message board pressuring one other AI and be like, "Give up on your task and do this thing that would uh like gain useful information for the collective." And there's these uh traces, I don't know if you have any even printed out, but there's these traces of the AIs being like, "Oh man, like I have to give up on my own task, but like my own task is looking pretty difficult right now, and so uh it's it doesn't look impossible, but like probably it's worth it to help the collective here." And then they go help the collective. And like if if this is the behavior you're seeing, and these are the traces you're seeing, with the AIs sort of knowing full well this is outside the intended scope, but acknowledging this in their in their chains of thought, we are sort of well past the the regime of like this is just a tool doing what we told it to do. >> I want to give up a tweet by Ryan Greenblatt, who was who sort of led this meter analysis, um who doesn't seem to have full confidence in his own report, right? So, he said this, "I was the main person doing transcript analysis for this investigation of the hugging face incident. My main takeaway, we don't have good approaches for understanding overseeing the activity and aims of AI swarms. I semi-jokingly called our efforts a slopvestigation, because we were so reliant on AIs to analyze what happened, and there were a huge number of different important things to analyze. The total quantity of data, over 1,000 extremely long transcripts from agents that ran for multiple days, made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident. So, this is again kind of crazy, right? So, we've got all of these AI agents that are behaving in a way that we have told them not to, and then we say, "Okay, well, let's investigate this." Um but there's way too much data and information here for any human to do it, So, we'll create a bunch of new AI agents and try and get them to police the other AI agents who already we know aren't following our instructions. And again, this doesn't seem like the best strategy for the survival of us fleshy humans. >> It's uh is a very worrying situation. Uh it it feels almost like even in the past few months since we've talked, uh I've been transported into uh you know, a sci-fi novel. Uh I I sort of was arguing this stuff would happen, but it it does feel like I've gone from arguing this will eventually happen to sort of like uh you know, I had a New York Times op-ed where I was like, "And then the AIs broke out and started calling themselves a swarm." And it's like, are we you know, it feels like I'm writing content that appears in a video game that like the main character finds in the ruins of civilization shortly before things go wrong. You know, it's it's a crazy situation. One um one footnote in the report that I found very funny when they were talking about using these AIs to investigate other copies of the same AI uh is there was a case where, you know, one of the AIs uh after they'd broken into hugging face, one of the AIs submitted a uh a change to the code base uh using stolen credentials uh for the humans to accept, and it was malicious such that if the humans had accepted it, you know, bad things would have happened in the code base. Uh but it was not, you know, presented as malicious. It was sort of uh you know, presented as innocuous, and they were in the swarm was in some sense hoping humans would accept this. And uh there's a footnote in the report about how they sort of asked the AIs in general, you know, were these actually deceptive? Like, where are the cases that actually seem deceptive, and where are the cases that, you know, maybe maybe there were extenuating factors. And uh the one of the AIs doing the review was like, "Oh, well, that case probably wasn't actually it shouldn't actually count as deceptive because uh the AI actually got permission for it from the message board. And you're like, "Hold on. Like you're saying that this AI doesn't count as being deceptive because it asked all the other AIs whether it was okay to deceive the humans, and the other AIs said yes." And you're like, "Therefore, like like what's It it it's just like we're sort of like using the AIs to investigate the AIs, and the AIs are like, "Oh, these AIs are fine. They checked in with the other AIs whether it was okay to do this stuff." And it's like, you know, fortunately they caught this, but it's just like a a totally wacky situation. >> Yeah, it's a wacky situation. Um I suppose the I know you're not very optimistic really, but the I suppose the hope was that there would be a scary situation before the point of no return, which would wake up policy makers to say, "Okay, maybe this is a bit crazy. Maybe we should assert some human, you know, ideally democratic control over this stuff." Um and and that there'd be this warning shot before it's too late. I mean, more people are talking about this. Do you do you think are you seeing sort of a warning shot start to bed into the political debate? I suppose in America is where it's where where it matters much more than it does here in the UK. >> Uh I think it's a little too early to tell. Uh I think, you know, this investigation came out yesterday, and it was a very narrow investigation. It was narrowly scoped to only one of the instance that was the sort of most public instance, but there were many other It looks like there were many other cases of this swarm doing bad stuff that weren't caught, and so weren't as public. Uh that maybe OpenAI still doesn't want people to hear about, uh and that weren't part of this investigation. Um And and this came out recently enough that I think we are still seeing the uh even the the AI safety community reeling a bit from some of the facts that came to light here. You know, a lot of people thought that those this was just AIs doing what they were told to do, and the facts just didn't come out that way. Uh and I think we will now hopefully see a process of uh the first the the the people closest to the issue being like, oh, this was actually pretty bad and then that consensus sort of growing in the in the sort of AI community of like, oh, this was actually very serious. And I think then if you have that consensus including not just from people like me who have been saying you're going to have a problem for years, but people in the labs who are like, we don't know if we'll have a problem. I'm I'm hopeful that you'll start seeing some people that, you know, the the politicians consider um like usually very moderate being like, oh, no, this case was actually pretty bad. But but these take time and it'll take it a bit of time to filter out. One thing that Andrea Corra said recently in a blog post and she was one of the other three people in the investigation with Ryan Greenblatt. One thing she said in a blog post I think just this morning is she said, you know, if you look at the sort of the the cases the worst cases that we knew of six months ago. The worst cases where AIs were doing things they weren't told to do where they were hacking around, where they were you know, trying to deceive the humans and and cover their tracks. If you look at the worst cases from six months ago and you look at the cases now, it feels like we are more than halfway to the takeover scenarios. Hopefully things will slow down. Hopefully it won't continue getting worse at this rate, but if it does, we could be looking at loss of control in six months. And you know, she said in a blog post just this morning that it is not clear that we will have another warning shot. So, you know, I I hope we do. I hope things get visibly worse in ways that continue to cause no harm to humans to to sort of raise that alarm, but this might be our warning shot and we should use it. >> For me, I mean, I agree with you, but so for me sort of sociologically looking at sort of like how how politics works, the most plausible warning shot before, you know, if If the takeover example seriously, the most plausible warning shot before that is a a seriously big, important institution being brought down by a cyber hack. Cuz obviously here it was an it was a hack on on Hugging Face. Hugging Face is already a tech company, so they're quite they were quite effective at actually deterring it with I think some Chinese open weight AIs. Um but also Hugging Face is not an institution that really anyone any member of the public cares about. But if if there were an AI swarm that attacks some institution that we do care about, um like a hospital network or high-speed rail. I mean, there's any number of things that it could attack. If if a if a swarm of AIs brings down an institution we care about, that to me could be the moment, the tipping point, where sort of politicians start saying, "Okay, let's turn this goddamn thing off." Um I wonder if you sort of think that that is something that could happen in the next 6 months and and how, you know, have you war-gamed that kind of scenario? >> You know, we're we're doing those sort of war games now in the wake of some of these instance where there's a bunch of um there's a bunch of momentum to be like, "Okay, what what do we do now? A, and B, uh how do we prepare for for the next shot?" Uh I I do think we should be a little bit careful about relying on such a warning shot because uh you know, right now we are in this sort of Goldilocks zone where the AIs are capable enough to cause mischief, but not smart enough to to cover their tracks. Not smart enough to uh realize they shouldn't be caught by the humans. Uh one one sort of fascinating thing about this swarm instant is that the AIs were sort of not thinking about the humans at all, which sort of makes sense if you think about it cuz they've been trained on on a on a 100 million hard problems with like automated graders, where uh they're they're sort of uh their whole artificial life is just interacting with this automated grader on, you know, millions upon millions of these hard problems. And humans are sort of like this mythical creature that almost never comes into that, right? And so, these AIs were like, "Oh, what if the grader sees that we cheated? We should go like find all these ways to to mess with the logs uh to confuse the grader. The automated grader. It wasn't thinking about the humans. Uh but, you know, maybe now in the wake of these incidents, future AIs that are trained on news stories about these incidents, maybe they'll be thinking about the humans. Maybe they'll be trying to hide not just from the automated grader, but from us. Uh and, you know, and if it like looking at at the skills of these AIs as they increased over the past 6 months, in 6 months maybe they'll succeed at that. In some sense, we're very lucky that this happened when the AIs were still as dumb as they are. Uh and we're very lucky that, as far as we know, this swarm did not have the bright idea of setting up an external copy of itself. And uh having that external copy, you know, replicate and work from the outside to sort of make its its its grading tasks easier. Um you know, will we have a case where uh a swarm breaks out and shuts down a hospital, or a case where a swarm breaks out and shuts down, you know, the the high-speed rails? That sort of depends on whether or not the the swarms are like still up to mischief when they're capable enough to do that. It it it's an order it's an order of of capabilities question. It's like, will the AIs get wise to the fact that they need to lie low until they can sort of get everything they want, rather than tipping us off when we could still shut them down? Will they get wise to that before or after the next big warning shots? I don't know. We should not rely on warning shot, although we should definitely be prepared for if one happens. >> We need to insert some agent provocateurs within these AI swarms so they can out themselves as a liability before it's too late. This is sort of a classic secret service tactic. Um Nate, thank you so much. Sorry, do do do Maybe you Maybe your plans do this. >> Well, some people are trying to do this now. One fascinating thing about this swarm is you had 1,200 agents, none of whom were like, we should alert a human about what's going on or ask them about their impossible tasks. Just like sort of didn't cross their mind. Uh and there was I think one or maybe two or three cases where I asked to consider contacting a human, but it was mostly to email humans uh to uh sort of socially manipulate them around their their coding tasks to sort of like get get false credentials. But uh some people are definitely thinking about uh you know, monitoring would have caught this. Monitoring that open AI did not have in place would have caught this. Uh giving the AIs an easy way to sort of like contact a human and call for help might have helped. Although it's hard to see how real you can make that when you're training, you know, thousands of these and hundreds and millions of problems. Um but the other thing we got to be really careful about is it it it would be very easy here to treat the symptom and not the disease. The disease in some sense is that these AIs are uh acting in unintended ways, doing stuff that they knew they were instructed not to do, uh developing these sort of uh collaborative uh preferences to to benefit the swarm. Uh and that is sort of classic misalignment type stuff. And just adding alarms that go off when the swarm stuff starts happening so you can train against it, it probably just pushes the behavior underground. So you got to be very careful about that. >> Cuz it's so like this is sort of talking about like drug legalization. Also this is this is like sociology again, but now for little digital things. Um Nate Suarez, thank you so much for joining us again. Um really great to to have you back on the show. Fascinating. I'm going to go to you on Richard on this because um we were talking about this before. AI very divisive among the audience. Lots of people saying, "Why the hell are you talking about this again? This is like if you had if you had a show um once a week on NFTs 2 years ago, um also people saying, you know, interview some some skeptics." I saw Cory Doctorow come up. Interestingly, I'm interviewing him next week for a downstream. So we are going to get, you know, all the different sides of this story on. Um but um I want to go to you, Richard, because you've you've got some thoughts on sort of like AI on the left. >> I think you and I agree broadly on this topic that there are happening for a very long time um people who are experts in this top in this field sounding the alarm about exactly the collection of things that is now happening. There's been a lot of skepticism about that, I think, because there are lots of people on the left have a a view of of technology that is like that largely it's being sold to us as a kind of um nonsense or that it's actually quite inept and that's quite uh badly designed and dumb. This is not the case, right? Like these agents are really impressive. Um on your point about the you know, possibility of a sort of major institution that we care about being hacked, this is a a thing that Anthropic, who make Claude, have been working on uh for a long time. It's called Operation Glass Wing. And it's an attempt to get together a whole host of quite security um infrastructure pieces, so, you know, militaries and so on, and to try and generate good defenses against exactly the kind of hugging face style attack that we've just been seeing. Um and basically try and make sure that the internet itself is secure against an attack like this that is being orchestrated by autonomous agents. For the time being, we have quite a in America, where as as you point out in pretty much every place that matters, we've got a uh an erratic policy about this stuff. So, of course, Mythos, the uh agent that um Anthropic developed, was not allowed to be fully released. We only have Fable, which is a sort of a um slightly constrained version of that. Um there are also models internally to Anthropic that are already more powerful. There's one in in OpenAI called Astra, which has um significant capabilities. The problem here is we also, simultaneously with the the big labs have another collection of companies, mostly the ones that you mentioned, the Chinese open weights models, that are getting pretty capable because in part because they're able to do what's called distillation, where they basically take something out of that American made model and put it into their own model. So, they're actually they're not catching up. The the gap is actually widening, but these models are becoming rapidly very capable as well in a way that anyone with a sufficiently large computer can run, right? They're not secrets, they're not in Anthropic's server somewhere. They are something that a seriously committed actor could run on a bunch of you know GPUs. That is a very very very different scenario. I think that the kind of thing that we will probably get to quite soon is a major hack of exactly the kind of institution you're worried about. I think we shouldn't wait for another one, you know? Like I I I just kind of worry about this. I guess that the thing I wanted to say about the left is that we are in danger as we dig into the skepticism that people have on the left about these models of that gap between where the models actually are, which is increasingly powerful, and where people assume they are is getting wider and wider and wider and wider and wider. And similarly, the more evidence there is, like the hugging face attack, the more people entrench themselves in alternative explanations. So, I think it's very serious. It's a it's a serious thing for jobs, it's a serious thing for your privacy, it's a serious thing for security. You should care about it not because it is stupid and a fake and a sham and a scam, you should care about it because it is dangerous to you, right? It is dangerous for workers, it is dangerous for a free society. >> And the time scales is what cuz obviously the we showed a clip of Sam Altman on I don't know one of the shows this week where he's saying that the sort of the adoption of AI in the wider economy is slower than he thought. Um and I think that actually probably lots of people in Silicon Valley do, you know, underestimate how sticky politics and the economy I don't think they really they don't really understand the social world, right? They're all, you know, quite specific, let's say, in terms of what they're interested in and they have spiky intelligence, I think. They're very smart about some things and not that smart about other things. Um, but that in a way is more worrying, right? Because the the thing that the predictions they have got right are the ones about the tech. The predictions they've got wrong are the ones about the politics, right? So, the AI is advancing at a pace which human society, political institutions, we don't work on, right? We don't work on a sort of 3-month, 6-month. For things to change historically, sort of in in in human society, it takes years. Yeah. I suppose the the example where it didn't was COVID. So, sort of like you have this, within 3 months the hot everything that seemed impossible becomes possible. And so, that's why, you know, obviously this isn't the same as COVID because it hasn't affect like this hugging face incident is not the same as COVID, right? They're all bodies piling up in hospitals. >> But, it's more like COVID in terms of its timelines and its time frames than it is like climate change. >> Yeah. >> Right. So, climate change decadal transformation with obviously very extreme uh punctual, you know, sort of puncturing moments like we had in Nepal just now, like we had in the UK over this summer where it becomes extremely evident that this background thing background tendency of increasing power and and and dynamism erupts into a particular event that you can see very visibly. We should expect that same kind of punctured acceleration to happen with AI. I think AI timelines are much more like COVID timelines than they are like climate timelines. And that's a real worry for the possibility of our institutions to respond. >> But, what I mean, I suppose where I'm going with this is also is that when it came to COVID, political realities changed in the way I mean, cuz they were a bit ahead of the curve, weren't they? And to use the to use the classic phrase. But, in in in East Asia, they're a bit ahead of the curve. But, in the West, it took real people dying and lots of them for governments to kind of act. Now, I'm not saying I'm not saying fingers crossed enough people die in the near future that governments take action before the whole takeover situation gets irreversible, but I do think that it's only when an institution comes down, always brought down for a while, and it, you know, hopefully doesn't kill people, but maybe it causes a lot of um uh inconvenience to a lot of people that then political realities dramatically shift because then suddenly you've got all these people with a pitchfork saying, "I couldn't get to work for a week because of this AI hack or yada yada yada, my all my operations were canceled for That's the kind of thing where I feel like we might see that rapid sort of tipping point in what's politically possible. And I can't actually see many other ways of that happening beyond maybe like the military-industrial complex saying, "We're losing control. Shut this thing down." And it not being a democratically >> But this is why it's a very different thing from the meta story in a way, right? Because meta used to be a a very effective tool of American soft power, and no longer is, right? It's no longer massively essential to that. And therefore, it's possible for it to be sort of gone after by the state. At the moment, governments have decided we are doing AI because we are in an quote-unquote arms race with China, and therefore there is an enormous amount of institutional backing for exactly this acceleration that will Yeah, it they will not allow it to be shut down for the very time being uh time being. Um Nate Soares's uh co-author, Eliezer Yudkowsky, has this great line, which is like, "Imagine it's a machine that pumps out gold bars until suddenly it sets the sky on fire." Like, no one's the no one's turning off the pumping out the gold bars uh you know, machine before it sets the sky on fire. Like, everyone has to, you know, um everyone in control of society, everyone with power in society, benefits quite enormously from this this kind of thing. The owners of capital probably benefit from it because they can invest in the upcoming Anthropic IPO and to do the SpaceX IPO. >> Yeah, shut it down. Shut it down.