China Wants A Deal To Stop AI Apocalypse

We talked on last week's show about the open AI hack of hugging face. It involved hundreds of AI agents breaking out of a testing environment, collaborating on a message board, sacrificing themselves for the good of the AI collective, um, and in hacking another company, ultimately committing quite a serious crime. Um, in short, it was exactly the kind of situation the AI doomers have been warning us about. So, how have Open AI responded? Well, they've released to the public their new more powerful model. Um, it's called Chat GPT6 Astra. One thing that I think people will immediately notice is if you have an idea and you want to work interactively with an AI to get uh, you know, a complex piece of a whole piece of software built. This is the first model to me where I could sort of tell someone like just give it a try. There's a there's a good chance it'll work. And, you know, I've I've watched people make computer games. I've watched people do sort of like home sort of DIY electrical engineering projects. Uh I've watched people do very complex simulations for some piece of science they're working on. Well, that all sounds lovely. What a useful piece of technology that we can now all download and if we want pay 20 a month to Open AI for. Um, however, what if this new powerful model also takes us one step closer to a world where we lose control of the technology? Um, and what could the next hugging face look like? Well, the experts are very much concerned. Ryan Greenblat led the meter report into the hugging face incident, which we reported on on last Friday's show. Um, this was his verdict on Astra. So he says, "GPT6 Astra appears to be a massive jump in opaque reasoning ability. It looks like it can solve hard competition math problems entirely in its head, as in without verbalized reasoning, while prior AIs could solve basic word problems. This seems extremely concerning. So if you remember from our report last week on the hugging face attack meter which is this sort of AI safety firm they could trace what happened at open AI because the the AI agents reason in English they have a chain of thought. So they can say I'm thinking should I do X or should I do Y. Well actually maybe B will get me hit. So you can sort of see what was the thought process they used when they took their actions. You can look at their transcripts to see what they were thinking. Now in this more powerful model that's become harder to do. So, we sort of we've said, well, these were very dangerous. Let's make them harder to interpret. That's basically what's happened. Um, OpenAI have admitted that themselves as well, so they're not denying this. Um, except they say it's not a big deal. Doesn't really matter. Um, so this is from OpenAI. According to our evaluations, GPT6 Astra shows a substantial decrease in chain of thought monitor compared to previous models. Overall, our alignment evaluations show that Astra is more likely than chat GPT 5.6 six soul to respect security and safety restrictions which make us confident in still deploying this model to the wider public. So they're basically saying we understand this model much less than our previous ones. We can't see what's going on in their heads. However, in our tests it hasn't acted that bad, right? So who knows we don't understand it but it seems to be better behaved. There's a catch here though because the reason they say it's better behaved is because they've done evaluations of this model. They sort of tested it in a testing environment. Um but the models we've been told um by the UK AI safety institute they often worked out they were being tested right so this is again what the doomers have long warned about right the AIs we don't understand what's going on they could and I should emphasize could because this is quite speculative they could be lulling us into a false sense of security right in the test in the evaluations they say oh we'd never do anything wrong but we can't read what's going on in their minds we could in the past and now we can't Because if if you're asking why would you have stepped backwards on that? I think it's because that could be seen as a waste of time, right? The most efficient way to have an AI is for it to work out what to do in its own language. It's been called for a while neural ease. So sort of instead of them speaking English, they speak their own language, which is maybe more efficient. So that can make them more powerful. It also makes it harder for us to understand what's going on. And if you don't want the big clever machine to take control, I think it would be helpful to be able to read its mind, which we can't do anymore. Anyway, this all suggests to me it's well past the time for governments to get involved. And once again, Senator Bernie Sanders is leading the charge. We need an immediate pause on advanced AI development, and a permanent ban on super intelligence, an artificial mind smarter than any human capable of operating independently beyond our control. Countries around the world must work together to prevent this nightmare scenario. That is why I am announcing that as soon as Congress reconvenes, we will be introducing legislation to do just that. Let me be clear. A super intelligent AI that escapes human control will not be an American problem. It will not be a Chinese problem. It will be humanity's problem. The legislation that I'm offering would direct the federal government to not just stop super intelligence here in the United States, but work to prevent it from being developed anywhere around the world. The future of humanity cannot be left in the hands of a handful of big tech oligarchs. The American people and people throughout the world must determine that future. >> I love Bernie Sanders. I mean, he's he's so ahead of the game on this, right? Lots of people were saying politicians can't really talk about this because it seems too wacky. It seems like sci-fi. Bernie Sanders, that was a clip from a 7-minute video I showed you. He sort of quoted, he's like, Dario Amade, the head of anthropic, says we can't control this. Sam Alman, the head of open AI, says we can't control this. And he reads things from the meter report. or he's saying, "Look, I'm not a computer scientist, but this seems goddamn crazy to me and we should shut it down." Um, also watching that made me feel even more like when I'm reading the meter reports or whatever, like I was in a sci-fi movie, cuz that's such a scene that you would get in the sci-fi takeover movie where there's a a politician who's ahead of the game, who's speaking to the public saying, "It's time to shut it down." Obviously, in that sci-fi movie, everyone says, "No, he's ridiculous. He's talking crap." Um, of course, also by mentioning China, which you heard Bernie Sanders there say, um, he was preempting um, the criticism that would come his way when calling for an AI pause. So, the billionaire Trump donor Bill Aman said this on Thursday in response to Bernie's call, would Bernie prefer our enemies to get to super intelligence before we do? Who votes for this guy? And that's a position the Trump administration has taken in the past. Do you do you think that the US government is capable in a scenario again not like the ultimate Skynet scenario but just a scenario where AI seems to be getting out of control in some way of taking a pause because for the reasons you've described the arms race component yeah the honest question of that is I I don't know because part of this arms race component is if we take a pause do the does the PRC not take a pause and then we find ourselves you know we're all sort of enslaved to to to PRC mediated AI. >> The context of that interview, I think we did a story on it at the time, is JD Vance has has had read the report AI 2027, which was basically saying the AIs could take over and kill us all. And he took it very seriously. Like he didn't really have a critique of it, but he says, "Yeah, I mean, maybe the AI could take over and kill us all, but also I'd prefer it to be American the AI that kills us all than Chinese. I know nothing more shameful than getting killed by a a communist robot. I'd much prefer to get killed by the liberal capitalist one, please. Um, as I say, that was JD Vance speaking to the New York Times last year. There are though now potential signs of a change of mood. So, Helen Toner, we've shown you her before as well. She was on the board of OpenAI. Um, she was one of the people who tried but ultimately failed to fire Sam Alman because he's a dishonest, lying soap. Um, she thinks there might be some hope for greater agreement between China and the United States. The US companies will say, "Hey, we have to keep pushing otherwise China will will win this race. What exactly it means to win the race is is a longer conversation, but the China argument comes up a lot. And we actually have Trump and Xiinping planning to meet in September in the White House. And this is crazy to me as someone who has followed US China relations for a long time and also AI for a long time. AI is right at the top of their agenda. That's really interesting. Is there something that they can say to um create an understanding that we do actually have a little bit more time and space here? Whether it's a you know each leader uh sharing an a plan to domestically look at what their industries are doing and and ask more questions. So, if the Americans were interested in a deal to slow down AI, which is a a massive if, you know, I I don't think this is a Donald Trump priority, but if they were to come around to a rational position and say, "We need a deal. We need to do something about this. It's getting out of control, would then the Chinese government want to play ball?" Um, really interesting question. Carl Chan is a fellow at the Brookings Institute, an expert on the Chinese tech sector. um and he runs the high-capacity newsletter. >> Actually, yes. I think some people would be surprised, especially in the US, that China might be willing to not just talk about some of these issues, but do something jointly with the US. I think the time right now is ripe because both countries are changing their minds about um AI risk and AI safety. Um the uh technology is moving very very quickly and we are getting these major incidents. Um I mean so far it hasn't seemed to you know the hugging face incident for example it didn't cause actual economic damage or or worse but I think it doesn't take a huge leap to start to see how these systems can cause real damage. Okay, that's really interesting actually because um I've sort of seen interviews of you in the the recent past where you've been suggesting that China isn't actually as concerned about these issues as the United States. Not because they're sort of they're more happy and open to have a dangerous AI, but they just didn't quite see AI in the same perspective as it was seen in Silicon Valley, which is as this, you know, potential godlike being. Um the argument I've seen you make before is that China wasn't AGI pilled. Is China now getting more AGI pill? They're starting to see this as a not a normal technology, something which is quite extraordinary, quite different. >> It's something in between. So I still think that the leaders in Beijing are not AGI pill in the sense that we think about say in the United States where they expect super intelligence to be around the corner uh completely transformative in every single domain of human life, maybe even wipe us off the map. Right? That's literally the kind of talk that we hear from Silicon Valley from uh maybe increasingly in Washington. I don't think that's what people in Beijing think. I do think there are some people in the Chinese AI ecosystem who are worried about you know so-called catastrophic risk. Um but I do think there's a sort of middle ground um which is still quite alarming and that I think Chinese policy makers are taking more and more seriously and that is the risk of major cyber incidents. the risk of the technology sort of getting out of control um going rogue doing things to important critical infrastructure, important uh information technology systems that have real world impacts and that now we are not sure if we can trace that back or much less uh uh reign that back in once it's out there. >> Has the hugging face incident in particular had an impact in in China because it's I mean it's had an impact in the west quite clearly. I think it has um it adds to a ongoing list of indications um from China that they are taking some of these issues more seriously. Um so not so long ago, China re released a AI agent framework. Half of that was about accelerating AI adoption and agents everywhere. uh how it can improve the economy, but a pretty good chunk of that was about um responsibility and control over agents. And um there's not a surefire way to deal with this. I think both China and the US are sort of groping around trying to figure out what what a good regul regulatory system should look like, but that was a big signal that it was already on Beijing's radar. Also recall earlier this year, OpenClaw was a really big trend in China. It was huge. It's this um basically AI agentic harness that allows people to you know run AI systems on their own computers at home and that can do really wonderful things. It can also sort of get out of hand and leak data or open your your computer or your IT system to uh cyber intrusions. So that's something that um China's cyerspace regulator was also starting to warn about. So I think that the hugging face incident is kind of the latest and perhaps the the biggest example of what could happen and how Beijing is starting to shift. >> So So you've said that China actually might be quite open to this kind of thing sort of reaching some kind of agreement whether formal or informal. Um Trump and she are meeting towards the end of this month I think on the 24th of September. Um what kind of thing do you think could come out of that that meeting? So I have relatively low expectations for this very first meeting and I think that we should have low expectations because I think the first time we had an official USChina AI dialogue under the Biden administration expectations were a bit too high. I think there was maybe some hope of even binding constraints um some broader AI agreement and when we didn't get that in the first try we felt like well this is kind of pointless let's give up and I don't think that this will be solved in one round of dialogue so the key first step is to start talking to start opening those channels of communication to start sharing notes at least about what happened with hugging face what are Chinese AI models doing are there incidents in China that um Beijing should or could share with the United States. Um, are there communication channels even between the AI labs, the frontier labs themselves? So, I think that would be a helpful first step. Um, eventually I would like to see us move up the ladder in terms of what the two countries can and should do together, but um, starting to talk about this is actually very important. >> And my concern is there's going to be sort of a switch between two arguments from the west and I suppose AI accelerationists on the west. the first to say we can't possibly slow down our AI development because then China might overtake us. But then I imagine they'll also say we couldn't possibly make a deal with China because China only wants to sign that deal because we're ahead of them and obviously these two things are slightly inconsistent. But I imagine that's going to be sort of the default position of of Washington. I mean is that something that you see as as likely? Would there be a way out of that? >> Yeah, the big problem here between the US and China is trust. Um, for the technical folks, you know, who want to see better AI safeguards, better AI guard rails, um, better safe pre-release testing, I think that there's still this geopolitical dimension to all this that we can't forget about, which is the US is very worried that, um, if we hold ourselves back, then China will race ahead. And China is worried that um some of this discussion about AI safety and AI regulation is not about AI regulation per se, but about trying to slow China down or trying to keep China out. Um so you see commentary from sort of the Chinese state media ecosystem where they're they're concerned about this being an excuse uh along with other things like export controls to to slow China's own development. So you can see the deep distrust uh from both sides. Even in the face of that, I would argue that both countries have a very strong national interest incentive in trying to figure something out here. Neither country wants their own models to be attacking their own systems. I think leaders in Beijing, the last thing they want to see is a Chinese open source model being used to hack a Chinese government website. Um, and so I think there's a lot of overlap actually in just pure sort of national interests. And I think that can escalate to uh something something bigger than than merely um standing at arms length. Another argument I've seen as to why um America doesn't need to wait to slow down because um you know the argument being you we can't slow down or the Americans can't slow down because then the Chinese will overtake them is people say that actually the pace of Chinese development is is kind of parasitic on the Americans because the Chinese models or the most advanced Chinese models they're often created by essentially reverse engineering the most advanced America models via a process called distillation. Um, is that your understanding of what's going on or do you think that the Chinese in theory could um overtake the Americans? >> The Chinese could in theory overtake the Americans. Um, distillation is a factor, but I don't think it is the main driving reason why you have Chinese AI models performing so strongly. And keep in mind, we have, you know, multiple Chinese AI companies, multiple Chinese AI labs producing models that um have come out and are so strong that they can't that they've come out so shortly um or so closely in relation to other US frontier models that they don't have time to to to distill. So I think that when it comes to public models from the US, we could even see that gap shrink um between the Chinese models and the US models or even Chinese models overtake on the public side. On the other hand though, OpenAI and anthropic seem to be now keeping some of their their truly best models perhaps in reserve uh for their own internal use. Um and then maybe they're distilling their own models and and releasing sort of a a modified version for for public use. So there there is a an issue there where um yeah at least on the public side uh that gap could could close or or even reverse >> and and talk to me about the AI ecosystem in China because I mean in our coverage of this um we're always talking about anthropic and Dario Ammedday and and OpenAI and Sam Alman these big characters in these private corporations which have their own very sort of specific interests who we sort of then talk about how they're lobbying the government and they're all these very sort of distinct organs with distinct interests. Um, what's the situation in in China? Is it is it comparable or does it look very different? >> I think on the surface it looks very very different because I mean just look at the personalities and the kinds of drama that you see in the American AI industry, right? Like Sam Alman and Dario could not even hold hands at that AI summit in India. um they have sort of a personal uh animosity not anything not even to say anything of the Sam Alman Elon Musk rivalry right so in a number of these areas you see these personality clashes really come to the four and I think that um reflects and and drive some of the uh company uh level competition in China it's a different story a lot of the AI founders um know each other um some were trained together some um like the founder of uh Z.AI AI also known as Drupal um was worked with the founder of Moonshot um the maker of Kimmy K3 and so there are a lot of overlaps and they are trying to build and learn from each other in the Chinese system. Um part of their open- source push is to share sort of knowledge share innovations especially on making these models more efficient so that the the whole is greater than the sum of the parts um and that the different Chinese AI labs can can build on each other's work. So at least on the surface you have that collaboration. I'm sure there are very intense rivalries also in China. >> Yeah. Yeah. I wanted to ask about that actually because obviously people associate Chinese frontier AI as being open source. Um in the United States it's all proprietary software. It's obviously anthropic and open AI in particular. They own the weights. Is that as simple as China is communist and America is capitalist. Therefore the frontier of technology is open source in China and it's proprietary in the United States. Is that's is that what's going on here? >> It is funny how you have this parallel between the sort of different uh ideologies or or regime types and the uh commercial strategies for these companies and the tech strategies for these companies. Yeah, I mean certainly China's open- source push um comports well with the broader message that Beijing is trying to send to the rest of the world. Um that not only, you know, are they this sort of collaborative uh open ecosystem within China, but that they're trying to position themselves as uh partners in development for other countries around the world, especially across the global south. That's the message that's coming out of Beijing. And I think it it it's sort of like ironic how much then the US approach serves that same reinforces that same message. Right? So when um when the US puts e puts export controls on um anthropic's latest model fable and stops other people from around the world including within the US from suddenly using that model um that sends a message that the US is more closed that Washington is more willing to um uh exert control over this powerful technology or that the uh US AI companies themselves are also able to exert more control and bring those profits back to Silicon Valley. So, I think it's funny how this is h has been set up. It's it's an eerie parallel. I think there's probably not a coincidence that you have some of this here. >> Um, and you can't talk about AI or no one can talk about AI in in the West without discussing whether or not it's a bubble. Um, and I wonder if there's a similar conversation going on in China or I suppose because of the difference of the economic model, there's not nearly as much money going into these Chinese AI companies. Is that is that not part of the debate over there? >> There there are some worries about a bubble. Um, I think if there's going to be a bubble in China, more people are worried about like the robotics bubble happening right now because there's been a huge push on humanoid robots for example. Um, but overall it seems like at this stage, China is taking arguably a more sustainable approach to how to do AI. Um, keep in mind in the US in contrast like the the major AI companies are building out data centers at the level of something like a trillion dollars of uh capex spend per year. I mean that's that's the projected spend for next year. Um the speed and um scale of data center buildout in the US is sort of absolutely historically unprecedented in the US and um is putting a lot of strain on local communities is getting a lot of backlash um and is subject to supply chain constraints and a whole bunch of other uh energy issues. Um so there is a question about how sustainable it is in the US and on top of that you have um just basically two players now running the show Open AI and anthropic. Um so you have a much greater concentration of industry risk whereas in the Chinese system it's uh more distributed there's multiple players and while they are trying to invest heavily in compute they're not making this sort of bet the entire economy on um on reaching AGI. >> Uh let's finish by talking about those robots um that everyone has seen sort of especially in the humanoid uh world games or Olympic games. I'm not sure exactly what it was called. Um, and I suppose I want to link this to the sci-fi scenarios we're all hearing about because I mean in the west at least there's this big division. I'm sure you know that I'm not sure where you fit within it actually. Um, in terms of people who think that these potential takeover sci-fi scenarios are plausible and people who think it's just absolute nonsense. Now often the people who think it's implausible, they think the issue is this is on a computer. How can it possibly harm us? I mean we'd need big we'd need AI robots before it would be a real sort of danger to us all. China has a lot of AI robots. It has a lot of AI drones. Is there concern in China that these things might at some point take over or is is that still seen as as far-fetched in sci-fi? >> Yeah, I think that's still in the realm of sci-fi. I think a huge um gap there is just the capabilities of these robotic systems are very very limited, especially the humanoid robots, right? It's it's sort of fun to watch the clips from the uh humanoid robot Olympics that were happening in China where you have these robots. I mean, they can perform very impressive feats, but they also end um their 100 meter dash by crashing into a wall or um they they they're sort of falling over the place and it's, you know, you know, we we might even sort of overestim over over underestimate their capabilities because they seem so comical at this stage at least. But one thing to keep in mind is that this divide between the sort of digital world and the physical world is not so neat as we might want to think. And that's where coming back to sort of the AI systems, I do think that they there are real physical world consequences if we don't have a good system for regulating uh AI um whether it's the energy grid or the um communication systems. And on the robotic side, I think actually if anything, there has been a lot of a long-standing concern about robotic safety, industrial robots in the workplace, how to make sure that you have clear lines between where human workers are working and where these, you know, um large industrial robotic arms are working. So, I hope that that continues and I hope that especially as we see the proliferation of robots in more and more settings, um that safety issue continues. If only we could have that kind of safety mentality with these really powerful AI systems that IU can can also do quite a bit of damage. >> Yeah, I mean I I have to say I agree. Let's let's hope something comes out of this summit um later this month. Um Kl Chan, thank you so much for for joining us. Always a pleasure to get you on the show. >> Thank you.