AI EXPERT: Xi And Trump MUST Get Tough On AI

When Donald Trump and Xiinping meet this Thursday, AI will be at the top of their agenda. Um, experts and some Congress people like Bernie Sanders have been calling on the two leaders to agree to slow down AI progress, but the signs so far aren't great. This was a headline from CNBC on Wednesday and it says, "Trump she summit puts AI safety talks on the table, but neither side wants to slow down." Of course, Trump has publicly dismissed AI safety concerns over the last two weeks. He's called them a hoax. Um Trump also posted on Monday, quote, "Whoever wins AI wins. We are leading now over China and everyone else, and I'm going to keep it that way." Um, now that's unlikely to be the kind of rhetoric that would get the Chinese leadership on side. That's not a great sort of premise for talks. Um this doesn't mean however there's been no progress at all when it comes to collaboration on artificial intelligence. Over the weekend, US Treasury Secretary Scott Besson met with Chinese Vice Premier Heath Lee Fang. Um and Scott Bessant told CNBC what they had agreed. >> I think the important thing was to start talking. So to set up the dialogue, we we've now formalized something called the USA China AI dialogues. uh we've agreed to meet again probably in two months uh in um in Shenzhen. Uh the second thing is we want to open a communications line an incident line uh so that we have uh constant communications especially in the event of some kind of an incident and then we want to start discussing protocols for what are so both sides can agree on what the leading AI dangers are. So, what are the stakes in AI talks between the United States and China? I spoke earlier to David Krueger, an assistant professor in responsible AI at the University of Montreal. Um, he previously worked at DeepMind and the UK AI safety institute. >> Right now, I think what we need if we want to reduce the risk to an acceptable level is an immediate indefinite international moratorum on frontier development. We need to stop building more powerful AI systems. Um, so that's something that I think could basically in principle come out of this. So Trump and and Xi Jinping could have a handshake deal saying, "Yeah, we both agree this is too dangerous and we're going to figure out how to make sure that nobody keeps building these more powerful AI systems." >> Presumably, you're not predicting that's going to happen. You just think that would be theoretically possible were the political will there. >> I think there's some chance. I don't know, maybe like uh 1% or something. Like I think Donald Trump is kind of hard to predict and I don't know that much about the the Chinese side here. >> Um I know you spoke to a number of Chinese scientists, didn't you, earlier this year because you were with Bernie Sanders when he was having, I suppose, a bit of a summit, obviously not with the leadership of the Chinese Communist Party, but with Chinese AI scientists who were concerned about this issue. Um I'm not sure what did you what did you take away from from that discussion? I kind of got the sense that they have a different attitude in general towards this where in the west I feel like people historically have been like guys we have to solve alignment. We have to solve the safety problems. Um because if we don't government isn't going to do anything. Um whereas in China it felt more like they were like guys we have to solve these problems because if we don't government isn't going to let us deploy these things. Um and I think that's actually more you know how how it should be. We should be unlocking capabilities through this research. We shouldn't be unlocking safety. Safety should be something that we get by default because of policy that prevents AI companies from doing unsafe things. And I suppose you you've been on this scene for a while talking about and thinking about this issue for for a very long time. You studied under Joshua Benjio. Um I understand. Um, and I suppose something you just spoke to there, I think is so peculiar to people because it's this idea that the people at the forefront of these AI companies, they think that they should be the um it should be they themselves who as private individuals or as corporations oversee the development of super intelligence and and they should be granted the societal responsibility to make sure this all goes okay and you're saying that from your brief impression in China obviously they had a completely different interpretation which was to say well we'll build it until it gets dangerous because if it were dangerous Xiinping is going to not let us build it anymore. Um how ingrained is that idea in the sort of American company leaders that they have the right I suppose to carry this responsibility and to take these risks themselves. >> I actually don't think it's like that. I think uh I I don't think they feel like they have the right to do this. I mean, I think there's some of that, but I think it's more they just feel like government isn't going to act. Uh and that's I think a really common view when I talk to other people about this issue. They're like this is such a you know um lucrative business and of course companies are just going to you know lobby the government to prevent regulation which you know we've seen a lot of that um and you know I think historically also people didn't get it like people outside of AI really weren't paying attention um and they're still not paying enough attention and even after chat GPT there was a lot of attention on AI and even some attention on, you know, AI safety and and loss of control risk, but you know, not not enough like the the appropriate reaction, right, when you learn that we might lose control of the AI systems we're building and they might destroy humanity is uh to basically be like, wait, is that at all likely? And then when you learn that yes, it is, you know, we're talking about like maybe 10% is a common number that gets thrown around, then to be like, why on earth are we doing this? How do we stop? We have to stop right away. Drop everything. Figure out how to stop this. Um, and obviously that's not the response we've seen yet. Although, we're getting closer and closer to that as, you know, we get closer and closer to actually building the thing. >> I mean, I suppose that the reason we're not seeing that isn't because there are a bunch of people who don't really mind taking a 1 in 10 chance of dying. is because lots of people still hear what you've just said and find it somewhat implausible. Um I mean my personal position is I think that if you know Jeffrey Hinton, Joshua Benjio, Stuart Russell, if they all think that this is a serious issue and they develop the technology and they don't seem to have any particular vested interest and you know I don't necessarily think there is a 10% chance but personally I think there is good reason to take what they're saying seriously and act accordingly. Um, but I suppose to our audience who aren't convinced, there's many of them, which I think is very understandable. So, like, how do you make this argument to people because you've been making it for a for a very long time? >> The recent polling suggests that Americans largely are convinced that this is a real risk. Uh, and so I think the problem lies elsewhere. I think I'm I'm curious which, you know, why it still isn't an issue of the prominence that I described, but I have a bunch of guesses about why. So one thing is I think even when people understand that it's a risk, they don't necessarily understand how urgent the risk is or how deep the risk is. So they might think that we have much longer than we do to come up with the solution or they might think that solutions are easier to come by. So a lot of the conversation now is about doing better safety testing and having better guard rails and having standards and regulations, but that's not, you know, the reasonable response here. that's not going to actually reduce the risk to an acceptable level because we know that those things are unreliable. We know that they are quite likely to fail when we get to super intelligence. We're building AI that is going to be like another form of life. Uh and it's going to basically be able to do everything that humans can and more faster, cheaper, better. We don't know how these systems work. We don't know how to keep them from misbehaving. we don't know how to stay in control if they do misbehave and we don't know how to test them to make sure that you know they're not going to misbehave. >> You were the founding I think research director for the UK AI Institute and that often comes up I think because people often say this is a good you know the sort of what can governments do and people look at this institute as an example of a government doing something quite effectively. Obviously, it's the Chinese government and the American government here who are really going to get to determine, you know, if if we take your sort of interpretation of the risks AI poses, then it's going to be the Chinese and the Americans who get to determine the future of humanity essentially. Um, but lots of people pointing to the UK um AI safety institute as an example of, I suppose, a middle power doing something quite useful. Is that your interpretation of the UK um AIS? >> UKAC, I think they do a very good job at those things. I think they're one of, you know, the maybe five, let's say, leading evaluators in the world. Um, so they definitely have have talent and skill. Uh, I think that is something that can be useful. Um, but ultimately I don't think it's very useful because we know that safety testing is flawed. And unfortunately, the way that it's been sort of enshrined in practice and in, you know, governance, I mean, it's self-regulation. It's not real governance for the most part. um but uh it's gotten the burden of proof backwards. So these safety tests are very different from the kind of um safety practices that you see in other fields like aviation, nuclear power, pharmaceuticals uh where they have you know they take safety very seriously and they have established processes for um you know seeing if something is safe. And one basic thing that we get backwards here is the safety tests we have in AI, they can only show that a system is dangerous. They can't really show that it's safe. And what this means is that in practice, the policy that we've implemented here is to say, let's assume that these AI systems are safe until we can prove that they're dangerous. And that's really backwards because it should be on the people building the technology to demonstrate convincingly that it is safe. Uh and unfortunately they don't know how to do that because of you know the technical limitations and technical immaturity of the field. Um, and and you know, there's other problems as well, like external evaluators don't have all the access that they would need in order to even do a a best shot at figuring out if it's if it's uh dangerous. And then often times there hasn't really been um teeth. If you do figure out, hey, this model is dangerous, then what happens? Well, Anthropic used to say, "We won't release something if we think it has a good chance of killing a million people or something like that." Uh, and then at the beginning of this year, they just changed that. They changed their self-governance, responsible scaling policy to say, "Actually, we would still maybe do that because we're in a race with other companies." >> And you seem to think a pause is plausible. I mean, you're campaigning on it. You're sort of really working towards raising awareness about this. What do you see as the most sort of likely sequence of events where that might happen? I mean, lots of people are suggesting we we need some kind of Chernobyl moment. Obviously, no one wants to sort of wish for a catastrophe, but a sort of minor one uh is what lots of people are suggesting might be the only thing that actually creates a political momentum to make it happen. And obviously, there are midterms coming up. It seems like AI is going to be an issue there. Um Donald Trump could have a damosine conversion. You never know with the guy. Um if if a pause were to happen, what do you see as the most sort of likely sequence of events that gets us there? Yeah, it's a great question. I mean, I would like to think that if we're all going around saying, "Well, nobody's going to do anything until a bunch of people die until we have a huge catastrophe, then we can take the next step and be like, "Hey, wait a sec. If we're all just going to wait for this catastrophe, why don't we just skip that part and just jump to doing something?" If we all know that we're just looking around waiting for that to happen, you know? Um, and and I do think that's possible. I think the idea that you need the catastrophe in order to get governments to act, I think that's just wrong. Um I think what we saw with uh Jacob Coxin and the new cycle there is that one person resigning can shift the conversation in a huge way. Uh and of course that's on the heels of the hugging face incident, which was definitely some sort of warning shot and actually quite a quite a good one because nobody did die. Um but you know it was still a huge huge gamecher just by itself just that one resignation. Um so you know what do I think is the most likely way to get a pause? Well I think it probably still is that we're going to see more warning shots because we aren't going to do the right thing like you know today. Um I'm I'm I think we absolutely could. I think it's certainly possible. Um there's a lot of you know things are still moving really fast. There's a lot of momentum. There's a lot of political will emerging. Um there's also you know a big difference between a pause uh of a few months at the frontier AI companies or this kind of pacing the frontier idea that's become popular um versus what I'm proposing which is indefinite an indefinite pause. So with no fixed end point. Um so it could be it still could be short if we you know we pause we get a grip on the situation and we're like oh actually you know we think we can do this safely if we just xyz. That's not what I'm expecting. I'm expecting it to take longer because there are these difficult unsolved problems. Um but yeah it could also take a long time right and and I think we we can't count on solving these problems on a deadline. That's why I think it needs to be indefinite is we need to give ourselves the time that we need to figure out what we're doing technically and then also like to figure out how we might want to integrate more advanced AI into society. Like if we're serious about building something that could do all the jobs that humans are doing, that's the most disruptive, you know, thing technology we've ever built. Um because society is sort of built on, you know, the idea that people's labor has value and that's where they derive a bunch of their power, including their political power. And and finally, this is I suppose this is jumping a few steps ahead of where we probably are because Donald Trump is not a particularly rational man looking to make a rational deal. But if if he were to be a sensible person, um, and I suppose if if if Congress decides to take action on this front, um, the big sort of technical issue everyone is going to start debating is verification. You know, if America stopped, how can they check that China isn't going ahead or even if you know, say Open AI pace the frontier? How can they know that anthropic isn't racing ahead? Is this a is this an irresolvable problem or is there a quite simple technical fix? >> I I think it's actually pretty simple. The hard part is knowing where the chips are and the factories that build the chips. Um if you know where those things are, you can just take them offline. You can just decommission them. You can dismantle the supply chain for the chips. You can, you know, destroy the chips. That's like a very very simple solution that I think would be quite robust and easy to implement to ensure that nobody has the means of continuing to build more powerful AI systems. Um now there's a lot of things you can do besides that right you can instead say let's let's monitor these chips. Uh you can for instance have people from different countries literally at the data centers um and inspecting that some protocol is being implemented correctly. you can say these data centers are only going to run certain kinds of computations. For instance, they're only going to do inference on existing models instead of training new models. Um, and we have, you know, a technical protocol for ensuring that that's the case. And then we have people from all the relevant, you know, signatorries, all the relevant parties, US, China, whoever else monitoring all the activities at the data centers to make sure those protocols are implemented correctly. Um, you can also like just take all the chips and like put them on an island somewhere and, you know, have the everyone will have the ability to blow them up at a moment's notice and then you kind of figure out what to do next. Um, so I think it's actually really easy to begin such a pause and then there's lots of options for how to continue it. Uh, what I'm also concerned with is the durability, right? uh because I want it to be indefinite. And for that, I think the main concern is that somebody somewhere will continue doing this in secret. And they could, you know, if they have a bunch of chips that they've squirreled away, they could build a data center, you know, underground somewhere and keep training. Um even if they don't have the chips, maybe they could build one of these factories, one of these fabs, although they're extremely extremely sophisticated and require regular servicing and maintenance. Um, so I do think it would be pretty hard to build one of those in secret and to carry out this kind of a secret development project if we just collected all of the chips and fabs that currently exist and um, you know, monitored them, took them offline, whatever. uh cuz it would probably take a matter of years and so intelligence agencies from your rivals would probably figure out that it's happening in time to have a diplomatic solution resolution to the to that um to that development and the threats that it poses. On the other hand, um if we don't get rid of the chips, uh if there's still a bunch of training capable chips lying around, I do worry that at any moment one of the countries might just say, "Hey, we're going to grab a bunch of these chips and keep training." And then you only have like maybe a few weeks or months potentially until they could be building something that is, you know, extremely dangerous. Um and that becomes much more tense situation. I think you can potentially resolve that by setting it up such that it's easy for anybody to again like destroy the chips or take them offline. So like if the US tries to seize them, China can just blow them up or something like that. But I haven't thought through all the geopolitics there cuz you know if that's the situation then China might decide to blow them up for another reason like just because they feel like it's to their advantage for some other reason and so you don't necessarily want to give everyone that power. So there's a lot of things to think through here still. Um, fortunately we're starting to see I mean it's it's a little bit later than I would have liked but we're starting to see a lot of research um coming out looking at this stuff and I'm I'm currently working on a very extensive feasibility study with some other professors here at Berkeley and other universities that should be out in like maybe a week or two. >> Fascinating. Everything is moving so fast in this discipline. You say sort of you know talk to people in some disciplines they say it might be coming out next year in in AI research it's coming out in two weeks time. Um David Krueger thank you so much for for joining us on Navar Media. really fascinating conversation. >> Yeah, thanks for having me. It's been great.