SPOOKED A.I. Bosses Call For Slowdown, Trump Says “NO!”

Last week was the moment fears of catastrophic AI risk went mainstream. And this weekend, industry leaders have responded. Anthropic CEO Dario Amade has published a blog calling to pace the AI frontier. This is a phrase we're hearing more of these days. Um Amade says the urgency for a slowdown is due to two developments this summer. first and the AIs are increasingly now being used to build the next generation of AIS which is speeding up development what's sometimes called um recursive self-improvement. Um and second he says the hugging face attack or the hugging face hack suggests that within 6 to 12 months an AI swarm could be capable of taking over the entire internet. Um this has been disputed. Um in response he proposes so in response to sort of these two developments he proposes that AI firms should be required to give access to third-party evaluators to assess their latest models. He says that anropic is going to go ahead and do this unilaterally without any regulations. But then he also suggests that frontier AI companies within democratic countries he's quite into this war between the democracies and the autocracies as sort of he calls them. That's his master frame. He thinks um that companies within the democratic countries should coordinate to establish common safety standards um as well as limits on the rate of unchecked AI progress. By the way, when he says the democratic countries, he just means the United States. Um and he says he wants some limited coordination with China. Now, I think there's some incoherence in his hawkishness on China and then his suggestion we can have coordination with China. We'll get on to that in a moment. Um, first off, worth pointing out, other AI executives piled in to agree with AMD. This was Sam Olman. Um, so he says, "I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks, committing to having independent evaluators with employee like access is a great idea and we will do the same. We'll have more to share soon." Um, Elon Musk um, for his part simply said this. Dario is right. Um Dario Emmed has appeared on CNN um to lay out precisely what he was calling for in this three-step proposal. Each step is more difficult than than the one before and Enthropic was kind of getting the ball rolling by committing to the first step. But we shouldn't kid ourselves that these next two steps are going to be more difficult and and we may not succeed. and and so my my proposal is a proposal that we should try. Now again from all the agreement from others that I've seen in just the first few hours of putting out the essay that creates some optimism but that needs to follow up with results. So yes, pacing among dem companies in democratic countries. Now the one of the important things that that I need to call for here is you know as we get to the pace of model releases right that's commercial activity and in order for companies to coordinate directly on commercial activity they need to do so working with the government and so the one thing that we're asking for from government right now right I mean there's also this question of what should the regulations be but there's something even simpler that the government can do right now which is it can sit down in the room and with all the industry players and we can all talk together and the government can make sure that we're you know we're we're not doing anything nefarious from the perspective of antitrust and collusion. We're just trying to make our system safe and to work together on making the system safe. >> So that reference to antitrust law there has got many people's alarm bells ringing. So, there's long been a suspicion um that one of the reasons these companies call for regulation is because they want to lock out competitors. They want to say, "Look, we're the only companies that can be trusted with this. We're going to have to collude and and lock out everyone else from from the system." Um I suppose Abid would say that's not exactly what's happening here. So they're saying, well, he's officially saying, you know, you can read in, you know, you can make your own judgment about what his motives might actually be, but what he's officially saying is we want to be able to discuss with OpenAI and DeepMind and whoever um how to slow down the release of our new products and how to slow down our training. And at the moment, it's ambiguous whether that would be allowed by antitrust law because obviously big companies, competitors getting together and colluding, that's illegal um because that's associated with price setting. Um, but they want to say we should be able to get together and discuss this. The American government can be there so we know that after the fact you're not going to drag us through the courts and say that we broke antitrust law. So, you know, many people looking at as cynical. There is a logic to it though. I have to admit. Um, in terms of his broader motive, obviously whenever this comes up, people say they're just trying to boost their share prices. Now, obviously we don't Anthropic doesn't have a share price at the moment because their IPO hasn't happened, but AI related stock has fallen by about 2% today. again it will recover that within a few days so it shouldn't not too much should be read into that but the idea of a slowdown doesn't seem to have like really pleased traders um I think the most sort of benile explanation for why he might be coming out and saying this now is that he recognizes that the AI industry in its current state is about to cause a very expensive accident right so the hacking face hack right that was an AI system where open asked them to to do a sort of hacking competition and that they weren't supposed to go on the internet. They weren't supposed to break into another company and then they committed a federal crime. Now, Hugging Face, the company, aren't going to take Open AI to court, so it's not going to cost them any money. By the way, Nvidia have just bought Hugging Face and Nvidia don't want anyone to go to court cuz they just want to sell as many chips as possible. So, they got off lightly in a way here, but I do think that anthropic think if we keep developing at this pace, we we might cause some real damage, right? and that will cost us that could really damage our share price. So there is a I suppose taking them at face value saying we actually do feel like we're going too fast and not necessarily because we think we're going to kill everyone but because we think that we are about to make a real mess and we're going to be liable for it. So help us all slow down before we break something really expensive because we don't want to have to answer to that. Of course, something else I find plausible is that they're looking at how there is now a big public and political push for regulation, and they want to get ahead of that argument and basically propose um you know, ineffective regulation, regulation that isn't going to really limit what they can do. I think all of I think it's probably a little bit from column A, a little bit from column B on on on this one. Where I differ with some of the commentary is I think they on on AI safety, Daryo hasn't just started talking about this in the last six months. He's part of a scene that has been talking about this for over 10 years. Um, let's look at the political momentum here because whatever Amade's motives, as I say, there are sort of various plausible options. Um, there needs to be sort of political momentum to make this happen, to make it effective. Um and on Sunday, the New York Times reported that Barack Obama, probably still the most influential person related to the Democratic party, he has urged um the Democrats to make over AI oversight central to their agenda. Um however, speaking from one of his golf courses in Ireland, um sort of in between calls for Irish unification, um Donald Trump has suggested AI regulation won't get support from the White House. >> We've seen a lot of concerning reports about AI. Should the AI industry slow down or be more regulated? >> Well, I say this. Look, we're leading China in AI. We're most sophisticated country in the world. And frankly, I want to keep it that way because whoever wins AI wins. And uh we could put guard rails. We can do this and that. But I think you have a lot of negative forces that are bringing it up that uh shouldn't be bringing it up. And they're bringing up things that won't happen. But whoever wins with AI wins. You know, it's an expression that I came up with and it's true. We're building billions and billions of dollars. You look at the plants and factories, forgetting about AI for a second. We have more auto factories under construction right now than we have had in many, many decades. Maybe ever. Maybe ever. >> If you weren't reassured by that answer from Donald Trump, take a look at this headline from this afternoon. NBC News saying Trump says AI doesn't need guardrails, only a high IQ president. In which case, we're really screwed. Um, he said this on Truth Social. I'll give you a few more words. He said, "The only control or guardrails that AI needs is a strong and smart high IQ president, and the USA has that in spades." He then says, "The Trump administration has stopped AI people," which he puts in quote marks, from doing bad or potentially bad things, which again he puts in quote marks, like Dario Andropic, who is now pretending to be a perfect little angel, and we will continue to do so. Um, Steven, >> yeah. >> What's your how have you responded to these calls by Anthropic and the AI leaders for the government to help them coordinate a slowdown? Um, how have you responded to Obama to Trump? What's your what's your take? >> I mean, it's sort of two separate things. I I don't know if you remember when the AI stuff all started taking off and I remember us playing clips from a Senate inquiry and Sam Alman was there and uh and they were talking about AI regulation. And the senators are like, well, you know, we don't understand this. And then they were like, Mr. Oldman, would you regulate? Do it for us. And and like that's obviously that was the moment when he sort of said, yeah, why not? I mean, maybe it wasn't politically impossible at that point, but I mean, so so I think I'm of the option that, you know, I want to I want to proceed with like loads of humility because much like, you know, crypto, I also don't understand the technical nature of of of this very well. But I do know that there's a very lively debate within experts in AI, you know, very big professor at Stanford who insists that all of this hugging face stuff is sort of ephemeral. It isn't really uh anything like the breakout. Uh then you've got other people uh you know like the guy uh who quit last week saying yeah 10% chance that we're we're going to all get killed. Um I do think that there is something you know the truth is probably somewhere in the middle somewhere. let's just use that kind of, you know, old saw. Um, but there is something advantageous for these companies in being able to get ahead of legislative agendas and regulate themselves. You know, when they say like, oh, the government can come and watch us and make sure that we're not doing like antitrust Um, that's sort of like, okay, the government can be a kind of observer to what it is that we agree and why we agree in the full knowledge that nobody from the federal government is actually going to really understand what it is that they're that they're doing, right? They're operating at a level of like jargon and complexity that is going to vastly outstrip the abilities of most federal employees to be able to to regulate. Maybe they'll hire some expert in, but then you know these are only you know they're not actually working on these things themselves. So I think there is an advantage to be gained here by sort of hyping things up and then saying actually only we working together can deal can deal with this problem. Um and the other reason why it's an advantage to get ahead of the legislative adventure um agenda here is because I think Obama is right not just from an AI safety point of view but from from a from an electoral point of view. The party that comes forward and says we are going to regulate AI. We are going to take control of this runaway industry um of you know potentially for all we know pirates or maniacs um who continue to develop things that they know are d are dangerous. We're going to take control of it and we're going to regulate it and make it safe. the party that puts that agenda forward is going to win because people do not like AI and they are actually quite frightened of it. So I think that sort of sense of a politician saying Democrats put at the heart of your agenda so you don't look like Trump who's just being like oh all you need is like my genius IQ in the White House and that will be everything will be fine. Um put it in the agenda. People don't want this. They will want to vote for regulation. They don't want data centers. They will want to vote in all sorts of ways to limit the speed at which this is coming down the road. Um, I think that is a that will be a vote winner. >> Yeah. I mean that we're gonna get on to China now actually because I think the simple argument is to say all of you people said that your technology might kill us all right and now you're doing it and we as representatives of our constituents are going to tell you don't do that. [laughter] We we don't whe some scientists say it might not kill us all. Some scientists say it will. you CEO you said it might do and we do not let people develop products where the boss of the company says it might kill us all right that would be the obvious thing for the politicians to say obviously the response they always then give is that oh then the Chinese will overtake us and that still does have I don't know I don't know how much sort of traction that has among the American public but it has a lot of traction in in Washington it seems and definitely in Silicon Valley as well so let's talk about China because for their part the Chinese government doesn't seem impressed with Dario Ammeday's proposal. Um, a state newspaper has blasted calls for an AI slowdown as a cold war tactic. Um, Reuters also reports on a Chinese Foreign Ministry spokesperson um who said, quote, "Fermenting various threats engaging in confrontation and malicious competition will only disrupt the process of global governance of artificial intelligence and is not in the interests of any party." So, they're seeing this as hostile. Um, obviously there is an argument say it would be in the interest of both the Americans and and the Chinese to slow down, but they're seeing this as hostile and they have a clear reason to be very suspicious of in particular Dario Amade. Um, so he is a China hawk. He he he self-describes as a China hawk. I mean in his latest essay he writes this pacing within democracies will be limited by the lead that US companies have over authoritarian regimes chiefly the Chinese Communist Party. If we slow down by more than this amount, then unpaced CCP associated projects will pull ahead, creating significant national security risk. I agree with Secretary Pessent that a Chinese lead in AI would pose grave danger for the United States and the world. The CCP associated projects will run the alignment risk that US companies are carefully preventing. And even if they avoid those risks, they will be in a position to militarily dominate democracies, for example, with AIdriven drones. Thus a key part of pacing within democracies is to keep democracies AI lead over autocracies as large as possible to give us the breathing room we need in order order to pace effectively. And this is I would like like to put forward kind of mad, right? The idea well also you need to have a bit of strategic empathy, a bit of cognitive empathy. If the Chinese are reading this and thinking this American executive is saying the moment the Chinese get a lead, they're going to use that lead to use drones to dominate America. Like they're think I mean we that wasn't our plan, but I mean maybe that's your plan, right? [laughter] You're saying we have to keep the lead. um otherwise everything is over because whoever gets this dominates everyone then they're going to think I you're not the kind of people I can make a deal with right of course that is going to put your you know your rankles up um Amade to that end recommends also hostile actions against China he says they the US should tighten export controls on chips to China and he says there should be a crackdown on distillation of AI models that's a technical term this is the process whereby competitors can reverse engineer the weights of a frontier model by asking you hundreds of thousands of questions. So, so for example, um and then training on the results. So, for example, you know, OpenAI have just released their latest model and then you have uh a Chinese company um this is I don't know if anyone's proved this, but this is sort of many people's theory. You have a Chinese company that has way fewer resources, doesn't have access to the same size data centers, but can create a model which is of of similar quality to the American version by asking it loads and loads and loads of questions and then reverse engineering basically the weights. As I say, many people think this is how the cheaper Chinese openweight models were developed. Um, so as I say, Amade explicitly wants to slow Chinese development down. Um, and again, in terms of his hawkishness, this is even worse. In a previous essay, Machines of Loving Grace, this is one of his first sort of big blogs he put up that sort of got widpread traction. um he went further and said that if the west successfully slows down the Chinese development um of AI then he says the democracies this is how he groups the world um he says the democracies can assert military and economic dominance then he says this it's very explicit he says if we can do all this we will have a world in which democracies lead on the world stage and have the economic and military strength to avoid being undermined conquered or sabotaged by autocracies and may be able to parlay their AI superiority into a durable advantage. This could optimistically lead to an eternal 1991, a world where democracies have the upper hand and Fukuyama's dreams are realized. So he's saying if we manage to maintain a lead in AI dominance and I suppose get to AGI first is the argument, then we can force China to become a liberal democracy. Now, I mean, if you're the Chinese people and you've had, you know, 10% growth for sort of half of the years of the past 40 years and loads of you've been lifted out of poverty, you think, okay, so they want to impose a democracy on us. How did that work in Iraq? How did that work in Russia? Yada yada yada. I'm not sure we're up for this. If you're the CCP, if you're the Chinese Communist Party, you're definitely not going to like that, right? So, the idea that you're going to say, "Oh, this Darai guy, he seems like someone we can work with. He literally wants to get to AGI first so that America can overthrow the Chinese government and replace it with a liberal democracy. Doesn't seem like a great recipe for cooperation. So I think Amodai has got some real I mean blind maybe we could say blind spots. I think there are some reasons why the Chinese are going to be suspicious of this guy. And let's look at a different executive because tonally at least um Sam Alman has been sounding slightly less hawkish. I think presidents Trump and she would get the Nobel Peace Prize together if they could agree on something that should be easy to agree to. Um, and it would be wonderful. I think that clearly the two countries are going to compete in lots of ways and this is going to be important socioeconomically, geopolitically. Uh, but no one they should be able to agree that no one should be like taking a certain level of risk um with the development process of this. And even if just the US and China um could agree on some shared standards and testing for development of of this technology. Uh I think that'd be a wonderful accomplishment that the two of them can deliver. >> Completely agree. I mean I think at the simplest form if there was just a ban on reaching RSI until it could collectively be safe, wouldn't that be enough? Um, I think it's very hard to say what a ban on RSI means. I also think that probably wouldn't be enough. But I don't think this is hard. This is like a onepage document. >> Feels a little like Independence Day, you know? Yeah. In terms of like there's something bigger than the conflicts that we have geopolitically. >> Like the movie when the big show when they're just like, "Let's like all forget our differences and let's all figure out how to act." >> No. >> Yeah. No. I think it should be like that. >> Yeah. All right. So, we need our Independence Day Nobel Peace Prizewinning moment from Chi and Trump. Maybe when they meet in September >> that would be great. >> No, you know, I don't I have I mean zero trust in in Sam Alman, but I think if we were to urge Donald Trump to come to some kind of accord with she when they meet later this month, that would be a good thing. Obviously, Donald Trump does really want to win a Nobel Peace Prize. I suppose where I disagree with Samman is that a one-page document is going to be enough, right? Because the interviewer puts forward some concrete practical options to say, let's not do recursive self-improvement. That's where AI models are training the next AI models. And Sam Alman says, "Oh, oh, actually, oh, that might bit common because OpenAI are already doing that. That their AI models are already training their next AI models. Anthropic are already doing that." So, to me, again, this sounds a bit like these AI executives want some sign that sort of I think they want electorates to think, "Oh, we're safe. It's fine." Um, and so a sort of A4 document that she and Trump have signed, which doesn't really do that much, or this sort of panel with AI companies in the American government that doesn't really do that much. That's the ideal for them because they they want to be in charge, right? They think they're smarter than everyone else. Ultimately, they would maybe quite like to get to AGI so they can upload their brains and live forever, right? That that that's the kind of world we're in here. Um, but, you know, I I suppose a an A4 document would be better than nothing. Uh, at this stage, maybe it would be a start. Um the Chinese by the way because I I sort of put forward earlier they they don't seem to really want to slow down development that they don't have the same kind of concerns at the moment about existential threat as are dominant in Silicon Valley and now sort of increasingly salient in the United States. But there are some areas where the Chinese government does seem concerned about AI development. Um the FT today has a report um on the Ministry of State Security warning of threats from AI and they have a quote from the state security minister who says hostile forces and people with ulterior motives are abusing generative AI technologies such as deep fake audio and video AI generated text and images and intelligent troll armies to cheaply and on a mass scale fabricate political rumors, disseminate harmful information and incite divisive sentiments. These activities constitute an attack on public opinion and cognitive warfare against China, directly threatening our political security, institutional security, and ideological security. Um, it's interesting the idea, Stephen, that like China are not going to put any guardrails on the I said this said this in my interview with Robert Wright on on Friday. You can check out that that interview. I love Robert Wright. Got his got his book here, The God Test. Put that out on Friday. Anyway, I was saying the idea that the Chinese are not going to be interested in guardrails, there's one thing the CCP likes, it's guardrails, right? The you can definitely make a deal with China on this, especially because the Americans are ahead at this point in time. But you probably can't make a deal with China if the if the premise of the deal is that the Americans are always going to be six months ahead. And by the way, if they get to AGI before you, they're going to overthrow your government. >> Yeah, exactly. They're going to give you the end of history or else, which is [laughter] the the quote there. Um, I mean I actually like just when you compare those quotes about China to the to to what you were saying about, you know, let's all get together with a government kind of watching and we'll sort of set our own, you know, we'll all agree to slow down together. Um, there kind of like you can think of like a positive break and an anti-break operating at the same time there, right? So the positive break like we the AI companies decide how we slow down, but we always link it not to safety but actually to the progress that China makes. And I think that's like a very, you know, that actually shows that safety is not at the forefront of of their mind there, right? If it was at the forefront of their mind, then safety would be like the absolute limiting factor. What's at the forefront of his mind is, I would say, a way of slowing things down so that you can evade legislation, but at the same time making sure you always retain a competitive advantage over a competitor who's some some way behind. That doesn't look like a guardrail, right? that looks like a kind of um a stop to national anxieties without giving up on strategic ambitions >> because the Chinese are their competitors as well. It's not Amade seem he's been talking about this for so long that I think he is a genuine ideological China hawk but like encouraging the Americans to stop selling chips to the Chinese he is helping to hobble deepseek and obviously one of the things these companies are very worried about is they've invested all of this money in training their models and all of the the money that sort of they're suggesting they deserve in their IPOs in their sort of upcoming um sort of introductions to the stock market is based on the fact that they have this very valuable technology which has a motor around it. But if you have if if if if um anthropic and open AI can say we've actually got this great really useful product um and we want you to pay us you know if you're a user $20 a month or if you're a business you know a few thousand a month the business might say yeah this is useful but I can get it I can get an open model from the Chinese for tenner so I'm going to do that sorry so they are terrified of the openweight models I actually also do think that there is a safety concern with the open weight models I don't think the whole debate about openweight models is purely just the American firms cynically trying to hobble their competitors I also do think that if you have these powerful models and anyone in the world can tune them and tweak them to do whatever they like, um that to me does seem worrying. So again, this is a complex issue. Um let's go back to US politics because um one person who seems to be taking Barack Obama's advice on leading with AI is John Oaf. He's a Democratic senator for Georgia and he's tipped to run for the presidency in 2028. And when you step back and consider it, the situation is absurd. This week, we were warned again of existential risk to the human species. Did you see it? >> And what fills us with dread is the crushing weakness of our politics against the massive stakes of our times. Tech titans dig bunkers while warning us the new intelligence they're creating risks human extinction. Congress debates youth sports and the president builds a ballroom. An ancient establishment barely capable of email sleepwalks into a technological revolution while the president dismantles precautions and calls us fools for worrying. Just listen to this. On day one, Donald Trump rescended the AI safety order meant to guard against catastrophic risks, including the engineering of bioweapons. Meanwhile, Don Jr.'s 1789 capital bought big money stakes in data centers and AI labs. 4 days before inauguration, the Emirati spy chief committed $500 million to the Trump family crypto business. And four months later, the Trump administration granted the Emirates the rights to buy our most sensitive AI chips. See, a capable president would move swiftly to secure and reassure the American people, rush inspectors into frontier labs, fortify the nation against bioteterrorism, demand legislation from Congress, and lead the world toward an AI treaty. But President Trump is compromised. A capable Congress would engage every committee in investigation and legislation to act swiftly and wisely. Those CEOs would be under oath next week. But instead, the president and the speaker of the house sit on their hands and hope for the best. So you see this scenario and you look at the state of our politics and it shakes your faith in the future. I understand. But Athens, we can and will reclaim control over our own destiny. [cheering and applause] We can and will meet our obligations to our children. And that begins with victory this fall. I've said it before, but it's always those moments where you see sort of like a political message and sort of a political candidate talking about um how we need to stop the robots killing us all that I feel most like I'm in a sci-fi movie cuz those are always sort of the moments that you get and the sci-fi movie always has that moment where they go from a niche concern to a mainstream concern then it becomes a political issue and that sort of is to signal that this is now you know something that is affecting the whole of society and we're seeing that there. Couple of people in the comments saying, "Why are we looking at American politics here?" When it comes to AI, the things that matter are American politics and Chinese politics. That's kind of the case. If if what you're worried about is sort of highly powerful AI models potentially causing significant damage, then what matters is Congress, the American president, sort of movements in America, and then what's going on among elites in the Chinese Communist Party. That's that's just the fact, right? I sometimes when it comes to why are you talking about American if it's like lo the local city politics of New York or something like that like why are we talking about that but when it's an issue which clearly has international ramifications and the Americans or the Chinese are the most important places where these decisions are getting made we we have to pay attention to the Americans it's just it's just more relevant than telling you what Darren Jones is saying about AI. All credit to Darren Jones. I'm glad he's talking about AI but it matters more what John Oaf is saying. Um, on the issue of the alliances that this can bring together, we got a Brit in this story that will make you feel sort of more seen. I suppose this was an interesting nugget on the unusual alliances AI might bring together. So, Jacob Coxin, the Anthropic employee who resigned um from Anthropic. Um Bernie Sanders and Steve Bannon are set to join forces for an event on Tuesday. Um the event is organized by the Future of Life Institute and will have Congress people from both parties speaking as well as union leaders. I love to see this. I think You don't want I think the same about climate change. I think the same about all of these big existential issues as much as possible. You don't want this to be polarized along party lines. You want you want people to think this is a big issue that affects us. Oh, let's try and sort of keep it that way. Keep it an issue that we all care about and that we don't have to think, well, climate change is probably not true because the left believe it's true or you know, XYZ is not actually a problem because the right say it's a problem. Yeah. I think this is good. I think seeing Jacob Cox and Bernie Sanders and Steve Bannon on a stage uniting for prohuman AI, I like that. Again, quite sci-fi. Um, in another interview over the weekend, let's talk about because let's let's finish by going back to Amade who is the reason we're doing this story today because of his his essay because on this issue of the intersection between politics and AI, these tech companies, um, Daramo said something significant that's got lots of attention. Um lots of people have been talking about this on on social media um where he suggested that significant government involvement in AI development could in theory be welcome. >> It seems like the fate of humanity is at risk. How much personal responsibility do you feel about where we are going? >> My view here is it has always been very strange that this technology is being built by by a private company. People ask me that question all the time. Why isn't this being built by government? And the strangest thing about it is I agree with them. I'm uncomfortable. Government didn't built this technology. This company, this technology came from the private sector. And we anthropic and I would hope other companies have done everything we can to try to uh have legitimate oversight mechanisms. I think the government and the public needs to have a stake. And this is why we've supported regulation of the technology. Regulation constrains the private companies. Regulation allows the public and its elected representatives to have a state to have a say and limits what the private companies can do. Now you might ask why doesn't government just control the technology completely? >> Would you be willing to give up the technology to the government >> to the to the right combination of governments? So the worry I have willing to hand over this company. >> I want to be very clear about this. The the worry I have, I am concerned that one single government could abuse this technology just as easily as a single company could. But I think a combination of democratically elected governments. I don't know about handing over, but some kind of oversight, some kind of joint governance. Again, that would be the work of years. But I wonder if that's the direction we need to go in. >> Stephen, is is Amade asking to be nationalized there? >> Oh, I don't know what I don't want know what he wants. I mean, you know, hand handing it over to governments, but actually then he says, but not hand not handing over really oversight oversight. But actually, I don't I I'm actually, you know, I think oversight's really important, but it's just as but just as important is ownership. Um the changes that are you know according to these guys going to be uh dumped on all of us by by this technology if we survive it uh means that um many people are likely to be impoverished unless there is some sense of shared ownership of it or some sense of redistribution of its profits. After all, it is built on the work of humankind, on literature, all the stuff we've written to each other on the internet, songs, um, discoveries, research, um, in all fields. Um, it only works because of us and it only exists because of us. And I think actually that is the that that is regulation is the first political conversation I want to hear. And the second one is I want to hear ownership and redistribution of the profits of these of these these things because I you know he talks about private industry actually those companies are massively invested in by the US government as well. >> Um on the most general sense I know because you know you've said before on the show you you think I've taken the AI cool drunken the AI cool a little bit. I want to know sort of how you're viewing this going mainstream because from my perspective when I was sort of looking at the inventors of the technology you know Jeffrey Hinton Joshua Benjio Stuart Russell all these people who created the technology won loads of awards for it saying this literally might kill us all and it wasn't being discussed politically I was like this is kind of scary right because these guys they might not be right but they should definitely be taken seriously and no one was talking about it so I am whether or not I think Amday or even John O. Whether or not I think these people are being cynical, I don't know. The fact that this is now being talked about to me seems like the public discussion is somewhat commensurate to the challenge and I'm feeling a sense of relief. >> Oh my Oh my god, we're now talking about this thing and I I wonder how you're viewing it. >> I mean, I don't I mean, it's commensurate to one challenge, right, which is the existential challenge. I mean a lot of this stuff reminds me of and you know I don't want to be like um um a critty hype you know like cyn cynic here I don't want to be like cynical coral critty hype but you know there was a lot of I remember like the the 2010s and uh you know all these centers opening up the center for existential risk for example got a lot of money for looking at problems with a very small probability of happening but would be absolutely catastrophic if they did loads of money put into AI stuff I mean a lot of this like AI research has come out of those institutes actually a lot of like the ethics of AI I and uh the decision-m processes like you know like the thought processes of of potential you know models of models of of thinking come out of the come out of those safeguarding comes out of those um and in the meantime like none of those institutes were looking at like much more immediate crises that were were you know were likely to hit us which wouldn't be existential but which would be really really really bad >> but their argument sorry their argument for that was to say that lots of people are already doing that. So they would say there's this gap in the they're not saying everyone should devote their research to existential risk, but they're saying there's this there's this under discussed thing which is existential risk. So we're going to put some money into >> No, but they've got a lot of funding that drew in a lot of brilliant people who might otherwise have been doing other things which were more immediate. And I think here when you say, you know, you're really glad that the political conversation has turned to this pressing issue. I think it is just simply one pressing issue. The other pressing issue is the much more likely thing that's going to happen, which is that there's going to be massive social and socioeconomic changes that come just through the loss of jobs. M um and I'm not really or or just in the changes in in the ways that that that we live and I'm not really seeing anything about that and in fact the only things that I am seeing are quite scary that I think could be applied when those changes come. So I'm I'm personally of the view that I actually don't think that people are going to tolerate what AI is promising unless it comes with with with with quite quick rewards and I'm not seeing any discussion of that at all. And I think when you know when I look at things and you're going to make me think I'm all like conspiracy headed but when I when I look at things like the animous that's building up around data centers which is the physical manifestation for many people of what AI is you know really objecting to them to having in their neighborhoods destroying their environments you know using up masses of water um you know massive ones being built next to you know towns and villages and beautiful areas which are which are dwarfed like in Georgia for example the animous that is building up around those is only going to spill out onto what is classed as you know critical infrastructure both here and in the and in the US um because you can't attack AI but you can attack those centers and I think what I see happening is actually this the walls being built against that so when I look at like the prescription of Palestine action here using terror legislation against direct action groups and then the banning of it as um an international terrorist organiz organization by the states what I think I see are actually the frameworks being built for suppressing the anger that is likely to come by calling national security issue. Exactly. >> So you say that AI is a national security issue because we need it to defeat China and that's going to stop you getting to the data center. >> Yeah. When people are disgruntled and angry and unhappy because their lives have changed so much uh they will then be criminalized classified as terrorists if for example direct action against data centers is is a way in which they express and try to shape shape things. That's what I I think I'm most scared of. >> Does that make sense? So in the sociology of the people talking about this because there is there are people who are very like Silicon Valley tech future is pilled >> and some of them and they've all been talking about this for for over a decade by the way. Um the possibility that AI is going to kill us all. >> Yeah. Yeah. >> Right. Amday um you know Jacob Cox I mean he's very young but on the blogs that they're on this has been the number one issue for like 10 years from those institutes you're talking about that they were funded. Right. So I don't think they've just cynically come up with this as a problem. And and then there's also within this crew the people who are also tech optimists but the accelerationists um this are kind of David Saxs um Andre these guys and they're they're probably more than anyone keen on these data centers getting built and they're saying no the risk is not real we don't need the regulation. So it I I I see the logic of your argument but I don't think it maps sociologically onto who's making what claim. But I don't think I don't really see why that that matters because I don't think like industrialists when they were building, you know, mills and whatever were making claims about how, you know, they they were thinking about what that's going to do to the larger economy, not what it's going to do to people um or what it's going to do to the environment. Those were never considered at all by those people. You know, if you look at people like effective altruists who are all over the place amongst these these people, you know, they spend a lot of time thinking about absolute fantasies and being really clever about how to think about absolute fantasies and sort of playing out like the consequences of, you know, like a future in which we are all all disembodied and living, you know, living on living robotically on other planets or consciousness has gone gone on beyond the human human posthum. >> Are we confident it's a fantasy? But it doesn't really matter whether it's I mean what they're doing is they're engaging in one part the part that they maybe they maybe most enjoy they find most sexy whatever of all of the kinds of things that might spill out of this technology and they're not engaging in other parts that actually affect like normal people right now >> I suppose I'm >> and that's what I mean by it's like these institutes right they all sit together they all like have the same kind of obsession about the same things because it's like really clever to have those obsessions and to think really hard about them but there are actually practical issues uh that are probably much closer at hand than extinction which are not being thought about. >> So what I mean is so oh so you're saying >> I think there's a sociology of like smart people >> but is the con is the connection between sort of there being classing attacking data centers as terrorism. Is that related to the people talking about existential risk or is that two separate things? >> I think it's just two separate things. I think it's just something that I think that there's kind of >> you're saying that's the that's what you're most worried about not that that's why Daryay is saying this. >> No no exactly. No, I'm just saying that that that that you know he's saying certain things have to be addressed whereas I think actually there are other issues which are more politically pressing that that are addressed like ownership rather than oversight for example. >> Yeah, that makes sorts of sense.