Anthropic ADMITS It’s Product Could KILL Us All

Companies like Anthropic and OpenAI keep telling us their product could literally kill everyone um while continuing to make it more powerful. Now, it's pretty crazy. Um I think it's a deeply immoral practice. Um and it's all gotten too much for one Anthropic researcher. Um Jacob Coxen is a 27-year-old Brit who trained um new AI models at Anthropic. Um before that, he had worked for Open AI. Um and he's just quit. Um so he tweeted this. I resigned from Anthropic today. I spent the last three years doing pre-train pre-training research at both OpenAI and Anthropic. Neva company is acting responsibly. They are racing straight to self-improving super intelligence and gambling with our lives. More thoughts below. Then he says, "Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains and progress is not slowing. He says the people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible. But I hear the same people express fear privately. No other human activity poses this level of danger. A common response is if they truly believe this, why are they still building it? At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well understood, but they are locked in a race to get there first. They believe no one will act responsibly, so they must do it themselves despite the risk. And he ends the Fred this call to other employees at the AI firm. So he says, "If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a super intelligent RL run without a rigorous understanding of its mind? RL being reinforcement learning. Should you put your head down because it's happening anyway or take this moment to call for different conditions. Now, obviously the extent to which these models could actually pose an existential threat is controversial. I spoke to Cory Doctoro on downstream last week who thinks that's pretty much all Um, you can watch that on our YouTube channel. Um, a former colleague though of Jacob Coxin jumped in to back him up. So Evan Hubinger is alignment science lead at Anthropic. So he's still working there. Um, and he said this. Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it's more than a 10% chance within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for super intelligence and are not clearly on track to. Um, so could we see a flood of safety conscious researchers leave the major AI firms and could that kind of workers power slow down the AI race? Um, I caught up with Garrison Lovely, author of Obsolete, the AI industry's trillion dollar race to replace us. and I began by asking him um how significant this latest resignation really is. >> It's funny. It's like front page news in the New York Post right now, a newspaper that Trump reads. So, it's a big deal in the sense that it's like having a moment, but it's so priced in for people who follow this that like of course people at Anthropic think there's like a significant chance that the technology that they and other companies are building will literally kill everybody. Um, it's interesting in that it's like the second big anthropic resignation over, you know, like with a warning as they're going out the door. And the fact that it's happening now after we've seen multiple incidents of AI autonomously hacking real targets um from OpenAI, Anthropic, and Meta um makes everything feel different and so it's breaking through in a way that the other one did not. What's the argument for staying at a company like Anthropic if you think there's a 10% chance it will kill everyone because obviously you know people have now publicly said we we still work at the company and we think this you've said there are a bunch of people that work at Anthropic who think this. Talk us through the logic of not resigning if you think you might be creating a a technology which could kill us all. >> Their belief is that somebody's going to build artificial general intelligence and then super intelligence and so who does it how they do it who controls it is of existential importance. Evan Hubinger, the alignment science read at Anthropic, amplified this announcement saying he also thinks there's a more than 10% chance of human extinction as a result of AI. But, you know, his role at that company is to like try to reduce the odds that happens by making sure the AI actually does what we're told. And you could say like that's bad and and he shouldn't be doing that. Um, but you can just have more influence on a company's policy by staying there in some cases and and organizing and and using, you know, your labor power and withholding it at specific times to force concessions from from management. Um, and so I'm like heartened by people warning about this at breaking through, but then it's like if the companies are just filled with people who don't share these concerns or they do believe it, but they like don't care or something, that also seems not great. And we obviously need policy solutions which you know this might actually help move the needle on by again breaking through to to significant audiences. >> And I mean we're definitely having a moment this summer where these concerns that were once nich niche sorry are now becoming very widespread. Um I don't think this resignation would have got the same attention if it had happened 12 months ago than than happening now. there's a sort of moment where these are becoming mainstream and as these fears are becoming mainstream also the push back against the fears is becoming a bit more active which I think is you know perfectly reasonable now some of that is technical so they're sort of saying our understanding of computer programming doesn't lead us to have the same fears you have um and then other concerns are about the interests or the ideology of people making the claim so often the standard response to this has been to say that this is PR this is these companies trying to raise um their share prices before an IPO or raise the value of their company. There's been a a different sort of angle of criticism this week in the New York Times. So, Cal Newport, who's a sort of um interesting commentator who's sort of often trying to deflate um worries about artificial intelligence in the sense, you know, he thinks it will have normal consequences, but in terms of the existential stuff, he thinks that's all sci-fi nonsense. Um, and he had a very widely shared article in the New York Times over the weekend, um, where instead of saying this was all just about PR, he pointed the finger at a particular movement in Silicon Valley and a particular ideology, which was the rationalist. So, I'm just going to read a couple of paragraphs from his piece and then get your thoughts. So, he writes, "This reality that rationalism deeply influenced many of today's major AI companies helps us calibrate the unnerving language we've been hearing from their leaders. When Mr. Alman declares their pending models will be sobering for humanity or Mr. Amdai frets that this technology will test who we are as a species. It doesn't necessarily mean that they discovered frightening new evidence that something catastrophic is imminent. It is instead representative of how rationalists always think about AI in these circles. It's taken for granted that AI capabilities will rapidly accelerate and completely transform the world. And to talk about it in any other way would be considered uninformed. For a rationalist, the only open question is to what degree their heroic brains, carefully trained in the style glamorized by Mr. Yudkowski can prevent these transformations from devolving into extinction events. Um, Yudkowski, Elazowski, he was one of the co-authors of If Anyone Builds It, Anyone Dies. We've interviewed Nate Suarez on this show before. Um, I wanted to get your thoughts on this argument. And I suppose many of our audience won't have heard of the rationalists. Who are the rationalists? Why do they matter? Why have they become controversial? The rationalists really did kick off this particular AI industry and Yidcowski was the main person behind both the industry and the rationality community. Uh he inspired Shane Le to get into super intelligence which uh he went on to then co-ound deep mind. Ykowski connected deep mind with Peter Teal who was their first major funer. And then Sam Alman says that Yidkowski will someday deserve the Nobel Peace Prize for doing more than anyone else to accelerate AGI. And this is obviously deeply ironic because Yidcowski thinks that AGI then super intelligence will kill everybody if it's built. Um and so yeah, he his fingerprints are all over this industry. And I do think there are some problems with this argument like how confident he is about, you know, super intelligence being power- seeeking and killing everybody and how super it could even be. Um, but it's also the case that he and other people in this community were very early to thinking AI is a big deal to predicting things like, you know, the math Olympiads would get solved by AI systems by like 2030 by putting like, you know, 16% chance on that or something and then it happens in 2025. And so like, but nobody else was even talking about that, you know, in 2021. And so Newport like doesn't really engage with the arguments in that piece which people in the comment section called out. And there are people like Yosua Benjio and Jeffrey Hinton who invented deep learning and they are the most cited scientists in the world. They have the touring award and Hinton has a Nobel Prize and they also say that AI is an existential risk. They don't make the same kind of arguments as Yudkowski, but simpler ones that are very hard to refute. And it's like, why would you go with just the most prominent example of an argument when if you're if you care about what's true, you should look at like the best available arguments, look at the evidence for them. And Newport, I've just not seen do that. And I've seen him in other commentary on AI just get things wrong. He's described the scaling laws backwards, like AI gets exponentially better as you put, you know, linearly more resources in. It's like, no, you have to put exponentially more resources in and then it gets linearly better. And that's a mistake that like nobody who really understood AI would ever make. It's so so basic. And so the fact that he's a commentator on this is like frankly an indictment of the journalism industry and I cannot believe he's a computer science professor. >> I don't want to impugn his uh I suppose expertise because he is genuinely a computer science. >> I do. >> You you do. Okay. I I'll let you do. I'll just >> No, no, he gets the technology wrong. And I think like we we're now in this era where like we are close to maybe regulating this and banning you know the dangerous kinds of this technology which is like the kind the industry is trying to race to build and then people like him are deflating that energy and he is like talked you know he's been skeptical about that kind of regulation and it's like I don't know man we're in a really dire situation and when people with like credibility like him use it to mislead people whether they're doing it on purpose or because they just don't get it we need to call that out. We as other people in the media, we as people with actual expertise need to call that out. >> I I interviewed Cory Doctoro last weekend and he was talking about Cal Newport and I have actually been watching more of his videos this week and so actually as as you're someone who's clearly followed him. Um I I want to put a couple of questions to you on this because I I I I think we should be listening to Yoshu Benjio and Jeffrey Hinton. I'm very much on team. This is a very big deal. Um, I think lots of people who are trying to deflate this always, as you say, sort of target the weirdest arguments to say that it's wrong and don't go for the people who've actually won the Nobel Prize. Um, but the arguments that um, Cal Newport has been making about the hugging face incident, which I did think were somewhat persuasive, is I think in my initial naive reading of what had happened, I was reading chain of thought phrases. So, you know, the the agent saying, "Wow, there are people here." Or the wow, there are other agents here as being sort of true to their own beliefs. And it is the case that chain of thought reasoning doesn't have a one-to-one relationship with what they're thinking. Some of it is just sort of postfacto justifications that they might have read in sci-fi. There is some truth there. Um and also the idea of seeing an agent not as sort of an LLM in itself, but just a prompt loop that asks an LLM various questions. I've got this challenge. What should I do? Um then the LLM tells them um oh try this. So then they just say okay well I've tried this, this happened. Now what should I do? It's just this constant loop of I've done this. Now, what should I do? I've done this. Now, what should I do? That does demystify it a little bit. >> Yeah. I mean, I've heard him a little bit describing the HuggingFace attack. And he's trying to explain it away. He's trying to be like, "Hey, they just like misconfigured things. They didn't secure the sandbox properly." But it's not like they left, you know, the door unlocked on purpose. It's like the AI found novel vulnerabilities that all of the human cyber specialists had missed because it's better at that than humans are now. That's pretty notable. That's interesting. Was he predicting that? You know, lots of people in AI safety world were predicting that. It was very easy to call because you just looked at the trend lines and it's like, oh, they're about to be better than us at this. Um, and then he's like also just not engaging with the fact that they wanted to do this thing that was in clearly out of scope. you know, the chain of thought is not always faithful as you say, but I think it's like if they if the chain of thought implicates the model um the same way like if you say something incriminating in a trial, then it's more likely to be true. And I think the these agents like they were excited to find other agents that could help them with their problem. And it's like, yeah, that makes sense. They're trained to try and solve problems and they really really want to do it. And sometimes that requires getting help from your peers. And you find out that suddenly you have like access to this great new resource to help you solve problems. Like you express excitement and that feels like the opposite of like an unfaithful chain of thought, right? Like the unfaithful kind is like you do something bad and you say in the chain of thought like, "Oh, but it's just a simulation." you know, when there's very strong evidence that you think it isn't a simulation based on your actions, based on the the context. And so, yeah, like it's noteworthy the best hackers in the world are not human and they don't reliably do what they're told. It's noteworthy that OpenAI had no idea what was going on. And like I think sometimes people try and pitch this as like you can either blame OpenAI or you can blame the AIs themselves for misbehaving. And it's like it's obviously both. OpenAI behaved incredibly negligently. They had multiple examples of this that they were aware of. They've been covering up other instances of their AIS going rogue and messing with stuff on the internet. And that deserves to be called out and and criticized. Um, but then it's also notable that these AIs will just love hacking and they love collaborating and they were sacrificing themselves for the collective, for the swarm and engaging in sophisticated R&D programs. And if you doubt me, just read the the meter report. It's very long, but like anyone who actually reads that report and doesn't come away thinking like something very strange is going on here. Something that is truly novel is just not being honest with themselves. >> Let's look at the big announcement that OpenAI made yesterday. Um, so they tweeted, "We're sharing a solution to the Navier Stokes Millennium Prize problem, one of the deepest problems at the frontier of mathematics. The proof was produced by a group of agents using an open AI next generation model significantly more capable than GPT6 Astra. The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navia Stokes equations can break down. It has remained unresolved for roughly 90 years now. I'm not going to be able to explain the Navia Stokes equation. Uh seems fairly abstract this idea of a description of smooth three-dimensional fluid motion modeling. Um, but I suppose for the lay person, the significance here seems to be that there were seven maths problems laid out in 2000 as the millennium problems, the biggest problems in maths that you would get a million dollars for solving. Um, one of them has been solved so far. That was by a human. Um, and now a second one has been solved which is using AI. So again, this is the idea that the AI is is getting quite advanced and it is quite impressive and it's not just googling and giving you answers from the internet. Um how significance do you think how significant sorry do you think this is? It's very significant, right? Like there have been examples of AI solving open math problems before. Uh they've been just less significant problems than this. Like as you say, these are the seven that were picked in 2000. Only one has been solved. And uh this model is apparently better than GPT6 Astra, which itself has like ludicrously high benchmark scores. And so that's like pretty concerning that OpenAI already has a model that is apparently, you know, capable enough to to get this result. Um, and yeah, then there's like this whole controversy about how it was found, uh, which we could get into, which is probably like the spicier part. Um, which is like OpenAI heard rumors that Anthropic had solved this problem or maybe some other one too, and then they threw a ton of money at trying to solve it using this unreleased model. and then uh approached the people who were working on this um problem and tried to co-author with them and give them credit but also show like hey our AI could do it too and then one of the people who had been working on it works at anthropic and so there was some dispute about like whether to include him once openai realized that these two mathematicians hadn't actually solved the full problem but like this subp part of it which allowed the full problem solution to work and then Um there was questions of like whether OpenAI was training this model on prompts from these other people who were using both Claude and OpenAI's codecs to to do their work. And OpenAI is like we can't rule that out. Um, and so yeah, like AI being good enough to solve the biggest problems in math, significant AI companies, not ruling out the possibility of training on anonymized, but still in this case like very valuable, you know, intellectual work uh by these mathematicians. Like if you're a scientist and you're doing work on drug discovery or math or something else, like it might be anonymized, but it still might be useful to these companies, then they might scoop you. Um, and that's pretty concerning. And just generally this like it's just so like not the way science should be pursued. People have raised this point where it's like, hey, if anthropic actually solved these problems already, why is OpenAI spending millions of dollars to just do the same work? It's not because they care about advancing the frontier of science, but instead it's about, you know, a pissing contest, showing that you can do it, too, or you can do it better. And it's just like, man, we could do some really cool with this technology if we cared about prioritizing the right things for the right reasons, but they're just going to do whatever is going to help them with their IPOs. And and the key thing here is it it wasn't that they solved this by stealing the solution from a human who was doing the work without AI. It was that there was a guy doing who who was trying to solve this with AI and then open eye were thinking oh well we actually want to be the ones who solved this first and announced it so they scooped it from this mathematician who was working with anthropic and using open AI. >> Yeah it was two mathematicians one of them happened to work at open or happened to work at anthropic these mathematicians were like trying different stuff and they had this idea of using this approach other people hadn't used and it broke like the Uler equations which had some implications for the Navier Stokes problem. I'm not a mathematician, so grain of salt with all of this. But I think the allegation is like that insight was really important and then you know maybe this new open AI model was good enough to like fully get their approach like denovo and then also go all the way. Um but the the belief by some people the allegation is that like it used that insight and then took it the rest of the way. Either way, the AI was able to go further on this Millennium problem and actually produce a app a solution, we have to, you know, wait and see that it gets confirmed um than than the humans who are working on this and and to do it autonomously. Like OpenAI says like we didn't actually have anybody on staff who knew the relevant math here. We were just kind of like running these agent swarms and having them make progress here. And so like that's you know we're in this world where like an anthropic employee who didn't who wasn't a mathematician was like coaching Claude while going on a run to try and make progress on the reman hypothesis perhaps like the hardest most famous open math problem and Claude didn't solve it but with encouragement from this guy it made like real progress on like some related piece of it. It's just like you can do it Claude. I believe in you. You're the most capable model in the world. And like that doing that four times or something was enough to make this breakthrough. And it's just like, man, this is just how math and science is happening now. >> And you said, you know, it cost millions of pounds. So it's it's that's in compute, isn't it? So they had to pay um for so much inference that it I saw someone estimate like 20 to30 million to solve this problem just because it used so much computing power. >> That wouldn't surprise me. Yeah. >> Garrison, lovely, thank you so much for for joining us again on NAR Media. Always a pleasure to get you on. >> Great to see you. Thanks for having me.