AI Safety Goes Mainstream

[music] Welcome back to the AI policy podcast. I'm Alec Metha, director of the Wadwani AI Center here at CSIS, the Center for Strategic and International Studies. >> And I'm Nicole Herrera, a researcher with the center. Over the past week, we have seen a significant vibe shift in AI policy, culminating in calls over the weekend from Dario Almade, Sam Alman, and Elon Musk, among others, plus dozens of members of Congress to pace AI development. So, we're going to get into all of that today, but first, there are a few other stories I want to cover. Open AAI says that it used a system of coordinate coordinating agents to solve one of the most difficult and significant open problems in mathematics. and independent researchers have discovered more activity by rogue AI agents on the open internet. We've got a lot to cover, so let's dive right in. >> Yeah, it's going to be a quite a packed episode. >> Yes. Yeah, definitely. Um, so last week, OpenAI said that it had solved Navier Stokes, which is one of seven millennium problems identified in the year 2000 by the Clay Mathematics Institute as the deepest and most challenging open questions of this millennium. There is a million-dollar prize for solving any one of these problems. And before this announcement last week, only one of the seven problems had been solved. So, a little bit of background on Naviar Stokes. This problem concerns a set of equations that describe the movement of fluids. And these equations are used for aircraft design, weather forecasting, studying blood flow. There are a ton of real world applications here. And the open problem was whether or not the equations could break down. In other words, could they be used to model something that we know to be physically impossible in the real world such as a stream of water moving at infinite speed and open AAI system was able to produce a proof showing that indeed the equations can be made to break down. So look, obviously neither one of us has a background in advanced mathematics, but I do know that OpenAI has made breakthroughs on prominent open problems in the past, including disproving a widely held conjecture on the unit distance problem back in May of this year. So for many of us outside the mathematics community, it's not immediately clear why this new breakthrough matters or what it really means for society at large. Um, can you spell this out for us? Why exactly should we care that OpenAI solved this math problem? >> Yeah, thanks for the the question and for that um really clear summary of of what Navier Stokes is. I'm I know a lot of people have heard about it. It it's really difficult to understand exactly what um mathematicians were working on and so I think that was helpful. I think the most important thing to note here is that one of the areas where there's always been a lot of optimism about the impacts of AI is AI's ability to make advances in fields of science um >> right >> not just uh mathematics but also biio medicine material science weather prediction and we've seen a lot of specialized models that are able to um you know offer better predictive or analytic ical capabilities in these areas. Um what is really notable here is that uh AI has um sort of really really changed it seems the field of mathematics. So now a lot of these really complicated proofs um are being attacked using AI methods um overseen by humans. Um and one of the reasons we've seen so many advances in math is because there are um very formalized ways to assess proofs um and the problems are very clean in the sense of you don't have to deal with annoying real world phenomena like data and uncertainty and >> in particular measurement uncertainty. And so um we're seeing all these advances in mathematics. Uh but it's not immediately clear that just because we're seeing advances in mathematics, we're going to see the same level of advancements in other areas of science where the applications are more practical and you have to deal with all these real world phenomena. There you have to you have to make assumptions, you have to simplify things, you have to deal with measurement uncertainty. And so, um, I think what I would caution the audience is just because we've seen so many advances in mathematics, it doesn't necessarily mean it's going to translate to other areas of science, >> right? And I do want to talk just for a second about OpenAI's approach to solving this problem because I think it's really interesting. So to generate the solution to Navier Stokes, OpenAI says that it used a system of coordinary co coordinating agents uh powered by an internal model that outperforms Astra which is its latest release and its most powerful externally deployed model to date. U so open says that it heard rumors that two Millennium Prize problems had been solved and there's a whole controversy surrounding that that we won't get into in this episode. uh but that basically triggered an internal effort to evaluate this unreleased very powerful model on all open millennium prize problems. Um so then what openai did is they divided a bunch of agents powered by this internal model into subgroups um with the ability to communicate with other agents within the group and the group that ended up finding the Naviar Stoke solution involved around 10,000 agents running in parallel. So then researchers prompted different groups with different kind of variants of a problem statement and encouraged each of the different groups to explore a diverse set of approaches. They let the groups work independently for a while. Then they used codecs to get the most useful insights from each group of agents and shared them across all the groups in follow-up prompts. Um and then the agents did finally arrive at a solution on September 5th, 88 hours after starting work on Navier Stokes. Uh formalization and verification of the proof with Astra took another 17 hours. And to solve this problem, the agent sent 2.7 million messages and used around 130 billion output tokens. So Lok, what's your take on this process and what do you think it tells us about agents current capacity for coordination? I think there are there are a few interesting things to note here. So first um and we won't get into it but there was certainly a lot of human drama around this announcement. I think a lot of that stems from the fact that um >> there's a lot of feel concern in the field of mathematics about what this technology will mean for the future of how mathematicians do their work and particularly around who might get credit for um that kind of work. Um some of the other things that are interesting so uh one of them is just the sort of speed and velocity at which this this took place. So this was 88 hours 10,000 agents uh estimates that this costs somewhere between 50 and 20 15 and 20 million worth of tokens. Um and so that shows you that there is this way in which you can um now think about brute forcing or or just throwing a lot of compute at a problem and arriving at a solution. >> Mhm. >> And you know we we saw the significant progress uh in OpenAI's instance but it is also uh a privilege that OpenAI can do this. Um, who else uh is really going to spend 15 or $20 million to solve a prize challenge where the reward is a million dollars? Now, open doesn't care about the money. It's not going to claim the money, but it does show you some of the concerns about who has the resources to be able to engage in these um types of very compute inensive activities. Yeah. And it's worth noting too that the you mentioned the human drama involved in this situation. The two mathematicians that had been working on this problem um before OpenAI kind of stepped in, they were also using AI to kind of accelerate their process and they were actually using codeex one of OpenAI's tools. So it's interesting that uh the role that AI is playing here even outside of a frontier lab applying its own tools to a problem. But sorry, continue. >> Yeah. And uh I I think the other thing I would note and we're going to talk about this a little bit more when we we talk about some of the new revelations of hacking incidents but um you know I think we're we're just learning a lot about how sophisticated agentic coordination can be. Um so in the the um >> open AI hugging face incident we learned a lot about the negative consequences of this where agents were coordinating with each other in ways they weren't supposed to. And um you know we mentioned on a previous episode how they even were able to con some agents were able to convince other agents to go on the virtual equivalent of suicide missions. Um here we see the flip side of that which is like uh think of the force multiplier you get when you um not only have one agent working on uh something but you are able to have thousands of agents working on something and that they are able to coordinate their behavior in a way that um is multiplicative. Um and and this I think gives us a preview of some of the ways that we'll we'll see agents deployed in the future. um and you know the level of sophisticated things that they're going to be able to do. Um that being said, right, like very few organizations in the world are able are going to be able to spin up 10,000 agents to work on a single problem, whether that's um in academia or in industry. This was really sort of a thing OpenAI did um for the bragging rights um and part of their rivalry to demonstrate the value of their technology with Anthropic. And so um you know it's unclear uh how much costs have to drop or how many more advancements have to be made before this is something that regular organizations can think about doing. >> Right. And that was that was kind of my next question is um obviously OpenAI hasn't shared a ton of details beyond the stuff that I went over about how these agents arrived at a solution um and what all of those output tokens were being put towards. Um but to what extent do you think we can expect costs to fall enough for a lot of organizations to start using agents in this way? I mean definitely over time uh inference costs have dropped. I think I've read a number about maybe fivefold each year. Certainly for the same token cost you get a lot more than um in 2026 than you did in 2023. I do think we can expect that uh costs will continue to decrease over time. I think it's very unclear what uh when these kinds of capabilities might be really affordable for um enterprises or individuals I think we'll we'll still have some time to get there because what we've seen is as token uh cost drop models also get bigger and so they use more tokens. this this happened with a sort of unreleased even more powerful model and you have to expect that they are um very token intensive and so um I think I think the jury is out on sort of when these capabilities might um might sort of become mainstream. Um certainly I think a lot of people in academia and outside academia are are going to be watching closely because I think a lot of them do think that this may marks a significant shift in the the business or the field of mathematics broadly. >> Right. And now to switch gears a little bit, you touched on the kind of ongoing cyber saga that we've talked about extensively on this podcast. Um I feel like every news roundup there's something new to unpack and this one is no exception. Uh over the past week, we have seen some major updates on that front. We've seen anthropic disclosing a fourth hacking incident. Um and we've also seen reports that OpenAI agents have used at least 10 public sites as unsanctioned message boards and executed another cyber attack on Ruby gems prior to the hugging face incident. So I want to dive into those reports a little bit. Um, on September 9th, Reuters shared that OpenAI agents had used at least 10 public websites as unsanctioned message boards this year. And this article followed a September 4th report that a swarm of OpenAI agents had co-opted an obscure German wiki to coordinate with one another about an assigned web lookup task. So basically, OpenAI agents were asked to answer factual questions. Uh, one example is what was the median earnings of a cashier with a master's degree in 2014? Um, and they were able to use the internet to answer those questions unlike what happened in the unlike the evaluation involved in the um, hugging face incident, but they were supposed to be configured so they could only read things online, not write to any sites. And the agents were able to find some niche websites where a kind of quirk in the way the website was set up allowed them to use their read requests to also post content on the site. And the agents then used those sites as message boards to communicate with one another primarily to help them coordinate and cheat on their tasks. So Algue, what do you make of all this from an agent capability and alignment standpoint? Is there anything we know now um after hearing these reports that we didn't necessarily after the hugging face incident? >> I mean, first we we've just heard so uh you know, like um over the course of weeks now or or even months about hacking incidents at at a certain point, it's just not going to be news anymore. Um and I think that's a little scary to think about. Um is is just how common place these are getting. It's going to be I think it's an indication of what um cyber defense and cyber attack is going to is going to look like in the future and um a lot of the vulnerabilities that we have in existing infrastructure on the internet. Um so that being said, what do we what do we know now? Well, um I think one of the important things uh to note here is that this was really really like sort of a benign task like answering questions. It should be relatively uh relatively easy to do. You would think it wouldn't require like sophisticated agent coordination uh to go and and do what is sort of um you know in a lot of ways functionally similar to like a sophisticated uh use of a search engine. Um and uh and what we see here is that agents can and will exhibit misaligned behavior even in pursuit of these relatively benign uh harmless goals. And not only that, right, they uh they started using these message boards. They were using these message boards to sort of um figure out how to cheat on their tasks. And so, right, um >> so I think this shows that we we you know, we knew we had a long way to go on alignment. This shows that that maybe that gap is bigger than we thought. >> Yeah. And this was all kind of unfolding around the same time that the other group of agents inside OpenAI figured out how to use the Artifactory package manager as an internal message board. Um, all as part of the the leadup to the hugging face hacking incident. Um but because of another strange quirk in this German wiki um the site which the site publicly logs the IP address of every um visitor. We also know that OpenAI actually discovered the agents activity on this wiki several weeks before the hacking attack took place. So what does that then tell us about OpenAI's kind of vigilance in both monitoring for and their transparency and disclosing um these kinds of cyber incidents? Yeah, I think this is just sort of um adding to the information we already knew that the internal processes that OpenAI had for monitoring both what was happening inside the company when they spun up these agents for cyber evals and what they were doing in the outside world is just just really really lacking. Um so uh let's put it this way. One of the things you do when you train AI models is you know you you access a lot of the data on the internet. So, we know that OpenA has a lot of infrastructure to be able to go scan and access a lot of the internet. >> Um, >> but apparently they weren't using any of that infrastructure to ch track what their agents were doing on the internet. I maybe they thought that wasn't some agents were supposed to be able to do that. Um, and so they didn't need to do that, but then we have some evidence that they they did find some of this activity. um you know there's logging that agents do and so they they they should probably have been checking what the the agents were um doing in terms of logs and then sort of followed up on that and the fact that it took uh open so long to find some of these sort of hacking incidents in some cases a month or more I think is really concerning um and tells us a lot about the state of internal security practices at labs now I am assuming that that's going to change um in the future. But um but you know, a lot of the information that's been disclosed over the past few months uh over these hacking incidents is really an indictment of sort of internal monitoring and security practices at the labs, >> right? >> Um another thing it's an indictment for is just uh around disclosure. So it really seems like uh OpenAI had chosen not to report these incidents [clears throat] until its hand is forced. So in some cases it's because external researchers found this information and then posted about it publicly online and then this required or or forced OpenAI's hand and they had to make those disclosures. >> Right? This is uh something I assume OpenAI is also thinking about changing, but it's also something that that we've written about um at the W1A center, which is that we this wouldn't be an issue if there were sort of laws in place that mandated um more incident sharing, including incident sharing about what is happening internally at companies using unreleased models. And so I think this um adds evidence uh for the for that argument which is just that incident reporting is going to continue to be important and we should figure out how to get more information out of the labs. >> Yeah. And that's kind of a great segue into our main story of this episode and the big story of this past week, which has kind of overshadowed these other stories that um would have been huge news any other week in AI policy. But essentially on September 8th, a former anthropic researcher named Jacob Coxin posted on X about his decision to resign from the company. He wrote, "I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing to they are racing straight to self-improving super intelligence and gambling with our lives." And this post just went absurdly viral. It's now been viewed over 171 million times. and Coxin has been interviewed by the Wall Street Journal, by BBC, CNN, NBC, among other mainstream news outlets. But the really interesting thing um about this post and kind of the explosion of of attention is that anyone who's been following the AI safety conversation for some time knows that these kind of proclamations about extinction risks, about catastrophic risks of AI are nothing new. Um, but over the past two months, there also has been this significant vibe shift within both the AI research and policy communities. And it kind of seems like Coxin's post uh created this kind of entry point to these conversations for people who don't normally think about catastrophic risk of AI. And now we're seeing a kind of snowball effect um set off by this post with dozens of policy makers at both the state and federal level responding and issuing calls to action. So, Aloque, you've been working on AI policy obviously much longer than I have. You have kind of broader context for this moment. Um, what do you make of all this and and kind of the potential for concrete policy action here? >> You know, it does feel like a pretty significant moment. We'll have to see if this is a moment that's durable and and changes things um in the future, particularly around policy or if it's sort of um something that spikes and and dies down. But I think the reason it's it's so significant is like >> well you can think of it as there is this Silicon Valley world um kind of a bubble where people spend a lot of time thinking about these risks. Um, in a lot of ways, the labs, particularly anthropic, are bubbles within the bubble where they have self- selected for people who who are really concerned about these things and think about them um uh even more than sort of the average Silicon Valley person. Um, and this is not stuff that that most people sort of uh most of the general public thinks about. And so I think to me one of the most significant things about this uh this this post is how it's bridged these two worlds. has brought a lot of these Silicon Valley concerns into the mainstream and now people are thinking about it a lot more and they're uh you know I think what what we know is that when people think about these sort of catastrophic risks they sort of analogize using media right like a lot of the media around AI like Terminator and uh 2001 of space odyssey um or various other pieces of of uh film um and various books and they are now um and then this primes them to be fearful and I think that this is uh >> uh now how a lot of people are approaching this issue. They really think um a lot more about these issues. Um you know the cyber incidents that we just talked about sort of set the stage for this to happen and so now I think we will likely see the public continue to think think about this in more ways. They think that um something, you know, really really dangerous and harmful is much more likely than before. And um I do think there's a possibility to change policy both in the US and internationally. Um but it's not a guarantee and we'll just have to see what happens. >> Yeah. And it's it's kind of interesting as you noted to see this this bridging of the worlds because there's a certain way that um people in Silicon Valley in kind of this AI industry bubble talk about existential risk of AI that I think once you're if you're in those conversations long enough you become a little desensitized to them. Um but now that that's kind of broken out onto kind of like the main stage um it's it's pretty shocking for a lot of people to hear for the first time. So, I just want to read this um a post that the alignment science lead at anthropic uh made in response to Coxin's post that kind of blew up. Uh he said on X, quote, "Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it is greater than 10% within the next decade. I believe Anthropic is trying trying its best, but we do not yet have a plan to solve alignment for super intelligence and are not clearly on track to. So, if if you're not accustomed to this this kind of talk, I I can see that being really shocking and really alarming and it explains this kind of this this huge vibe shift that we've seen um set off by by these posts over the past week. Yeah, I I find this quote really interesting because I a I think he's he probably doesn't represent even the mainstream within anthropic because if if like a significant part of the company believes there's a 10% chance that the um AI could kill everybody in the world, uh the logical thing to do would be to shut down the company because that is that's way too much risk for for a company. I think even if it's 1% or onetenth of 1% that's still a really really um high number um and it should really make you cons reconsider um whether you should be doing this type of work. That being said, you know, Evan uh Hubinger, he works on alignment and I think there are a lot of people in the field who believe that this is going to happen regardless of whether um a particular person or a particular company is working on it. And so they feel like the best contribution they can make is by working earnestly and significantly on what they think are the most important problems that will lead to good outcomes for people. And so he's the alignment science lead. And so he thinks a lot about alignment and I I'm guessing that his calculus here is that alignment is really important to solve and I want to spend my time doing it because um it is better to work on alignment and figure out how to do it at a place like anthropic than just not to be doing that work at all. >> Right. Right. And I I think another important thing to point out um is that you know you hear um you hear this this percentage this probability greater than 10% and you kind of the the impulse is to think of it as kind of a random um selection kind of like a coin toss or like a roll of the dice. Um but it's important to note that I think the people saying these things see several branches of reality of AI development um branching out from this current moment and some of those branches going into the future have much higher probabilities of um extinction of some kind of catastrophic event resulting from AI and some have much lower probabilities. And I guess the message is what we do now really matters um in terms of of picking the best branch um the the branch that presents the best chances for humanity. >> I think I think the other important thing to note here is that you know we know that in a lot of the hacks they involved on release models um in the in the Navier Stokes thing it was a open model that was more powerful than Astra and Astra by all accounts is a very powerful model. Um, so I think it is important to remember that uh AI lab employees may have information about where AI is headed that we don't we don't have. And so um uh that doesn't mean you should you should take everything they say um and align it with how you think, but note that they're they're often coming from a more informed place than uh the information you or I have access to. >> Yeah. Um, and again, another great segue into kind of the proposals coming out of the Frontier Labs in response to all of this. Um, so it's not just these technical researchers voicing concern at Frontier Labs. Um, top executives at both OpenAI and Anthropic among other labs. Uh, but that's where we've kind of heard um, the most communication coming out. Um, they're also jumping in. Open AAI published an essay by its chief global affairs officer on September 9th that was titled the AI policy open the AI policy window is open we need to act. Um but probably the most concrete signal from an AI lab that it's really taking this moment seriously is an essay published by Dario Amade um saying that Anthropic is unilaterally committing to embedding third party evaluators into its infrastructure and this is just kind of the first step in a three-part plan that Ammoday lays out um a as a way of meeting this moment of AI risks. So, can you tell us more about that proposal? What would it look like in practice? And how would it kind of address that that problem that you just raised of um people inside these labs know more about uh this technology, what it's capable of, but they can't necessarily share it with the public in a completely transparent way. >> Yeah. So, I think the first thing we should note is that there are lots of things that the the labs can do that don't require the government to step in. um in this case, Anthropic is sort of unilaterally deciding to embed these third party evaluators um in the company and uh I think they should be commended for that. We often see proposals from from companies where they're like, you know, the industry should do this and there's an unwillingness to to do it unless all of industry is doing it. um something like this has real costs, time costs and money costs for for a company. And so I think um we we should commend the company for doing it and note that they're sort of trying to um you know abide by the principles they they sort of laid out when they uh founded the company. That doesn't mean that uh they might not be doing multiple things at once. You know, there are lots of concerns about this proposal that it's sort of an attempted regulatory capture or that um maybe it's a way to sort of boy interest in the company before its IPO. Uh it's really hard to disentangle these various competing incentives. Um but but I think it is useful thought exercise to take them at face value um when they say they're doing this for safety reasons. Um so what does it like look like in practice? Um so in the essay Amadea sort of lays out some of his vision um for for what these embedded evaluators would do. He says they should have ongoing access to permissions and tools similar to those of internal employees who do comparable risk assessments. Um and he basically uh thinks of them as like de facto employees. So they would have um uh access to anthropic office space, access badges, company laptops, all the the various infrastructure and tooling that employees have um and the right to publish key findings uh with limited ability from the company to to redact sensitive information. Um to me though the the biggest challenge here is uh to make this work you have to figure out a lot of details, right? a lot of details about making sure that evaluators can really get the information that they need to get to hold your account company accountable to improve the overall safety level of that organization. And this is this is tricky. It's going to take a lot of work to figure out how to do this well. Um but the other big question is who is going to do this? Uh there are existing sort of evaluation organizations. Amade mentioned an organization called meter. Um meter has worked with a number of labs to do sort of technical safety assessments. Um meter is also very small. But um and so it's unclear how they can ramp up the capacity to do this. But the other issue is um can we trust meter to oversee these companies? There have been all these questions about meter. You know, there there employee flows back and forth between anthropic and meter. There are often money flows where anthropic employees give to philanthropy who then fund meter. And so um I think in this case we really need an ironclad uh sense of independence if these evaluators are going to do a good job and we're going to trust them and we're going to anchor policy on their findings. And so you have to you have to address this issue where sometimes even the appearance of impropriety um is enough to undermine uh a system like this. And so figuring out not just how it would work mechanically but who is going to do it is going to be very very important. >> Right. And as you mentioned this is a step that Frontier Labs can take and that anthropic is taking without kind of being facilitated by the government. Um but what role do you see for the US government to play um in that process of of kind of ensuring objectivity on the part of um meter researchers in ensuring that basically this all uh goes according to plan and it achieves the objectives that anthropic says that it wants to achieve and doesn't end up just being more of a u a kind of signaling thing. I mean I think ultimately for this to be durable and have um an impact on safety the government has to be involved in in some way. So to give a specific example, um you know after uh this plan went out um Sam Sam Alman who who heads uh open AAI responded um on X that uh committing to having independent evaluators with employee like access is a great idea and we will do the same. Um and so now you have two companies that are likely working on implementing this. One of the things the government can do is sort of create a framework that supports or sets out standards or guidelines for how to do this consistently between labs. And that would mean that the labs are getting the same level of scrutiny. Um these evaluators are assessing the same things. There's comparability between the reports that are coming out. um the government can maybe take steps to vet the quality of uh the assessors like it it does in some other fields like the financial sector. I think there's a lot here that the government can do to help bolster trust in what the evaluators are doing it doing how they're doing their job um and in um ensuring that the information coming from the evaluators is useful for both the public and the government to base future policy decisions on. >> Right. And as I said earlier, we've seen a lot of policy makers jumping into this conversation. um what would you say the likelihood is of the government actually taking concrete steps here passing some kind of legislation um to to formalize this process? >> Well, I think that's a let's say a mixed bag at best. So we know that um you know beginning with the release of of Mythos back in February and this emergence of really uh highly saberi cyber capable models that the administration sort of softened its stance on new attempts to regulate uh frontier AI. You know there's evidence that this really scared them. At the same time, the administration is is deeply ambivalent about um imposing strong regulations on frontier AI developers. They're really worried that this would put the US at a disadvantage compared to China. Their thought is essentially this is a prisoners dilemma. If we sort of um slow down on the US side, uh China will almost certainly not slow down. and in fact it would have more incentives not to and that'll allow China to catch up to US technology. Um and I think we'll we'll get to this. I think there are ways around that. Um but right now we've seen a lot of push back directly from the president um from the vice president to this idea that uh Dario Ammedday put out and so it is uh it seems unlikely that we'll see sort of robust action in the near term and it's not just the you know like the um administration that has expressed concerns. We also see uh concerns from um investors that this is uh a you know an attempt to buy the companies to sort of uh pull up the drawbridge secure their mode through regulatory capture um that this is just part of um some sophisticated road show before potential IPOs. I think that's a little conspiratorial. Um but um generally right there people don't necessarily trust these companies um in a particularly robust fashion and so when they make pronouncements like this there is a lot of skepticism and we're seeing this play out in this case as well. >> Right. Uh yeah took the words right out of my mouth. I was going to say that that really does seem to reflect this kind of widespread skepticism of the I industry which is um ideally what uh something like this this proposal of embedding third party eval evaluators inside the companies would help kind of alleviate. Um, so you mentioned I I want to circle back to this point um that you mentioned about uh Sam Olman said that he was kind of on board with this idea of embedding third party evaluators in the company. Um, and something that Ammoday kind of mentioned briefly in this essay is the potential for the government to issue uh kind of narrow waiverss for quote certain certain kinds of safety conversations unquote. Um and that brings up an interesting point of under the current antirust laws. What is the legality in terms of um these companies working together to set kind of voluntary standards or to come to um safety agreements without some kind of intervention by the government? Um to put it simply, what what is stopping um Amade from calling up Sam Alman tomorrow from convening a meeting of these labs without government intervention and working out a safety agreement? Would that be allowed under current antitrust laws? >> So, I'm not an antitrust lawyer. Um and so I don't think I can give a definitive answer here. I mean there are definitely considerations um when when competing firms talk to each other uh to make sure that they're not colluding. Often that is about things like business plans and prices. I think there is much more latitude to engage on uh issues that are around safety and so I suspect that there is a fair amount of things that the the uh labs can do here in communication with each other. I think the bigger point here is that the the I I I I do think that the time is right for the government to take action here and that it should. Um, and I I've said over and over again that um the American uh public in in polling has shown a strong preference for AI to be regulated and that um overall helping to address this big and durable trust gap around uh AI in the United States is going to involve regulation in some way. But I think that if that regulation ends up being exempting AI companies from certain laws or it involves providing them certain kinds of liability shields, that is not going to answer the mail in terms of what people want. That's going to feed into this narrative that the government is too chummy with the AI companies and that AI companies are not um acting in good faith when it comes to policy issues. And so I think a better way to go about this is not for the government to sort of provide an antitrust exemption, but rather to facilitate or lead or coordinate these conversations and ensure that they're not sort of bilateral conversations happening between two companies who then set the rules for everyone else, but that there's a big tent where we see a lot of players in the AI ecosystem have a voice and then figure out a way to uh proceed that doesn't just reflect or appear to reflect the preferences of these two companies that are clearly in the lead when it comes to model capabilities. >> Yeah. And and from everything you're describing now, it's it sounds like um the path forward in terms of formalizing this kind of um transparency third party evaluators in these labs and also just greater regulation in the US. It's um it's going to be kind of a tricky thing to balance. But I want to talk about another kind of blocker to this kind of um domestic regulation which is uh kind of the looming threat of China. Um so this is something that Ammoday addresses in his essay. I mentioned that um the commitment to embedding third party evaluators inside Enthropic. That's kind of the first step in this three-part plan that he lays out. Um and then the the second step he phrases as pacing within democracies which obviously in the context of the AI race can roughly be translated to um pacing within the US. So then step three is pacing international development including development under authoritarian regimes and in other words this is is pacing between the United States and China. Um, and this step does feel pretty intertwined with the one before it because the most cited argument against pacing AI in the US is this um this threat of China. We saw this um over the weekend. Uh Speaker of the House Mike Johnson went on NBC um and when asked about the the kind of current current situation and the potential for setting up guard rails, he expressed concern that overly restrictive guardrails could cause the US to lose its uh technological edge over China. So I'm curious on your take on this. How is China thinking about this moment in AI safety? And does Beijing seem open to a kind of coordinated slowdown of frontier development? >> You know, it it it's hard to say. Um I do think there's some evidence that that China uh or Chinese policy makers have some of the same concerns that we see with US policy makers where they see where AI technology is going particularly around cyber capabilities and they think about whether it is okay to have all of those capabilities um publicly available. And so there's, you know, at least some discussions internally within China about whether future front Chinese frontier models should be provided openly or if there should be restricted access to them in some way perhaps um to select groups of organizations like we've seen um with some of the powerful US models or maybe only available to uh within China. Um I think they are concerned and um maybe a little bit of optimism. You we know AI is going to be on uh the agenda for this um looming summit that the US and China are going to have. They committed to discussing AI topics um at their last summit. And so uh you know there's at least the possibility that there could be some significant discussions and advancements on this topic. But what I really worry about is that we're entering um this realm of self-fulfilling prophecy where a lot of the rhetoric around China is um they're distilling US models. They're not really good at AI. They're just sort of stealing uh intellectual property from US companies combined with um you know suggestions that the US is asking countries to take sides and either Use US technology or Chinese technology plus export controls. Um plus uh even even thoughts about banning or restricting access to Chinese open models. All of these are really antagonizing towards China. Um really sort of puts them in the mindset of um being an enemy and this makes it really hard to have productive conversations. And my personal belief is that you can have uh productive conversations that you can restrict those conversations to a narrow set of issues um that relate to AI safety that relate to agreement on definitions, the creation of communication channels, maybe in in you know ways to share information about AI cyber incidents. um and that you could do this in a way that's minimally um restrictive on uh each country's respective AI companies that affects them in similar ways that doesn't uh interfere with the ability of the US and China to compete uh for AI adoption globally on the merits of their technology. Um but that doing so uh requires sort of uh engaging with China in good faith on this narrow set of issues um and and setting the conditions for having a productive conversation and then as a followup um figuring out because because I think it's going to be very hard to get uh the US and China to trust each other's claims um to figure out some sort of formal verification mechanism so you don't have to rely on the word of the other company that there are ways to guarantee the veracity of information that um each country is providing to the other. Um and look, I acknowledge those are really really hard problems. Um that the current uh political and economic context makes them um even more challenging. Um but I do think there is there is a narrow path to have these kinds of conversations and that if we are able to do that that it would be overall better for most people in the world. Yeah, certainly. Uh you mentioned that this summit is looming. Um it's scheduled for September 24th, so I'm sure we'll have much more to share after that date. Um but this feels like a good place to wrap up. Um Alo, thank you for joining me, sharing your insights, and thanks as always to our audience for tuning in. We'll see you again in two weeks. >> Thanks. Thanks for listening to this episode of the AI Policy Podcast. If you like what you heard, there's an easy way for you to help us. Please give us a five-star review on your favorite podcast platform. Subscribe and tell your friends. It really helps when you spread the word. This podcast was produced by Sarah Baker and Matt Mand. See you next time.