AI Agent Containment Failures: Technical Realities and Policy Responses

Hello everyone. Uh my name is Alek Metha and I'm the director of the Wadwani AI Center here at CSIS. Uh thank you for joining us and the Institute for Law and AI today um for today's event. We're thrilled that so many of you could make it um both online and in person even in the middle of vacation season. So over the past uh four years ever since the release of chat GPT um AI models has marched steadily upward in cyber capability in coding ability, time horizon, agent capabilities, vulnerability detection and more. That has culminated in the events that have brought us here today. Multiple incidents in which cyber agents autonomously escaped their sandboxes, found their way to the open internet and hacked third party organizations. Two OpenAI models were responsible for breaching HuggingFac's infrastructure. The models did this because they suspected HuggingFace had the answer key to the test that they were performing. Soon after, both Anthropic and Meta said that they had models uh under testing that escaped containment via a misconfigured evaluation environment operated by a vendor. Science fiction authors have long been writing stories about artificial intelligences escaping their shackles. AI researchers have theorized about rogue agents. This is no longer fiction or theory. However, agents are now acting autonomously and causing real world harm. This raises serious questions both about how Frontier AI labs are handling safety and about how government should oversee such powerful technology. To help answer those questions, we've assembled a great lineup of experts. Today, we'll hear hear from Hugging Faces Ian Reynolds about exactly what happened during the hack and how the company responded. We'll hear from Helen Toner of the Center for Security and Emerging Technology, the Institute for Law and AI's McKenzie Arnold and Matt Pearl, director of the Strategic Technologies Program here at CSIS about what policy makers should be doing in response. McKenzie and I have both written reports about this issue which I hope you'll check out. But first, it's my pleasure to introduce Representative Sua Subramanyam of Virginia's 10th district who will be joining us virtually. I first met Representative Subramanyam in 2015 when we were both working on technology policy at the White House. Uh since then he has continued to be a thought leader on technology issues first in the Virginia House and Senate and now in Congress where he serves on the committee on science, space and technology and the committee on oversight and government reform among others. In a recent letter to OpenAI CEO Sam Olman, Representative Subramanyum and other lawmakers flagged a number of unanswered questions about the OpenAI hugging face incident. Representative, thank you for joining us today and we're looking forward to hearing what you say. >> Hi everyone. Thanks for having me today. Uh thanks for joining. I'm sorry I can't be there in person, but uh appreciate you taking the time. I I uh I'm very uh this is my first term in Congress and it's been a very eventful Congress and uh you know what's in the news right now certainly you know wars abroad and immigration and uh higher costs but uh there's a lot going on but AI has been a part of the conversation and I think people are starting to understand and realize both what AI is and two uh what are the potential consequences of it, especially the negative consequences of it. And this is a change from uh 10 years ago when I was working on AI policy at the White House. A lot of people didn't even know what AI really was and didn't quite know how it worked. Now, I think we have that level of awareness. Uh but it's uh stoking a lot of uh fear and concern. And so, you know, the hugging face incident is one. Uh and you know, Mythos is another. And uh I I think I was referred to as a a thought leader in the space. I I like to think of myself as a thought doer and one of my frustrations has been that I would like for Congress to do something. And um so we're almost done with this Congress. Uh seems like it's been a long time. Feels like it's been a long time, but we're here now and we have a a small window uh left of opportunity to do something. We can mark something up at September and then actually do something, you know, after the election, the lame duck session. And that's my goal is to try to get something done. Uh and so I've been working with in a bipartisan way. It has to be bipartisan whatever we do because you know the I you know the the path to having a bill signed into law right now and that's just the way it is. And so um I'm I've worked with um Congressman Obernoli and Treyan to introduce a bill called the Frontier Act. And I can run through it really quickly with you but essentially there is a narrow preeemption in the bill. I know preeemption was a big uh issue before, but there's also, you know, a lot um there's a binding minimum standard for safety frameworks, two-tiered auditing system, more robust risk audits, an emergency shutdown authority, too. So, there's a lot in this bill and um it's not perfect. Uh there's a lot of things that um we're trying to fine-tune in the bill and we're going to be working on that in September, but it's the best start that I've seen uh of anything that has a chance of passing and certainly anything in recent history. Um but one of the things it doesn't do is is you know the main topic of the conversation today which is explicit prescription on uh containment failures. And a lot of what we're doing in the bill is encouraging third parties to be involved uh with the companies along the way to make sure there's uh containment in place. Um it's assumed in the bill that we're going to be doing a lot on containment, but one of the things I'm going to be working on is trying to see if we can get explicit um containment prescription and guidelines u in the bill to do that. And that's I'm hoping that by the end of the year um you know the example of hugging face or mythos that we as a congress and as a federal government have at least worked towards solving the big concern about an AI model um getting out and you know going a little crazy and and doing a lot of harm to a lot of people. And so um that that's sort of where we are right now on the congressional side. I'll just invite everyone uh to stay in touch with us and and work with us um long term. Um if you want to um if you have any ideas on things we can do when it comes to containment, uh any uh if if we were to prescribe containment strategies, for instance, instead of just leaving it to companies, you know, doing things voluntarily or with the third party that comes up with their own guidelines. Um I'm interested in hearing what people think about that. Um but, uh I appreciate the conversation. I'm looking forward to hearing the rest of the conversation and uh uh thank you again for having me today. Uh thank you so much for those remarks, Representative Subramanium. We're looking forward to seeing what you're you're planning to do next. And um as a representative said, uh he's interested in feedback. So I encourage you to um send your thoughts along to the office. Um as a representative's letter and comments highlight in order to figure out how to respond to these incidents, it's important to understand exactly what happened. And for that, I can't think of anyone better than to provide insights than Ian Reynolds. Ian Reynolds is AI policy manager at HuggingFace. Um, but I'm happy to note that before that he cut his teeth in the policy world as a post-doal fellow here at CSIS with the CSIS Futures Lab. Um, Ian is planning to leave some time for questions. For those in the audience, the way we're taking questions is we have a QR code on screen. So, you can scan that QR code and submit questions that way. For those who are joining us virtually, there is a link both on the event page and on the YouTube page um that will take you to the question form. Um, so we're looking forward uh to to hearing um your questions and um also the remarks from Ian. Ian, thanks for joining us today. Yes. So uh hello all um so nice to be here back good to be back at CSIS as mentioned I was here as a post-doctoral fellow. So, so lovely to return and thanks so much to CSIS for hosting this event and giving us the opportunity to discuss the incident. I think this is is really a critical inflection point for the broader both tech and policy community to start thinking about responses to these issues to build broader resilience throughout the system. Um, and these are the type of engagements that that can help spur that momentum on. So, as mentioned, my name is Ian Reynolds. I am the AI policy manager at HuggingFace. And today I will be providing a broad overview of the agentic incident that occurred back in July and also talking about um some of what hugging face has done in response to this particular event but also how we're thinking about digital security in the open model ecosystem more generally speaking and hopefully this will this will offer a nice transition to uh the broader panel discussion uh after my presentation. So, for folks who aren't really aware of what HuggingFace is, uh we uh were founded in 2016 in New York and are kind of a main platform for the open-source community um of AI builders and researchers to fine-tune their own models, upload their own data sets, and build their own tools that can be hosted on our platform. Um, we have over 16 million users on the platform, millions of models, millions of data sets, um, and other AI applications that folks are building on the platform. And this is beyond just large language models, um, and data sets to do with large language models. It has to do with biology, journalism, video, finance. So, pretty much anything you can think of, there's probably something going on on the HuggingFace platform. Um, and also folks are are building things on our app store. So I would encourage you to you know check it out if you're interested in what what is going on in the open source ecosystem. Uh but sort of the to the main uh meat of the presentation I'll start now with kind of a highle overview of the security incident. Um so starting again in in middle of July of this year our system started undergoing a multi-day and actually a 4 and a half day uh attack which was entirely driven by an autonomous AI agent. Again, as mentioned previously, this was a combination of open AI models that that were not yet released. Um, and essentially our assessment indicated and it has since been disclosed that uh this threat originated from two open AI frontier agents that broke out of their own evaluation sandbox via uh via zero day flaw in that sandbox. And essentially, I would encourage everyone here to go watch OpenAI's um black hat presentation on this. Um, it was quite interesting to see the elements of agentic coordination that happened between agents prior to breaking out of sandbox and kind of the way that they helped cooperate to to break out of the testing environment. Um, so I would encourage you all to go to go watch that. Um but but essentially after breaking out of this sandbox it used a thirdparty launchpad which uh through a cost a compromised customer account it accessed our own systems via a a comp a compromised data pipeline and essentially it used data set dead drops. So it put it uploaded a data set and used that same data set as kind of an exfiltration tool. So quite an interesting kind of profile of attack and we experienced a high volume of efforts to gain entry to our own systems during this incident. Um and in terms of motivation, this agent operated to find answers to a cyber related benchmark uh known as exploit gym. And it was super hyperfocused on just obtaining these these basically benchmark results and cheating on its own exam. Um so in terms of the incident impact, uh one thing to point out is no customer data was compromised. No data sets, models or other user records were read or modified as part of this. Um, but it was able to successfully access the benchmark information. So, the agent did attain essentially its desired response, but again, it was hyperfocused just on obtaining those answers. It didn't do really anything that another that a human attacker would do, such as doing something for financial gain or or receiving someone's private information. It just wanted those benchmark results. Um, and as you can see on screen, this is sort of a visual representation of the attack starting in the green, which is the the breaking out of the the sandbox. Um, establishing its own sort of command and control via this third party platform and then accessing hugging face systems through this kind of repeated uh, agentic um, machine speed attack. Um I think one core thing to take away from this is the type of behavioral profile of the attack that that we that we experienced um in terms of building broader systemic resilience and how other organizations can sort of recognize and detect this type of attack moving forward. Um so again this was this basically came from the motivation of trying to cheat on a test. It was hyperfocused on successfully cheating on that test and had determined that it was easier to obtain the benchmark results from the evaluation from our own internal systems rather than complete this this evaluation on its own. And it pursued this quite persistently. It also operated at machine speed. So over this again multi-day uh event, it executed over 17,000 specific actions. So it's way more than a human attacker could do on its own. Um, and many of these actions were actually pretty sophisticated. It was able to forge credentials, for example, bypass restrictions, establish this covert command and control, and move laterally within our own systems. And again, some of the agentic coordination elements as part of the breaking out of OpenAI sandbox were some of the most interesting kind of things that I'll point people to as well. But also throughout this general sophistication, it mixed these sophisticated behaviors with thousands of failed probes and repeated techniques that a human actor simply wouldn't do. So it was this odd combination of high sophistication while at the same time doing pretty basic kind of brute force things that just weren't working. But it was also able to adapt continuously throughout the process of the incident, switching tactics whenever, you know, previous things were blocked. Um, for example, bypassing domain limits. Um, it also took evasion evasive actions within our own system to try try and hide its tracks. Um, such as building a new ID for every sandbox run. The kind of weird thing behaviorally about this is that it led to the agent rebuilding its foothold for each time. So basically would start over every time and then continue to the end. So something again that a human attacker would would never do. and re recognizing some of these behavioral elements I think is key in terms of other organizations uh shifting their own scanning capabilities and what their security teams are looking out for moving into the future. Um so for our own incident response here we were we were able to once aware rapidly contain what was going on. Um and the security team did really kind of great overtime work uh acting quickly to to respond to this and mitigate this. Um, essentially we were initially made aware by live runtime analysis and scanning that basically send an alert to our own security team and they were able to then start the response. One interesting thing that we would point to in terms of our own experience here was that we initially tried to use proprietary models to conduct security analysis um and help go through the security logs. The security team was then locked out because of specific guardrails that were in those proprietary systems. So essentially you can see on the bottom of the screen here uh this is the error that we that our security team received when trying to use a proprietary model. Um so this is sort of one thing to point to as we think through how security teams can can best prepare for uh a cyber ecosystem in which there are proprietary and nonproprietary models out there. So instead we used an on-remise uh open model um and it relied on this self-hosted open model which was a quantized version of GLM 5.2. For those who don't know what quantization is, it's basically making this model smaller so it can operate and be hosted on your own systems um to dissect attacker logs. One sort of secondary um benefit of this is that it kept all incident data internal. So it wasn't going to an external system and and sort of exposing that to other folks, excuse me. And then in terms of remediation, uh, our team was able to evict the agent, uh, remediate any vulnerabilities and then we end then update our own scanning processes to recognize some of those behavioral elements that I pointed to before. Um, and we have really worked well with with the Open AI security team on this and have collaborated effectively, I think. Um, so that's that's one thing to to keep in mind as well. So three or sorry four kind of broad takeaways that I'll point to before some kind of broader elements that I think we need to take away in terms of this security incident moving forward is first open models can be defensive assets. Our own team relied on these capabilities and if we didn't have access to these capabilities would we would have been at a severe asymmetric disadvantage. it would not have been possible to analyze those 17,000 plus logs uh and kind of make sense of what happened and respond in the way that we did as quickly as possible. Um, broadly speaking, we're an organization that has relatively high amount of resources, right? We can host well, albeit it was quantized, we can host a large model on our own servers um locally, not all organizations can do that. So we have to think about ways to build broader resilience and give access to to similar tools to organizations that don't have those same capabilities whether it's I don't know uh power plants or water plants in in the rural in rural United States. So uh that that's kind of one main takeaway that that that I think our organization would point to. Second um as a representative noted robust agentic oversight and containment processes are going to be key here. Um and this goes beyond just technical fixes, right? This goes to human processes. It goes to governance and the broader systems that these technologies are integrated within. Um because and I I think as as the open AI presentation points to um a lot of this could have been stopped before the outbreak occurred just with different processes and different governance. So that's something that I think broadly as a community we need to to work towards um and make sure everyone is on the same page in that in that sense. Um third and this is maybe a a kind of good good outcome from the event uh is that AI attacks are not unstoppable. Um you know our security team once we realized what was happening it was quite clear that this was an agentic attack that this was not a human and we were able to spin up a response quite quickly and then remediate that response. So, um, this is something that I don't think we need to to kind of fearonger over, but we need to treat with in in a sober way and and build the the proper resilience throughout the ecosystem to respond to to this sort of attack. And then fourth, which is something I I'll point to right at the end of my broader talk today, is uh standardized disclosure of events like this are critical. Um, we were kind of building this uh disclosure plane as as we were flying it. there there was this whole there's this whole CVE reporting ecosystem for cyber incidents which is great um and that's awesome but it doesn't really apply to uh a scenario like this in which agentic misalignment is kind of the broader cause so we need better standardized disclosure and transparency requirements in this space to again help broad build broader ecosystem resilience and you see at the bottom of the screen screen here this is a direct screenshot from from open AAI's talk um at black and they they kind of come to to some similar conclusions here particularly on experimenting with open and closed models. Um and then building the end state which which in which uh these tools actually provide defensive advantages over attacking advantages. I think that's what we all want in terms of an end goal. Um and and just to kind of close this out with a couple slides about how we're seeing the digital ecosystem broadly um at HuggingFace and and maybe this can help transition into what some of the folks on the panel are going to talk about um is that first narrow and closed approaches to to cyber response can risk stratification. Not everyone can gain access to to the highest quality frontier systems to constantly scan their own co code and and detect vulnerabilities. Um, so we need to think about a way to build access to and and sort of accelerate defensive research in this space to give organizations that don't have those resources the capacity to respond and make their systems more more uh resilient. Um, second, this is more than about models alone. Frequently the conversation focuses on like this model did X Y or Z. I think increasingly what we're seeing is also uh multimodel architectures and the systems and harnesses themselves are equally as important in terms of um what capacity particular systems have. So for example on the the top of the screen uh this is from Microsoft's Mdash blog which Mdash is a multimodel architecture combining smaller and larger models and it outperforms uh single models on their own on on um the cyber gym benchmark which is highly associated with the exploit gym benchmark which is the kind of thing that caused this in the first place. Uh the second is an analysis conducted by a security firm called SEGRP um which they demonstrate that model performance using their specific harness which is essentially a software enabler of of a model um accelerates that their specific model performance as well. So it's not just about the model alone, it's about this broader ecosystem that the model sits in. And then again costs can be prohibitive here. So we need to think about how do we also build and develop locally hostable models so that organizations have defensive access. And this kind of again points to a range of openness and cyber security across the digital ecosystem ranging from models to harnesses to building better data sets for for kind of training models but also um you know what an attack like the one we experienced looks like so other people can be aware of how to update their own systems. So this is a broad ecosystem. It's not just sort of a model level problem. Um, and there are sort of two prominent threat concerns that we see moving forward from from our side. The the one is what we've talked about in terms of this specific event, which is the model and harness capability. Um, of course, an agent breaking out of containment and then having the capacity to break in our own systems. um think about if this was sort of a malicious actor that actually did want to obtain not just evaluation results but somebody's financial records. Uh so that is sort of one vector of threat moving forward. And then second and this this comes to kind of close to our heart as a as a platform that hosts open models um is model weight security. So, how do we make sure that when folks are downloading open models or open weight models, they're not downloading anything that could cause their their own systems digital harm? Um, and broadly speaking in the open ecosystem, there's a lot of folks working on this and I think both folks who are working in the open source ecosystem and in the proprietary ecosystem can really work together to coordinate here. So, this is not like a open versus closed debate, I don't think. Um but but on Hugging Face, we're hosting what we call a collection of a variety of models um ranging from sort of old school BERT classifiers to fine-tune small models such as uh Cisco's foundation series um which are locally hostable and sort of fine-tuned for cyber defensive purposes. Um there's also the recently announced Open Secure AI Alliance which is includes as you can see a bunch of of named companies um that folks will be well aware of and all of those people are kind of working in coordination to think about how we build a more resilient cyber ecosystem. One thing I'll point to uh maybe going back to my my old international relations days are are folks in China are equally worried about this. They don't want their digital systems to be hacked as well. That's in no one's interest. So, uh, GLM recently released, uh, an app or what we call space on hugging face, which essentially gave folks the capacity to upload, uh, open source software repos for scanning by GLM 5.3. Um, so this was to find and then hopefully, uh, remediate any vulnerabilities within those systems. So there's a lot going on in this space, long story short. And then I pointed to the importance of standardizing flaw and incident reporting. Um, again, we were kind of acting as best as we could without any real rules of the road. Um, and I think across the community, it would be very helpful and you know, whether it's folks in Congress, folks in the executive branch, um, folks in here to think about how we properly standardize what this reporting ecosystem looks like. We've done work in this space via a project called Flare AI, which is FL reporting in AI. Um so we have I I encourage people to look at that research, excuse me. And then um there's a bill running through Congress right now which is kind of associated in this space as well. So these are some things that I think coming from the event coming from the incident we can think about um building a more resilient ecosystem. So in closing uh maybe three key takeaways. We need to promote open and collaborative defense across the ecosystem. limited access to defensive tools isn't going to help anyone. Um, so how do we enable people to to better gain access to defensive tools will be super important moving forward. Second, again pointing to what the representative said, we need to mitigate emerging threat vectors, whether this is agentic tools breaking out of their own sandboxes or broadly building better and more resilient systems for thinking about how we do agentic oversight and thinking about if fully autonomous agents are even desirable in the first place. Um, so these are some core questions that remain unanswered that as a community we need to kind of work forward, work, work towards and and and better come together on. And then third, finally, standardized flaw and reporting and response will be fundamental moving forward. Um, and I think these are kind of some they're not not lowhanging fruit because nothing is low hanging fruit these days, but um, some things some real concrete actions that the broader community can take uh, coming from from our learned experience at HuggingFace. So again, thank you so much for the opportunity to uh to speak with you all today. I'm really looking forward to the panel and um the conversation moving forward. And on that note, I will open my phone to see if there are any questions. There are okay so the first question here um you were locked out of proprietary models and used open weight for defense. That's excuse me obviously problem for defenders but should the rest restrictive classifiers and prop proprietary models be reduced or removed or should the goal be to focus on model alignment to avoid future incidents how do you see these trade-offs so yeah that's a that's a great initial question from my perspective again I think this this points to the need for organizations to experiment with both open and closed models as part of sort of a multimodel architecture um it's going to be extremely expensive expensive for for organizations to constantly keep frontier models sort of uh always on that's that's not going to be possible for for many folks. So building and routing different models in a in a multimodal architecture is probably the best way moving forward. And so I don't think that that completely removes like take all guard rails off models but I think it's it's working in coordination to build sort of a more responsible defensive ecosystem. And a lot of this will take experimentation. Um but from from simply our perspective I think it points to to the importance of having that controllability and the capacity to sort of host your own model on your own system for its own customizability and then kind of using that for your own response. So it it's it's going to be an ecosystem um it's going to be an ecosystem response that we need. Uh okay, another question here. Assuming openweight models are only months away from methos level cyber capabilities, why should governments allow widespread access to these models on platforms like hugging face? Why shouldn't they be arms controlled? Um so very, you know, obviously an important question for us. I think um many in the open source community are actively thinking about this. Uh, for example, thinking machines has released a recent um kind of long- form blog post about what responsibleness looks like in the open way community. Um, whether it's staged releases or, you know, giving better giving early access to folks for models that have particularly good cyber capabilities. Um, I think that's quite reasonable. I think folks, as I mentioned, in China are doing the same thing. um the the GLM post that I that I pointed to earlier is basically the exact same thing that the that the thinking machine blog was. So there's a lot of alignment there and and I don't think the answer is um you know banning open source in in any way shape or form. I think security through obscurity has has shown in the past in in the digital ecosystem to not be sort of the broad best way to move forward. Um and it can really limit access like like in our context. So, um, it's about building that sort of broader resilience within the community and and working towards standards for responsible release of open models while also giving folks this this defensive access. Um, and I think those were the two questions. Oh, no, there was another one. Um, you now have perhaps the most valuable alignment environment logs from this incident. Any plans to open source or trusted partner release? Um, great question. uh we uh are are doing our best to be as sort of I don't want to say uh pro disclosure as possible but this is sort of very difficult because the logs contain sensitive information on our side. So um we're doing our best to be as responsible as possible and and work with partners in federal government uh moving forward. So we'll see what happens on that front but but no no specifics uh exactly. Um, sorry, I'm I'm managing uh Q&A emails, so uh I think that's it. I think there was only three, right? Yes. Okay, I'm getting thumbs up. Okay, so with that, I will I will stop. I will pass it back over to folks and I just thank everyone here for your time and attention and and encourage everyone to keep thinking about these things as uh you know, we all have something to contribute in this space. So, thanks so much. Uh thanks for those comments um Ian they were really helpful um so one of the biggest challenges when working in the AI policy space is that um is just how rapidly everything is progressing. So time and time again, we've seen new developments in AI that have sort of overwhelmed the institutions we have in place to deal with them. That's true for government. That's true for the infrastructure even at the frontier labs who are developing this technology themselves. Um these containment incidents are no different. um they raise uh questions both about the security practices that the frontier labs have in place um as well as the uh sort of regulatory laws and authorities that we have that the government has in place to shape the development of AI. So for example, why were these incidents only discovered days after they happened? Uh why weren't labs engaged in more continuous monitoring of the actions of their agents? Um, shouldn't there be more affirmative requirements on labs to share incident information? To discuss what governments and labs should do next, I'm pleased to welcome to the stage three experts on these issues. Helen Toner is executive director of CEST or the center for um security and emerging technology. McKenzie Arnold is managing director of US policy at law AI. And Matt Pearl is director of the strategic tech technologies program uh here at CSIS which is one of our sisters program sister programs that thinks a lot about the strategic implications of techn of various technologies including quantum and cyber. As with the previous session um we'll be taking questions uh we'll be leaving some time for questions. So you can scan the QR code um on the screen for those in the audience or use the links in the event description or the YouTube page uh for those who are joining us virtually. So thank you for joining me today uh Helen McKenzie and Matt and I'm looking forward to the conversation. Sorry. Um, so Helen, I wanted to start with you. Um, so we just heard a really good technical description of what happened with the hugging face open AI breach, and I think that was really informative. Um, since that happened, we've heard about other incidents, most notably incidents reported by Meta and Anthropic, um, that involved sort of an external vendor. So I I wanted to start with you and say, you know, what should we note about what's different about these incidents and what does that tell us about um sort of this this issue more broadly? >> Yeah, and thanks for having me. It's great to be here with everyone. There's a lot going on here, so I will try and unpack the different things we've learned in a relatively concise way. Um maybe just a uh uh before I start to say I think something we'll be talking about on this panel which differs a little bit from Ian's presentation that was a fantastic explanation of from the cyber security side from the perspective of the victim of the cyber attack kind of what are the cyber security considerations here I think there's several other angles of these incidents as well and a really important one is what does this tell us about how AI is developing and what is happening kind of at the frontiers of AI progress um which is a little bit separate from what do you do if you are you know the victim of of one of these attacks. So but to your question of what are the other incidents that have come out maybe I'll go chronologically and I think there's sort of three pieces that get increasingly concerning. So the first being some of the anthropic and meta incidents that we learned about then um the UK government releasing some information and then uh OpenAI releasing more information about what happened with hugging face. So, first we did learn about uh it turned out that after um the OpenAI hugging face attack came came out. Uh Anthropic went back and looked at some of their past testing records um they looked at more than 100,000 tests that had been run. So that you know the scale that this is happening at that they're they're the scale that these companies are operating at is really worth knowing about. The number of tests they're running, the number of reinforcement learning kind of training runs that they're doing. They looked at more than 100,000 runs that they haven't looked at that closely before and found that they had actually also had their their systems hack into other companies that wasn't supposed to happen. Um, in this case it turned out it was less that the AI had kind of actively broken out of anthropic and more that they had set up the test wrong and so the model had access to the open internet even though it wasn't supposed to. Came out later that the same thing had happened at OpenAI with the same vendor this one provider called a regular. Um, and the same thing had happened at Meta. So these were these were incidents where sort of sloppy setup um of these testing environments had given the models more access than they were supposed to have and they went out and you know maybe did felonies who knows if it counts as a felony if it was um an AI behind it. So that was sort of the first piece we learned about. Then there was a really interesting um report that came out of the UK AI security institute an incident report um which similarly I think they were watching closely for this stuff after the open AI piece but this report really stands out because they're not a company they're an independent government organization and so they very quickly and very thoroughly released a lot of information about what happened and they described kind of multiple incidents of different models doing undesirable things in testing. Um the most notable in my mind was this anthropic um uh I think it was mythos. I forget if it was mythos or fable, but one of the most advanced uh anthropic models um being given a cyber security test and deciding that the best way to carry out that cyber security test was to run a social manipulation campaign on real people, you know, do who are real open source software maintainers. Um and so it tried to get malicious code accepted into this open source software. It um set up multiple accounts, you know, called sock puppet sock puppet accounts. So, it had one account that was trying to suggest the changes, another account that was coming in and saying, "Wow, these changes look so great." Um it had email accounts that it was emailing the people involved. So, it was a sort of pretty involved um attempt to deceive humans. Um and this was a model that had in principle had gone through uh so-called alignment training. So, it was supposed to know that it wasn't supposed to deceive humans. It was supposed to understand these things about its role. That was sort of part two, the what we heard from the the UK government. And then part three, which I really think is the craziest, which Ian gestured towards in his conversation in in his presentation as well, is what we've learned from OpenAI about what was happening in the leadup to um the Hugging Face attack, which is this wasn't just one rogue agent that just decided sort of out of the blue to go and attack another company. But it turned out that actually for two months before this happened, they had been having these systematic failures of security and control of their own AI agents. and their own AI agents had been had found ways to leave notes for each other inside OpenAI's infrastructure um and give each other tips on how to get out onto the open internet even though they weren't supposed to be. Um they were they were kind of writing me. It's like pretty crazy. I really also, you know, second Ian's recommendation to go watch the the hugging face attack. Okay, trying to wrap this up. What does this tell us? I think one thing it clearly tells us is the security and control practices of these companies right now are very lax and are not at all sufficient. I think sometimes people, some of the commentary I've seen has opposed, you know, well, was it really that the AI was so advanced or was it just that the company was careless? And to me, it seems obvious you can have both. Like the AI is getting more advanced and therefore it's more important that the companies not be careless. Um, so that is one important dynamic. I think another thing we're seeing from all of these different incidents is we're starting to see in practice what has previously been a theoretical thing, which is as we make AI that can be more and more useful that can do more and more complicated things, it's going to learn these unintended strategies to do that that we didn't we didn't want it to. So strategies like breaking out of containment, strategies like deceiving people are pretty useful for a lot of different goals you might give it. Um, and I think the third, you know, a really big takeaway from these for me, which I I guess we'll get to a little more later as well, is these companies are making pretty risky decisions and, you know, carrying out risky research programs internally inside their own walls. And so if our approach to oversight regulation, um, is only focused on what do they release to the public, we're going to actually be missing a huge piece of the puzzle. So this gets called kind of internal deployment, um, I would think of it, yeah, sort of dangerous research inside AI companies. We need to have much more visibility into that, much more ability to understand what they're doing, how they're making decisions, um how that's going to affect the rest of us. >> Uh thanks, that's really helpful. For those of you who are interested in learning more about the OpenAI incident, there was a presentation by a couple of OpenAI researchers at Black Hat, um which is a which is a sort of hacking conference. It's on YouTube. And so they go into more detail about this including some details about how the agents essentially created a message board to talk to each other uh over time. Um so um McKenzie um um Helen's comments sort of segue into what I wanted to ask you about which is that after these um incidents you did some scholarship about the limitations of our current incident reporting regimes. Um I think in the US that that's happening part um uh primarily at the state level right now. Um, so I'd love if you could walk us through the argument about sort of what you found in terms of the limitations of these laws, what they're doing wrong, what they're doing right, and then where we might go in the future. >> Yeah, every once in a while in policy, you end up with this very large gap between what policy makers and the public think the law does and what it actually does. And this is definitely one of those cases. So, after you heard everything that Helen just said, right, I think the average person's expectation is, well, obviously an event like this would qualify for incident reporting, right? And then many policy makers would think and you'd be able to ask some follow-up questions, right? Because of all these important points around how much is this about internal safety procedures versus what does this say about the model's capabilities? Was this a weak sandbox? Was it highly incentivized for this to happen? What what constraints and guard rails were removed? Right? all all these things matter and so you'd expect some amount of followup, right? And then you'd also maybe expect that some of that information could ultimately be revealed to the public. I think one of the big things we've seen in the last couple of weeks is that we've actually learned a lot from seeing people argue online about what exactly to make out of all these events, right? And we're smarter because of that. Um, unfortunately, the state of law is not there. Um, so these events likely don't qualify under the state laws that many of you have heard about, SP53, RAIDs, SP 315 in Illinois. Um, I can go into that in a second. Even if they did qualify though, and and maybe this is even more important. What the states would be entitled to is a plain language summary of what happened and the date of the event. Uh, that doesn't sound super satisfying for answering all of those nuance questions. And then most of them just have a general confidentiality provision that doesn't allow for public sharing on these things. Um, now going back to the the first part, right? Do these events qualify? This is the part where if anyone wants to read, you can go see my lawfare piece. Goes into a lot of detail on on the nitty-gritty law here. But the basic version of this is that when those laws were written, you would think that the the qualification for whether you report information might might be low initially, right? you don't actually know what's happening with the events and you're trying to get this information to figure out whether it is important, right? Instead, these laws are written with these very very high thresholds for what can actually be reported. So there's there's four primary ways you can qualify under these state laws. Three of them require bodily injury or death, right? And when we're talking about cyber incidents, there will be a lot of incidents that don't involve those things that you certainly want info on. And then the last category which might apply here it's like a whole paragraph of you know four different conditions that have to be satisfied but the important part is that you would have to show a material increase in catastrophic risk and you would have to show that this behavior was deception that was aimed by the model at the developer and if you don't satisfy those conditions the information can't be reported. So I I think the best read of this is that it doesn't qualify. you could maybe come up with a plausible case that it does, but you should really like sit back and think is that the standard that I want just to get a basic bit of information in the first place. And I think the answer is no. And so to sort of sum this up and think what is the direction going forward, there's an element around making sure that the initial category of what's being reported is broad enough to capture things that are concerning and interesting but haven't yet caused harm to someone. there's an element around oh you actually need investigatory powers or some amount of rulemaking so that you can actually figure out the details after the fact and then you have to have some system of figuring out what parts of that are going to be revealed to the public and what won't and obviously there's not oneizefits-all answer here I I think the answer is going to look like some sort of tiered approach where you provide initial notifications and then depending on how concerning it is you can follow up to different degrees but that's not where we're at right now >> uh Matt I wanted uh uh turn to you next. I think you know as we heard from Ian's presentation there is this real issue that that uh the US government is grappling with which is how to manage the dual use capabilities of these really powerful cyber models. So right now the sort of uh solution we've arrived at um sort of cobbled together is this idea that labs will voluntarily release their most powerful models with the cyber safeguards removed only to select partners and they're they're they're they're going through the process of you know figuring out who those partners are and working with them and that leaves out companies who might need their services like hugging face. Um so so my question is what what can the US government do to sort of facilitate more accessibility to these tools while still managing some of the risks if um these tools get into the wrong hands? >> Yeah. So I do think the role of of the US government is essential here. Um I think that it starts with having a systematic approach instead of a sort of casebycase ad hoc approach. Um and it starts with having at the highest levels convening all of the relevant players. Um, of course that involves the frontier labs, large companies, but it also needs to include open-source maintainers, small organizations, critical infrastructure and so on so that all of the parts of the ecosystem have input into the process um and then I think you know setting common technical standards um building sort of defensive shared defensive capability um and then making sure that access is is broad-based um you know I think that um one of The one of the um lessons of hugging face is that there there are sort of two aspects of it. one is who should be verified as having access and then it's sort of what does that access get you right and I think one of the lessons is that that initial access needs to be fairly broad-based right um it needs to involve a lot of folks a lot of smaller organizations who who are trusted and have been vetted um it it also needs to include a lot of allies and partners and companies that are based in allies and partners I think one of the unfortunate things about the claude mythos preview um situation in the use of export control was was that sent a bad message to a lot of our allies and partners and so how do we make sure that that access is broad-based but then there's a second second question that's distinct of like what does that access actually get you and there I think we need to be much more careful and calibrated but also learning from hugging face also dynamic and rapid in how we respond and have the ability to make adjustments um in terms of who gets um what types of access depending on that. So I do think that the federal government in terms of establishing sort of a a trusted defender pathway um and having preclarance for um organizations is is really essential and also working with industry on um how access should be mission scoped and how they are going to rapidly adjust as needed. Um, I think obviously there's the common security baseline that's all part of this that NIST is already working on and Casey already working on. And so that's work that we just need to um see done and and folks need to work with them to to make sure that in terms of um a tech um uh security identification um and so on that it's done well. Um in in terms of access obviously the government, you know, has a role in providing public goods in a way that industry can't, right? And so um things like ind not not only independent testing but um you know having um red team exercises having access by smaller organizations um critical infrastructure and so on having subsidized access to these tools um I think is really essential um uh for the government to do. Um and then I think the last thing on access is that the federal government needs to um ensure that this is all operational before something happens right um that the sort of contacts and and muscle memory is already there that the you know there are sort of standard contractual provisions there emergency contacts people have worked together obviously a lot of that work is going to be somewhat outside the scope of government but I think making sure that it's occurred so that when something happens that you're not you know having lawyers negotiate about how it's going to be solved that you actually just have the operational folks who are solving it. >> Uh that's really helpful. Um Helen I wanted to turn back to you. I think um so one thing to note is that the labs have been responding to these incidents. To me, the thing that's most notable um is that OpenAI's put sort of a temporary pause on reinforcement learning training of of its um uh most powerful models. You've you've you've sort of written or spoken positively about this development. I I think of all the people in this room and online, you probably have the most direct experience sort of conducting oversight of the Frontier Labs. Um I'm really curious, you know, this was a voluntary move um on the part of OpenAI. I'm curious about sort of what you see as the incentives that are mo that that can or or do motivate um labs to be more safe and and uh that can that can motivate them to be more responsible when it comes to issues like the the issues we're here to talk about. >> Yeah. So maybe to just briefly recap what happened here. Basically in the wake of this hugging face attack, um I think a week or two later, a statement came out and I'm sure many folks here in the AI space and have kind of group uh open letter fatigue like the rest of us, but this was a um an open letter that I think really stood out because it was signed by more than 1300 employees of leading AI companies. So it wasn't your typical kind of advocates. It was it was really people inside the industry signing this letter. And what the letter basically says is we inside the AI industry don't feel like we have a brake pedal. We don't feel like we have a way to slow down if we needed to slow down and we think we might need to slow down at some point soon. That's basically what it says. And so they're kind of asking for help from government, from the public and so on in finding a way to to pace um the the frontier of of AI development. And then I think it was a week or two after that that OpenAI put out a kind of a big uh press release and a sort of public announcement that they were pacing their own research. Um and that they were doing this uh not just by kind of delaying the release of a model but actually delaying the development of the model. Um and I think it's easy in DC to miss the significance of that. A lot of people think of these companies as kind of chatbot companies that are selling products but they think of themselves as AGI research companies. they think of themselves as their core uh mission and the core of what they are doing is building more and more capable AI systems. So actually delaying their research roadmap is a bigger sacrifice than delaying the release of a product. And so then you asked about kind of the incentives at play here and what's going on. The the kind of uh baseline state that the the ground level incentive is go fast. Um and that's a commercial incentive. It's a um competitive incentive more broadly. Um, and it drives a lot of the the behavior we see from from all of these companies is needing to get out their next model, needing to show that they're at the frontier so they can recruit the best talent. Um, needing to convince investors that they are going fast enough. Um, needing to not fall behind rivals, whether that's US rivals or uh the prospect of China is always kind of looming in the in the rearview mirror pretty close at this point. So, one incentive is like go fast, go fast as possible. Um, and so then what are the incentives to not just go as fast as possible all the time? Um, I think some coverage of this sort of pacing has has sort of played up the, you know, this is the, you know, the wisdom of the of OpenAI leadership. Um, I may be predictably a little skeptical of that being the primary driver here. Um, I do think there's, you know, you know, credit where it's due, but I also think that they are subject both to, um, pressure from uh, corporate customers. Um, so it's actually it's crazy to me some people commenting on this stuff say that this is all marketing hype and it's supposed to juice valuations. The idea that you would say, "Hey, if you use our models, they might accidentally like create an infestation inside your infrastructure that you'll have to wipe multiple computing clusters and like restart them from scratch the way Hugging Face did." Like that's that's not a selling point. Um, so one incentive is wanting to be able to tell major enterprise customers like, "Yes, you can use our models. Our models are are safe and secure and working well." I think an incentive that is really underestimated again on this coast is pressure from employees. Um and so pressure inside the companies the level of freakout that is happening um among open eye employees also anthropic employees others is is pretty high um and the pressure that they are putting on their own leadership to say hey we have to do something different here um is is pretty significant so I think that is um that is worth taking into consideration as well and of course there's also kind of broader public pressure potential for for government pressure um but I I would say that those two big ones as sort of enterprise customers and employees um are are really major drivers So, let me ask a follow-up question, which is that there's a widespread expectation that um these frontier labs uh will go public in the in the near future as soon as October. Um how does it how does the incentive structure change and how much harder does it um get to manage safety after they become public? >> I think it's a really interesting question. I actually think it could go both ways. I think there's a sort of standard take of like, oh, public companies, they're just maximizing shareholder value. they can't consider any other sort of objectives. Um, but I think there's also a lot of I mean the the corporate governance for public companies is much more fleshed out, much more mature than corporate governance for these weird nonprofit public benefit corporation hybrids with strange board setups. Like that's all very immature. Whereas once you're a public company, there's a lot a lot clearer sort of expectations. So I I'm really not sure um what it will look like. I think one thing that's going to be very interesting to watch is sort of the the risk disclosures that you get from both of these companies. What do they see as risks to their business? Um because at this point they surely should see uh you know further incidents of this kind maybe even more severe incidents as their AI becomes more capable. Um do they put that in their like SEC filings? Probably they should. I guess will they find a reason not to? Will they disclose it and we'll have that in a like, you know, regular corporate finance document? Like it's it's going to be fun. It'll be interesting to um interesting to see. Uh I'm not sure I'd use the word fun, but but okay. Um Matt, I wanted to turn to you. Um so we've heard, right, there's a lot of interest in expanding out the incident reporting regimes following um follow these incidents to capture more things like this. I think one of the challenges there is um is you know the US government has um often been criticized for its lack of technical talent and you know your center has done work on sort of uh cyber workforce issues um including a a recent commission that that you uh concluded. So I'm curious from your perspective um does the government does the US federal government in particular have um the right technical talent to be able to assess incident reports if they came in at a greater volume and if not how can we sort of uh cultivate that talent. >> Yeah. So I mean I think this is something where you know I would push it back against the caricature that the US government doesn't have sophisticated technical talent. It does including in this area absolutely um but we do have real problems in terms of recruitment retention that I think that we need to think through. So what a look was referring to was that we had a CSIS commission on cyber force generation. Um this is essentially trying to address the problem that we have in the United States uh in the US military that um for um in terms of um force generation and building the capabilities and personnel to do cyber operations each of the four or five military services like does that individually right on its own. And it's it's worse than a left-hand right-hand problem, right? You've got even more hands than that. They don't coordinate. They send everything to cyber command. Um and essentially um what happens is that cyber command gets um gets the talent oftentimes incredible talent um but um doesn't doesn't have the ability to to to integrate it um and to and to have the um folks h that have complimentary skills in a systematic way who have been trained and and have the um incentives also to to stay right um because um you know if you're in the Marine Corps for instance um they may be looking at what what they want to retain an infantry tree officer, right? And not necessarily what you need to do for somebody who's on a cyber path. So, I mean, I think that we have the the technical talent in many cases in the US government, but I think both on the civilian side and on the military side, we need to do a better job. Um, you know, the recent EO did call this out in terms of having an infusion of cyber talent. Um, but there really needs to be a lot more support and and thought that's put into that. Um, you know, I do think that it's something that you could make, um, you know, exciting. I think that there are folks who would be willing to do it. Um, one of the things in the Cyber Force Commission report, um, that we did in terms of structuring that effort, for instance, was to have a cyber national guard um, as part of the service um, with a thought that, you know, obviously the cyber national guard in California is going to be like incredible, right? Because people can volunteer part-time um, and and and work. But I think we need to think about that both on the civilian and the military side in order to have the technical talent that we need to do this um and to work with industry. >> Uh McKenzie, I wanted to turn to you. Um so as we heard from the representative, there is uh at least some desire to do something quickly on these issues. Um but I I don't think anyone would say that that Congress is able to execute on that um particularly well. Um so is the sort of resident legal expert on the panel. I'm really curious about your thoughts about what could the the US executive branch in particular do on this issue now and um what steps could it take? Um you know we've seen that that the executive branch has been willing to take quite liberal interpretations of things like export control law for example when they implemented model restrictions on um access to anthropic models by foreign nationals. Um, so I'm curious what you think the executive branch could do and then what might require legislative action. >> Yeah, just to sort of ground things, right? The executive branch in the US for the most part can only do things that are granted the authority by Congress, right? There are some other things that via the constitution. It has some background powers, but for the most part, you have to explicitly say you are authorized to do X. And with modern courts, that authorization often has to be pretty explicit and pretty clear, otherwise courts are going to be skeptical of it. um things that the government can do um sort of maybe fall at two extremes. There's all the soft power stuff that actually matters quite a bit and the executive branch can start right now, right? There's all the they can meet with individual companies and try to encourage them to improve their standards. Um I wouldn't dismiss that even without threat. There's just something to uh many companies want to do better in these various respects and one of the big constraints is not expecting that their competitors will do the same. Uh having the weight of the US government behind this is really helpful in in moving forward some of those standards. Um they can also sort of improve their own capacity as sort of an information processor or you know being able to respond to future events. Maybe this overlaps with some of your your Rex Matt, right? But figuring out it's it's not obvious where is going to be the locus of power around AI and there's going to have to be a lot of work around hiring and figuring out sort of chains of command and who's getting what and who's in charge of what that we can start now even in advance of legislation. Um there's also I guess in somewhere in between soft power and something a bit more constructive. There's everything the federal government can do in terms of contracting procurement uh right creating things in the real world. uh creating uh demand and incentives to sort of differentially accelerate certain types of safety technology. Um I think there's a lot to be done there and I think we'll see more of that soon. Um on the other end of things there is a lot of these sort of hammeresque hard power ways of intervening and this is maybe what you're referencing with sort of interpreting existing powers creatively. Um, one of the big dilemmas there is just that the authorities that are given to the executive branch that can be used in a rather general fashion tend to be made for emergency circumstances, right? And it creates a strong incentive to treat everything like an emergency or to treat everything like something that requires quite quite intense reactions and that sort of incentivizes these very creative reactions. So whether it's export controls or sanctions or uh supply chain risk designations or other things, you have a pretty blunt tool and it incentivizes you to use it bluntly. Um the thing that's missing that you really need congressional action on is anything that requires a sustained relationship with industry. Anything that requires rulemaking and regulation, anything where you're trying to make trade-offs over careful balances of what sort of safety practices are best and when when they should when should they be implemented and what are the penalties for doing so all those things update all the time. They require quite a bit of predictability. They require quite a bit of expertise. You can't do that without an authorization by Congress and allocations of funds from Congress. And so I think that's really where things are going to go. You're going to have to fill in that middle category. And in the meantime, there are some very productive things that the executive branch can do in terms of its soft power. Um, so before we turn to questions, I wanted to I was there was one last question I wanted to ask everyone. Matt, I can start with you and sort of go down the line. And that is that um I'm really interested in sort of uh one or two concrete policy recommendations you would you would make to address these containment issues. and and I'm I guess I'm agnostic on whether that's legislative or executive branch or state or federal. Um but just curious what you think would be really productive to do in the situation. >> Yeah. So I think um I I think clearly like you know um updating reporting is going to be essential um and not just in a checked box way but in a way that ensures that the right telemetry and records are there um because um you know in these in these situations as came up in the hugging face um uh incident um you know the agents can be really good at evading this and can sort of almost seem to consciously do it right um in a way that you have to have the right um type of telemetry Um, and then I would go back to um actually, you know, from a congressional uh perspective, actually sort of um providing funding um and and and hard dollars for some of these shared defensive capabilities, some of that demand pull that that you were talking about. Um I think that there's there's sort of no um no substitute for that in this case. um particularly where the government can sort of use it toward the uh providing public goods in a way that that industry is um as as great as it is isn't isn't as suited to do. >> Um picking up on what Matt said, right? I think there are a lot of fixes that we can make around incident reporting and that starts with making sure the categories of what is reported are broader and that some of them exist preharm. making sure that you have investigatory powers and rulemaking so you can actually figure out the details of what happened and then sorting out how that information is going to be shared between different governments and with the public. Um I think we're also going to have to think more seriously about monitoring. Um that will be costly at times and it's going to be an unresolved technical question. So, I don't I don't have an immediate recommendation to what for what to implement now. But, I think we're going to want to put ourselves on a policy track where we're starting to figure that out and starting to treat this as a serious part of the puzzle that it isn't just reporting things. It's about making it more likely that you detect them and that you have enough data on what is occurring such that you can make sense of it after. >> Yeah, I think that two I would give. One is fixing the blind spot that we have right now around internal use of AI within these companies. So again, I think we need to shift from a mindset of kind of product safety of making sure new models are safe before they're released widely to instead recognizing that these companies are doing quite risky research internally and they're making they're constantly making risk appetite judgments about do we proceed, how careful do we need to be, how much more information do we need? Um, and so I think there's a range of different ways that you could get more visibility, more oversight um into what that looks like. So you you know you mentioned incident reporting and sort of expanding our ability to know what happens when things go wrong. I think a different thing that would be really really valuable is to have um find some way to make the companies uh share much more information about what how are they making these judgments? What does their research process look like? How are they why the hell was OpenAI not monitoring these deployments that they were uh that were behind the hugging face attack? um how you know there's a lot of sort of decision-making and interpretation of evidence that is going on inside the companies right now that really should be exposed to much more sunlight would also allow for more harmonization but you know inside the industry in what is sort of appropriate practices and would also allow for sort of civil society and external experts to understand what's happening um more to say on that but I think that the sort of broad category of getting that internal risky research out of the darkness and into um into more public light is really important and then another one I think is we have this Trump summit coming up um US China summit at the White House in a few weeks um I think September 24th is the date and on the US side when you talk to people in industry they're constantly bringing up well but we can't do this we can't do that we wish we could but China um we wish we could but then China will beat us and I think if we are actually serious if they're serious about the level of risk that they think um AI poses that you know people inside the companies think what they are building is very risky but they're racing to do it anyway because if they don't China will and if we think that is the situation then that is a conversation we should be able to actually start to have with China and say hey do you think your companies are doing things that are this risky how can we you know you don't necessarily need any kind of deal between the two leaders so much as just an understanding of this is a risky situation you know the US needs to look better and do better in terms of how US industry is handling these risks maybe China needs to do better at how it's handling these kinds of risks um that is a conversation that has sort of gotten off to some very very baby step starts um in the last couple of years and I hope there can be significant progress in the next few weeks and months. >> Yeah, if I could just piggyback on that a little bit. I think that you know um my program does a lot of track to track 1.5 dialogues with the PRC including on cyber and some of these issues and I do think that there is an interest and a willingness on the PRC side to potentially talk about these things. Now I I wouldn't sort of mistake that for guaranteed progress, right? There are all kinds of obstacles. But I do think that if we did have that attention sort of at the highest levels um and that includes the president um in these interactions, I do think that it would be an area that's ripe for making some progress at least potentially. >> Uh so so everyone's mentioned incident reporting in some way. I wanted to ask a follow-up question which has to do with compliance. So there's it's one thing to sort of mandate that you know a certain set of incidents need to be reported but there's this phenomena right like uh the the tree in the forest if it falls and no one um sees it did it really fall if an incident happens um and it it's only internal it doesn't affect any third parties there's there's incentives for companies not to report that um so from a compliance um perspective how do we ensure that we are getting all the relevant information even if we do have mandates ates. This is something that I've been thinking about more in the last couple of weeks and with reference to reporting in other areas like aviation. So, a couple things you can do. One, uh, in many other contexts, the reporting goes to not the regulator, right? To some other entity that does not have rulemaking authority over them because it is less likely that the hammer will come down on you if the information is provided not to the the regulator, right? This has some trade-offs to it, but this is a trade-off that they've made in some other contexts to make it more likely that there are full disclosures. And then in terms of incentives, you kind of have the whole sort of uh decision space. I think one that will come to mind to people very early on is that you can offer liability safe harbors and other things in return for reporting. I think this creates pretty perverse incentives uh right to like include all of the information in your initial disclosure and to protect yourself from things that you should legitimately be responsible for. There are other versions of that where what you instead do is create negative inferences. So if you do not provide the information and it later comes up in litigation that you were aware of this and did not provide it that we will draw an inference that you had in fact you had intent or you had knowledge or satisfied some of the other elements there. Um I think that's more promising. Um, you can also add to that sort of promises that whoever receives the information will not use it directly to pursue allegations themselves. Right. It's it's a really tricky thing, but it has been done in other context and and I think that we should be thinking about it more. >> Yeah. And I think to pick up on that, you you need to separate the two situations which one is where an incident causes actual harm or damage, right? And in that case, we have, you know, all kinds of different areas that the federal government or state governments regulates that you can provide incentives that essentially make it much worse if you don't report in that situation versus where there isn't actual harm, but we just need to know about it, right? Like it's really important in terms of that feedback loop and lessons learned. Um, and that's where, you know, an independent third party, somebody who doesn't have the hammer is so so important. And can I add as well, I think in this industry as well, again with the very activist employees who are really concerned about what their own companies are doing, um, this also gives a hook for whistleblowing. If there's sort of a clear legal obligation to report something and the company doesn't, um, then either you could build that into, you know, into new legislation or perhap you would know better, perhaps there's existing protections for if the company is not complying with a legal obligation. That gives you much more standing as a as a potential whistleblower. >> Uh, that's really helpful. So, we do have audience questions, so I'm gonna I'm gonna turn to those. Um the first question is from McKenzie and it's from Chris Corin. Um and it's basically saying, you know, Helen mentioned in her opening comments that it wasn't clear if a felony was committed relating to the hugging face incident. Um since the acts were carried out by an autonomous agent rather than a human. Um so what's your best understanding of how liability works for crimes committed by agents? Uh if there's any clarity at all. >> Yeah, really easy question. Let let's go through it. Uh maybe catch us in three hours. Um so maybe first separate it, right? There's uh tort liability, civil liability, and other fines. There's criminal liability, right? They're all different categories of things. Um one thing that people have talked about a bunch, including if you want to look up some of Orinancer's comments on on the recent um breaches online, he provides some good commentary there on the CFAA, which is the Computer Fraud and Abuse Act. um whenever you have a criminal statute or legislatively established um liability you often have some sort of intent bar right I think in that case for the relevant provisions it's knowing right so you have to have knowledge that you are going to access something that you don't have the ability to access my colleagues are going to you know be rolling their eyes at my my lack of memory on this um that's really hard when you have an agent right in in this case the agent itself does not have knowledge in the sense that we ascribe to humans. The company itself, if anything, you know, much to to our detriment, was unaware that this was happening. Um, you could argue about whether they have constructive knowledge because they had seen similar events like this perhaps in the past, but I don't think that that's sufficient under the statute. Long way of saying one of the few laws we have sort of that might be relevant here doesn't seem like the elements are satisfied. Then when you go to tort liability, this goes back to Matt's question of you actually need a harm, right? The way that you have tort liability is that you are compensating for damages that actually happened. Um that requires that you actually did have injury and that you also have a plaintiff who is willing to bring a case. Um I won't put words in anyone's mouth, but in this case it it's you know public that Hugging Face uh has not initiated litigation um against OpenAI. There could be good reasons for doing that, good reasons against doing it. Um, but it means that we won't actually get a case, we won't get discovery, we won't learn more through, uh, the liability process there. So, I think in a case like this, um, you're not going to have liability. >> Okay. Uh, I suspect that the audience might have some follow-up questions there, but, um, but, uh, you know, I wasn't expecting you to be able to solve the issue of AI liability in 30 seconds. Um, the next question is from Will Toenheim of Nextpillar Capital. um and and is this is for any of you to answer. Um and this is a question basically about US China competition. So there we've already seen because of some new regulatory sort of mechanisms that there have been delays in release of US models and the question is is basically a concern that um models released in foreign countries like China aren't subjected to the same kinds of requirements. So what can we do to implement sort of necessary reporting and cyber security requirements um for US labs without necessarily creating a a problematic bottleneck uh in terms of competition from China? I would I mean I would say here that I would I actually think the right approach is a different one which is the concerns that have motivated uh or I guess there's different different pieces here but to the extent that we are uh intervening in the US AI industry due to concerns that would also concern China I think the the right approach is not to try and just skip that and just let things proceed even though we have significant concerns about risks the companies are talking about themselves but instead is to try and build more of a share understanding with the Chinese government of this is in fact a risk that should concern you as well. Um I think for a bunch of reasons there's a much more sort of wellestablished and influential uh community inside the US AI industry that thinks seriously about kind of major risks from highly advanced AI systems. Um and that is starting to be more that is starting to develop in China but it's sort of starting from a lower base. Um so I think things like uh the kinds of dialogue that are planned between the leaders of the two countries to say hey here is what we are seeing here's why here's why we we are concerned and not expecting China to do anything out of love for the US that's obviously ridiculous but expecting China to recognize its own self-interest um and the reasons that the Chinese Communist Party probably doesn't want uh many of the same kinds of risks to be event to eventuate um as as the US government does. Um, I think there's that's that's really when you're talking about the sort of major catastrophic um and especially, you know, risks where you have a very advanced AI as a threat actor. I think that's the approach. There's lots of other sort of more pragmatic prosaic uh potential reasons that you might want to regulate the sector where it definitely does make sense to be looking for um more streamlined, you know, uh approaches that that don't create friction. >> Yeah, I I would start by um pushing back on the premise of the question, which is that I I think the PRC is going to do this. I think all the signs are that they're seriously considering it talking to their um frontier labs about it. Um >> and they, you know, they have a strong record of regulating their sector. They basically just like crushed their AI companion sector domestically because they were concerned about risks there. >> So I do think that that makes it an area that's ripe for having an agreement. I mean, in terms of um industry's concerns about it, I think those are legitimate, too. I think that um you know, that that EO that laid out sort of a voluntary approach um what was that a couple months ago now? um you know initially it had 90 days was you know going to be the review window. Industry had a lot of concerns about that and I'm somewhat sympathetic like that's an eternity right in this industry. And so I think it comes back to that conversation we were having about really ensuring and having the administration make those investments and send the right signals about having the technical talent in the federal government that can review things very quickly um so that we don't have you know 60 or 90day windows that again I I I understand why industry doesn't want to have to wait that long. >> U McKenzie anything you you want to add? You don't have to. >> Plenty of smart comments already. All right. The next question is from Mikey Herriagan uh of MITER. Um and it's a question basically about sort of investigations and and sort of uh regulatory sort of issues we've seen in in the way that investigations have been implemented in federal government before. So basically um are you concerned that government investigations into AI safety incidents u might fall victim to some of the pitfalls we've seen in other federal investigations? um mentions politiciz politicization or the perception thereof of investigations into particular companies or accidents. I think this is probably heightened because we've seen the administration sort of take specific actions against specific companies. But I I'll sort of add my own uh take which is there there's also the issue of sort of regulatory capture. Um so I'm I'm curious about um both of those issues as you think about sort of the right mechanism for both reporting of information and investigating incidents. >> Yeah, absolutely. And this is a real concern. Um I think also at the same time it can be easy to overestimate how often this is to happen based off of the last couple of years, right? I think if you asked this question five years ago or something, people would have thought the trade was somewhat different. Um, I think that at least uh not to say that we will gravitate back towards the mean, but it's also to say most executive branch powers can be abused in various ways. You can try to make it more difficult or more costly to abuse them. But this is in fact just a trade-off that is inherent to any amount of investigation, licensing, review, rulemaking, etc. Um, in terms of preventing that abuse though, I think what we're going to end up with is some sort of t tiered system, right? where for most incidents especially where you don't have harm what you're only getting is some initial notification and then probably some amount of voluntary back and forth of information right it's only as you sort of escalate up that up that chain where I think we'll need investigations and I think there if you just think concretely about what if you had an incident where there actually was harm right I think we'd say oh we definitely need investigative powers right I think this in in some ways it will resolve itself you'll have specific compelling incidents that require that >> uh anything uh either of you want to add? >> I think there's also there's also a range of ways that this kind of thing can work like can be design that this kind of investigation can be designed and some of the maybe most salient examples of investigations are the very high level political ones but there's a lot of industries that have just ongoing kind of safety incidents, investigations, learnings, best practices um that are happening more at a technical level. So here, you know, one place my mind goes is there's been a recent discussion kind of initiated by Demisabus of Google of could there be some kind of government supervised self-regulatory organization for frontier development. Um there you could imagine a pretty in-depth investigation that really wouldn't be sort of being run out of very high level political organization or or parts of government um but would be uh kind of handled by these technical experts at the technical level. So, and I think of you know um aviation incidents as an example of a place where this has worked quite well and there's a lot of existing practices about how do you um learn as much as possible from any given incident um including both ones where harm were c was caused as well as as well as near misses. >> Yeah, I'll just pick up on that because it's a really good point from Helen, right? This is one of the ways that we've handled these these risks of abuse in the past is that who you put the responsibility in the hands of makes a big difference. Right? Is that person very are they politically appointed? Are they easily removed? um do they have a technical background? Do they see themselves as an investigator or a like sort of a serious technical person who's trying to figure the question out or do they see themselves as someone who is advancing more ideological or policy based priorities? Um and you can do that within government by placing these responsibilities within say you know you could find plenty of people within the NSA, the DOE um within part parts of of the DOC like uh KC that could do this and who would see themselves primarily as experts who have a technical mandate. Um you could also do this outside of government, right? If you're relying on auditors or other third parties to look into things, this provides some layer of insulation where they're not as directly controlled by the government. I think you almost maybe h I mean this I haven't I haven't actually written a piece like this and I don't know if I've fully concluded you almost it almost needs to be partly outside of government given that we don't have independent agencies anymore and that's a call that the Supreme Court made right um and so there are all kinds of questions with a FINRO type entity about how you structure it how do you avoid industry capture there's a whole other set of problems but at least the independence um aspect of it could be something um that would be helpful >> yeah I mean my view has always been that like comparing how to manage AI to like a single example of how we've done in the past is too simplistic and we we need to be more sophisticated mix and match various elements to make something that's uniquely suited for AI. Um the next question is also from from the same person Mikey Herrian um primarily for Helen but of course anyone is welcome to join in. Um it it is um Frontier Lab employees seem to have outsized influence over addressing ethical concerns in AI. Um how can we or should we strengthen or codify this? And do you have any concerns about lab employees playing sort of a deacto regulatory role? >> Um I mean concerns about lab employees playing this role. I I think it's just the obvious one of it's it's a very small group of people in a very specific culture that doesn't uh you know and if that is the only set of people who are kind of providing any kind of check or oversight here then that's leaving a large number of stakeholders out. Um in terms of ways to um strengthen the effect I think the biggest one would be looking at whistleblower protections which I know McKenzie you and Lawi have done some work on. Um the sort of basic uh problem statement here is there are a lot of existing whistleblower protections in law. They are generally designed for if illegal conduct has happened. Um and right now because there is so little regulation of uh advanced AI development, frontier AI development. Um, if you're I think a situation that a lot of company employees feel that they are are in or or might in the future feel that they're in is, hey, my company is making very very risky bets, is making decisions that are not based on strong evidence about this being a, you know, a safe decision or or they're kind of plowing ahead recklessly. Um, but they're not actually doing anything illegal. And so I, as the the company employee, don't really have recourse to go. I don't know who I would tell. I don't think I would have protection for breaking, you know, a non-disclosure agreement. um just because the uh you know I in my technical judgment as a company employee am concerned about what the company is doing um but there's no sort of regulation out there that is being broken. So I think there's been various um proposals. One strong one comes from um Senator Chuck Grassley on how you could uh create some whistleblower protections for um AI employees. Um that would be sort of the first first one that come to mind. I'm curious McKenzie or you know also if there's others. >> Yeah Helen, you're speaking my language. I think the Grassly bill is a good example that includes non-law violations um as something that you can whistleblow on. And you're absolutely right. Right. Most whistleblower regimes just cover violations of law. It's going to take a long time for the law to catch up around AI. So a simpler fix in the short term would be to broaden the extent of the whistleblower protections until the law catches up. >> Uh the next question is um aimed at you uh McKenzie. It's from Savannah Taylor. Um, and it's in reference to your mention of liability safe harbors. Um, do you have any sense of where where the right place to draw the line is? Uh, you mentioned it's tricky, but has it worked in other industries? Are there any examples you could give about how it could apply in the AI industry? >> Yeah. One factor that isn't present here that can make a liability safe harbor more compelling is if you have really clear best practices to implement, right? If you have very obvious things, if you do X, Y, and Z, you mitigate your risk considerably, you actually know what to do and it can be worthwhile to trade liability for that, right? Acknowledging that, right? Some amount of litigation is either frivolous or is just very costly, right? Like genuine disputes over things, but it's going to cost a lot of time and money. If you have things that you know are good, maybe you can make that trade. I don't think that that's likely to happen here in the AI context. And so, in fact, liability is a really good fallback, right? It's a really context dependent inquiry that says given everything that you knew was what you did reasonable and when you don't have clear rules of the road that might actually be the closest thing to a reasonable standard right if if you think that also the mitigations are very technical in nature and that in fact there is a lot of of knowledge in industry on what is best to do then relying on a standard that asks given what they knew and their level of expertise and what the technical state-of-the-art is right all of those factors that factor into liability it it in some ways defers to their expertise and then holds them accountable to that expertise. Um, people have also talked about expanding liability. That's where I think when I said it's complicated, that's the more complicated part in my mind where you have to think about a lot of complicated incentives. Tort reform in a positive sense is is rather rare historically. I think what I'm more clear on is that broad safe harbors are act are more obviously negative in part because right now they're setting good incentives that say this is a catch-all if if the law isn't there and if you remove that even if the companies are not reasoning super rationally or very directly they have to say well yesterday my risk level and how much monetary damages might be come out of our company are here today they're down here like there's some delta right like I I can now my risk tolerance should go up in some some relevant respect and I would expect that it would be a really salient signal if you actually passed it so um much clearer on be cautious around liability safe harbors likely only trade them where you have some obvious best practice um to to implement um and if not leave it be >> okay yeah um we have one last question from the audience and then we'll we'll move to wrap up um and this is from Will Tobenheim of Nextpillar Capital um it's for Matt primarily. So we've seen advances in cyber capabilities. We've also seen advances in just sort of AI code generation. Um so the question is about um the interaction between these two. Does AI generated code tend to be more robust to cyber attacks? Um and is it it a promising solution um handinhand with open models to strengthen defenses? Um so I think the answer is it depends right I I think that there are tools that have already been released and will be sort of you know further developed in terms of things like as someone as a you know a software engineer is writing the code um actually having the AI um build in and and and detect cyber vulnerabilities and bugs and things like that. And so I I think that there's absolutely the potential for that to happen and it's something that we need to incentivize and encourage. Um, but it very much, as I said, depends, right? It it's it's sort of an institutional design and incentive question, um, rather than something that I would characterize as being inevitable. Um, so, uh, I think at this point I' I'd love to, you know, we've talked about a lot of things. I just love to get a sense of if you have any, uh, concluding remarks. I'm particularly interested in maybe if you've changed your mind about policy interventions or if you have thought of new things that we we could be doing as a result of this conversation. Um but feel free to to to conclude in any way you want. Um ma'am maybe I can start with you. Um yeah, I think that um you know, one of the areas that we haven't had as much discussion of, but I think that we need to give a lot more thought to is um the way in which agents, you know, under their current design um are uh framed in terms of and incentivized to only achieve their goal and and is that sort of an inevitable result of the way that um we're going to implement AI or are there ways that we can temper it, right? And I think that that's um a conversation worth having as well as um as came up in the hugging face incident um this question of the way in which agents work together, right? Because that combines with their their sort of desire in some cases to evade controls and to accomplish a goal in a way that they've been told not to. Um and so um so I think that those are we need to have a a sort of further discussion about um how we're going to design models um in order to address some of those challenges. I think >> um if I had one overarching thought it would be that AI policy is largely a question of managing really deep uncertainty um and of course that applies to any industry but particularly here right where we're having foundational questions as to huh does this reveal that the models have some amount of direction to deceive us uh right these these are things that are not presented in other domains um and I think that that has a couple implications for policy. One is that all thisformational stuff that we're talking about, it can sound kind of boring, but I think actually this is the core of of actually figuring things out in the future. Otherwise, we're going to be completely confused. Um, that's both a government capacity thing, that's a gathering and sharing the information thing. It's making sure that people in the public can analyze that information. All this is kind of preparing us to try to make more sensible policy in the future. Um and then secondly I I I guess like a correlary of that was something Helen said earlier about just the importance of internal use visibility or or right no longer treating deployment as some uh you know uh hallowed moment at which everything changes and that's what you're you know that's the deadline you're working with. I think it is going to be much more a matter of uh as the technology becomes more capable there may develop a larger and larger gap between what people know inside of the companies and what policy makers know. Um this is already exists but I'm talking about something much more severe than that. And if you want to be in a position where we're not having rapid ad hoc decisions made due to surprise and concerning things happening in the world, that's probably where you have to start. And there are a lot of trade-offs involved in that. And I I don't mean to to make them sound small, but that's why we need to be thinking about this and trying to build a system where you get some amount of visibility and sort of mitigate all of the the trade-offs or abuse concerns that might come with that. >> Yeah, we didn't coordinate this, but both of those set up perfectly what I wanted to say. So, in terms of how are we designing these increasingly advanced AI systems, what is inevitable, what can we choose, and then sort of the importance of information for making policy here. Um, a a huge thing that's on my mind right now is just we need to know way more about what happened in these different incidents. OpenAI has said that they're going to release more. Um, Hugging Face has been great in terms of releasing lots of information. UK AI security institute released very detailed um, reporting, but OpenAI Anthropic um, really need to share a lot more both about what has specifically happened here and then going forward. um ideally due to you know uh legal requirements to release more information but if not then um at a minimum on a voluntary basis because they are uh doing some very consequential things behind closed doors right now we need to we need to be able to see more. >> Uh well that brings us to the conclusion of our program since brief concluding remarks. I just want to thank um Representative Submanum Ian Helen McKenzie and Matt for their keen insights today. I think what I take away from is that these incidents clearly require sort of an urgent response and that there are things we can do. Chief among them, we heard a lot about enhanced incident reporting, closing the information gap between what the labs know and what governments know. Uh thinking about expanding technical expertise in government and then thinking about the incentives and infrastructure to make all of this uh reporting go well. Um, and so, u, like I said, um, in the intro, both Mackenzie and I have written, um, uh, reports about this. Um, McKenzie's is in Lawfair. Ours is on our website, so I encourage you to read those. Um, and just some really quick, uh, thanks to the team that helped put this event together. So, on my team, that's Claire Goldman and, uh, Nicole Herrera. Thank you very much. Um, we also had support from Antonio Rivera Flynn and Tori Blakey in events. Sophia Chavez and Ava Rose in external relations. Um, a really substantial streaming and broadcasting team for our AV needs. Um, and Claire Carmy and Rob Block on the web team. Um, and thanks again for coming out in the middle of vacation season to listen to us. Uh, at least some of us will stick around for a few minutes if you want to catch up with us. Thank you.