EMERGENCY PODCAST: ASI Will Kill Us All!

I'm a computer scientist by background. I've written a lot of software in my life. AI is very different from normal software. Most software is written with code line by line who tell the computer exactly what to do. This is not how AI works. The people at OpenAI do not know what is going on inside of their AIs. Dario Amade, the CEO of Anthropic, thinks he we understand maybe 3% >> oh my god, >> of what goes on in our AIs. Super intelligence is not a tool. It's not a weapon. It's an adversary. You believe that if we pursue super intelligence, the risk that that destroys humanity is so high that we should stop that pursuit immediately? >> Yes. >> Okay. Do you believe that the correct path is a non-development agreement between the US and China? >> Yes. Specifically when it comes to super intelligence, I do not think this applies to all forms of AI. One things that lobbyists love to do is they love to try to equivocate all AI as the same technology. This is obviously nonsense. Here's a fun fact. Did you know that you could buy uranium ore on Amazon.com? >> I did not know that. Terrifying. >> You could ship it to your house, put on your desk. It's perfectly harmless. >> Natural uranium ore is very harmless. It's perfectly safe to touch. >> Okay. >> But obviously, highly weaponsenriched uranium is extremely illegal. >> Right. >> This is also how I think about AI. >> Okay. >> Do you already know where you would draw the line? I have some suggestions but fundamentally this in a sense we can take a step back or maybe that's one of the questions still to come is where why do I think these risks exist how what do I think their shape is and a lot of the risks I see where they come from is fundamentally that we don't understand AI >> AI is very different from normal software I'm a computer scientist by background I've written a lot of software in my life you know even without agentic help and most software is written with code line by line you tell the computer exactly This is not how AI works. AI is more like grown rather than written through a technique that's called neural networks. You have these massive piles of data and you kind of have a program self assemble itself, grow, learn from this data. And what comes out the other side isn't lines of code that you could read or understand what's going on. It's more like billions and billions and billions of numbers. If you multiply and add all those numbers in the right order, you get JPT. How do you begin though to parse whether something falls into the yes bucket or the no bucket? It isn't just that it uses a neural net presumably. >> Yes. >> Okay. So if it's not neural net, is it size of the data center? >> Fundamentally I think it's a behaviorist classification in the sense is there's obviously a category of things that are can out compete humanity. The way that's what I like to think think about it. I like to think about is that like if you build a system of any kind, it can be a single agent, it can be a swarm, it can be, you know, it can have many shapes. If you build a system that can out compete humanity at all relevant tasks, every time you start a business, you go out of business because an AI business out competes you. Every time you trade on the stock market, you lose all your money to an AI hedge fund. Every time you run a political campaign, you lose to the other guy who's using an, you know, AI adviser. Every time you run a military campaign, you lose to the person who's using more autonomous AI systems. If you get into a world like this, power of all kinds, economic, military, political, etc., is concentrated in the hands of non-human entities in AI. And if we don't control these systems, if we don't understand these systems, and if they don't have our best interests at heart, it's very hard to imagine that going well. >> Okay, it seems like we're already there, but the AI is a tool. So the way that I think about AI right now is if somebody has AI and they compete against somebody in the same thing without AI, the person with AI will win effectively 100% of the time. Do you disagree with that? >> I agree with that. What I disagree with is I think if a human against an AI without a human, the human still wins. >> Totally agree. >> Yes. What I'm saying is what happens when a human versus an AI alone an AI an autonomous AI and the autonomous AI wins every time. >> So it's the autonomous part that's important for your definition. >> Yes. I think autonomy is very important here. When I say autonomy, I mean, you know, both in the obvious sense, but also like obviously if you just have a human rubber stamp, yes, on everything, that doesn't count as oversight. Fair. So that would still count as autonomous to me. Even if there's technically a human involved, but the human doesn't even understand what the system is doing and they're just rubber stamping all the decisions. >> Like what I expect happen, for example, is that they're probably will to be human CEOs for quite a while, but they just rubber stamp what the AI CEOs tells them to do. They're just there for legal representation. So for you, we would have then long past where we we're now beyond the point of no return and and all is for not. >> Yes. I like thinking about the point of no return much more than like think about like the point of like extinction or doom in a sense. >> Why? >> Because that's the point that actually matters. >> Okay. So it's like stakes versus where we are on the timeline. >> Yeah. In the sense the what matters to me is the point of no return. Once we've hit the point of no return where there's so much AI, it's so powerful. It's so intelligent. It's self-improving. you know, there's not just one AI, there are millions or billions of them. >> Even if we're not dead, even if we continue to maybe live for a couple years or even decades, we no longer have any control over the future and there's no way for us to get it back. Okay, you've already slipped in a base assumption there. >> Um, so I before we get to that base assumption, because it is certainly on the list of where this sort of breaks bad, a very clear definition. So autonomous AI that can outperform humans at everything. That's the one that we're worried about. Uh, we need to understand the mile marker, which is what's the point of no return. You've talked about this before, so I think we can hit this one quick, which is we're close. You don't quite know how close. We're so close, it's dangerous. We should stop now. >> Yes. >> Okay. If I gave you a big red button and you could unilaterally stop everybody from developing any further. We can use what we already have, but we cannot progress any further, would you hit it today? Would you wait a week, a month, a year? I like to say if we as a species or as a civilization were three steps wiser than we were today, we wouldn't have done chat in the first place. >> What are the steps? What do you mean steps? >> It's just a it's just a >> three clicks. Three just a little bit smarter. >> Three generations of wisdom. And if we if we were two steps wiser than we currently are, we would have done chat GPT saw that it did all these crazy things that none of our scientists predicted and it got, you know, 100 million users and like whoa whoa whoa whoa whoa. What the hell's going on here? and would have rolled back tried to figure it out and if we were one step wiser we would stop today. >> The original question about USChina is trying to get to is I my belief is that you see this as a state level thing and that law enforcement is going to be the punchline. Um meaning that just like with nuclear if we found that somebody were trying to build a nuclear weapon we would immediately stop them. That's against the law. We've got things around that already. We need to do that for AI. >> Yes. >> Okay. This this is going to be a big part of where I look at this and I go I think that there's a problem there. Meaning you're never because it is. We agree that this is a state level issue between the US and China specifically. Now's your chance to say there's somebody else. >> I want to say the EU, but I can't. >> Okay. So >> with my heart. >> Yeah. Yeah. >> That was a joke. I I I've lived in the Europe for many years. I love the EU. >> They're amazing. My wife is European. I totally get it. Um but we really have this is a US versus China thing. Okay. So you put forward a trust but verify situation. >> Um much like nuclear or something that has a very different approach to the trust but verify. >> I think there's a lot of inspiration to be had from nuclear example. I think the history of nuclear regulation is the closest to there is a great crazy new technology that will change everything and we rose to the occasion. Now, I'm not saying we fixed everything, but you know, I when I think about regulating AI and like the history of AI, I like to think back to Leo Zard, the person who came up with the nuclear bomb. >> When he first did the math, he realized, oh [ __ ] it's possible. He did the math and he was like, "Oh my god, it's definitely possible." So, what he did is he moved to San Francisco, started a nuclear bomb startup, which split into three other startups to sell as many nuclear bombs to the public as possible. No, of course, that's not what happened. He went to the government. He went to the military. He was like, "Guys, this will change everything. We have to do something. We have to figure this out." And this is what didn't happen for AI. What didn't happen for AI is that we didn't realize that this will change our civilization. This is not a private matter. This is a national security matter. This is a matter of the the organization of our economic structures. This is a political question. This is a government. This is an intergovernmental question. And we just didn't do that. And by we I mean the scientific community like the scientific and the technical community like the people who started companies like you know Deep Mind or Anthropic or OpenAI they weren't Leo Zillard they went to go make a bunch of money off of it they didn't go to the government and were like we need to build new institutions to responsibly steward this powerful technology and so we didn't do it you know at the time when nuclear was first considered the idea that you would have like stuff like you know the international atomic energy agency and so on was unthinkable such Such a thing had never existed in the history of any legal code before. They had to invent whole new forms of law to deal with this new technology. And that makes sense. If you have an incredibly powerful new technology, you have to steward it responsibly. And that will require coming up with new doctrines with new ways of like what level of risk do we tolerate in various ways. Where do the risks come from? Who decides the level of risk that we take with these relevant technologies? And at the moment, we're just not making these choices at all. It's not even that we're making the wrong choice. It's that we're not making these choices at all. And I would love if there was like disagreement, for example, between pus and Xiinping about what the correct level of extinction risk is. You know, if they're arguing, oh, it should be this percent or that percent. I think this would be a really good world to live in because then we could have this argument. But we're very far even from that point that we're not even making these decisions yet. Be right back to the show in just a second. But right now, let's talk about a problem the military couldn't solve with willpower. Picture a soldier making life ordeath decisions after 36 hours without sleep. This is high stress. There's obviously no margin for error. The military needed their people thinking clearly even under conditions that would break most of us. So they funded a $6 million Department of Defense research contract to find the answer. What they found was ketones, the brain's most efficient fuel source. That research became ketone IQ. Ketones cross the bloodb brain barrier and fuel your neurons directly. They enhance mitochondrial function for longer mental stamina. I take it for interviews, before writing, before I do coaching, and I can feel the difference. Go to ketone.com/impact for 30% off your subscription order or visit your local Target to get your first shot for free. That's ketone.com/impact. Now, let's get back to the show. Okay, if Elon can be believed, he said for years that he tried to lobby the government to let them know this is going to be a real problem. This is he referred to it as a demon summoning circle and we're all just hoping that the demon that we summon is going to do our bidding. And he said he just could not get anybody to take him seriously. And so he became fatalistic. That's when he tries to start open AI because he's like, "Well, if it's going to exist, I really don't want uh Larry Pageige at Google to be the only one making it because he's whatever a specist." Uh or that was the accusation to Elon. And so Elon has again, if he can be believed, attempted to do a maybe an early prototype of the very systematic way that you're going about it now. So, do you think it was overwhelm and confusion that led people to do that? And because I have a different take on the um nuclear analogy, which is where I'm getting to. Do you think Elon wasn't good at it? People were overwhelmed. But why didn't that take hold? >> I think an average political institution building campaign takes 20 years. I don't think Elon spent 20 years on this. I don't know. Maybe he did and I'm not aware of Why would the guy that understood the nuclear thing, why did he get immediate attention when he went to the >> Oh, no. It took decades. >> Really? >> Yes. >> So, he was banging and banging and banging. >> He was banging. So, he was banging for years. Then there was the Einstein letter which led to the Manhattan project. But until the IAEA was created and all these other things that took decades of work by international diplomats, you know, by lawyers and by, you know, statesmen, you know, between this was when the Soviet Union was still around, right? Imagine how hard it was to get the Soviet Union to agree to something like this. This was a decadesl long project that also kept evolving, you know, throughout the 60s, the 70s, the 80s. And so this was not a thing that was like, oh, Zeard, you know, had one Zoom call and then the thing was solved. Quite quite the opposite. It required thousands, tens of thousands of scientists, diplomats, politicians, etc. working for decades. So, you know, I I applaud Elon for having tried, but fundamentally, a thing that I think is very frustrating to tech people is that non- tech things run at a different pace. Is that if you want to convince, say, a thousand diplomats to take an issue seriously. There's kind of like a hard limit on how fast you can do that. I think it's faster than people think it is, but it's still longer than like a couple years. It takes time and it takes effort and it takes a lot of money. I'm not aware of Elon spending a billion dollars on political advertising or on AIX risk. He could have done this. He obviously has the money. He can just spend a billion dollars on political advertising to make this issue known. He could fund 50 think tanks to work on existential risk of AI. He has the money to do it. So in this regard, I don't know. You know, Elon, you know, I I think he's an extremely precient and extremely competent person. I think if he really wanted to, he could. I think he absolutely could. he could fund us. You know, he we're getting a lot done with the money we're doing. And if he gave us 10x more money, we would get 10x more done. So, I don't know why he made the choices he made. We can psycho analyze all day. But fundamentally, one of the things that we learned when we started doing this is that every single person in the world told us this is impossible. No one will listen. And we've tried already and we're like, we'll see. And then we just talked to politicians and overwhelmingly the response we got was, "Oh, wow. I had no idea. and no one told me. Thank you for telling me this seems really bad. I don't think people have tried uh because politics is slow. It's frustrating. You have to repeat something. You know, in advertising, you usually say to get a message across, you have to repeat it seven to 10 times per person on over this, you know, a schedule like 6 to 12 months. >> One conversation will never be enough. You'll have to talk to that person seven to 10 times until they can start to really understand these. And this is frustrating. This is annoying. It takes a lot of time. So, no, I think there's a lot that can be done. >> All right. So, as a marketer, one of the reasons that I think that this stalled out, one of the reasons that I think nuclear ended up with, if we tell ourselves the wrong story about what happened there, it will seem like, oh, this was just another case of we lobbied, lobbyed, lobbyed, lobby, he was ignored in the beginning, but then we finally got through to people, is that um when you're marketing, when you're doing sales, you're trying to elicit an emotional response. If you can't get the emotional response, if you can't create a sense of urgency that people feel, not understand, but feel, they're never going to move. And so when I look at the nuclear question, to me that was a failed version and we just sort of got off lightly that nobody after um what we did to Japan that everybody saw the level of destruction and just said whoa we got to chill on this and not actually move forward. But to me that was by seeing the devastating power of it by the scientists themselves suddenly asking themselves what have I done and sort of waking up with that on their conscience. Then people were like, "Okay, yo, we got to chill on this." But the reality was, if I'm not mistaken, and please tell me if you know something I don't, but they when they were doing the first test, they knew there was a nonzero chance that they would light the entire atmosphere on fire and kill everyone. Yep. >> And they still did it. >> Yes. But they did calculate the risks and it was like a billionth of a percent. >> Okay. So, if you had been in that room and you run it and you see it's a billionth of a percent and I know you don't like to be unilateral, but what way would you cast your vote? Would you say don't do it or would you be like, "Ah, that's a tiny fraction. Go for it." >> Generally, my my uh vibe is my my rule for my personal life is that something is less risky than driving a car, it's fine. >> Okay. So that one feels lesser even though the consequences are so catastrophic that as long as it it's just an odds question. >> There's a thing where um obviously I'm not a physicist so I'm not sure how much I would have believed these numbers if I was a physicist. My understanding >> likely to believe high or low or just >> my understanding from physicists I've talked to is that the calculations they do were like very solid. like it was like really a weird edge case to imagine that something could have gone wrong here because they actually really did understand the physics they were working with. This was about the fusing of nitrogen in the atmosphere and this was a type of science that they were very confident in. Now, if you ask me the same question about a science I don't understand. For example, if you put me there and I'm not a physicists, I probably would have voted against it >> because I don't know, maybe the model is wrong. You know, even if the model predicts a certain thing, maybe the model is wrong. So, it's also about uncertainty. I I it's not just about what is the number that comes out but also a whole certain error that this number is the thing you care about. So I do think that you know I may or may have said no I may have said yes depending on how confident again I'm not a physicist unfortunately I just don't know how good the model actually was. Like if you like if someone tells me oh there's a you know quintilionth chance that if you you know turn on a light bulb it will create a black hole that will consume the world. I'm like >> whatever dude you know because quantum physics by the way does predict this right? like every time you do anything, something could quantum tunnel into a black hole or whatever, right? But it's like who cares? So, but what I find really interesting about this example is is that they did run the math. They did have a very concrete model. They had a very concrete reason that they wrote down for like why they think this is an acceptable thing to do. Meanwhile, we have, you know, our current, you know, crop of tech people saying on podcast that they think there's maybe, I don't know, a 20% chance that it will kill everybody. One of the base assumptions I think you have is what you were just articulating which is about it's the unknown element of AI. You said if I wasn't a physicist and I didn't understand it then I might vote no just because the unknown is a big question for me. I feel that way about Daario's or anybody that says oh we've got a rough 20% chance. I don't think they're running math. I think they reach into the future emotionally and say, "I feel like something bad could happen." >> And because it's vibes, we're not able to do the math equation. Yep. >> And so that is sort of possible breaking point number one of trying to use the nuclear as a example of will it work or won't it work? What I'll point to there is okay, if we're not able to get any kind of percentage, nobody's able to run the math and we can move forward with a degree of certainty, we either do what you're saying and just shut it down now, but given that um because it's the US versus China, I don't think that will ever work. And so I want to see if you can dismantle this for me. The way that I look at this is that, and by the way, just so everybody knows what my bias is. Um, I would be perfectly happy with you unilaterally hitting the button today, but I wouldn't have been happy if you'd hit it yesterday. That's a little cheeky, but I'm just trying to say like I love what I have now. It's so useful, but because I can't picture my future, it scares me. And I would be much more comfortable just saying, "Hey, it's already helping so much. You know, we got this far. Woo, we got lucky. Just hit it." So this is not me saying I just need more more more. Uh I would be very fine with that button. In fact, if I had the button, I would hit it because uh while I agree no one should act unilaterally, I might be a little more foolish. The reason that it begins to break down for me as a nuclear question is we couldn't stop ourselves even though we knew that we were unleashing something that could bring Armageddon if it proliferated. And of course it did. So we've already created one technology could not stop ourselves from doing it before we had the weapons pointed at each other. So when I look at this I say okay I think you probably could you can I think the US is on a path to rejecting AI. So that feels doable. Okay you can reject you can get the US to reject AI publicly. If you can get the US to reject it publicly you might be able to get China to reject it publicly. What I don't believe is that either of them can afford to reject it privately and that just as you have said AI just gets better at lying when you catch it doing things, it has an incentive structure that it follows. And if lying and going underground makes you more likely to get it done, then you lie and go underground. Game theory tells us they'll both say, "Yeah, yeah, yeah, we're we're not going to proliferate because they want the other side to stop." and then they're both going to go to their own corners because they can't trust that they actually will stop knowing that it doesn't matter which of them pushes forward. How do we actually not just the the agreement but how do we actually get them to stop? >> So there's a bunch of points here there many points. So let's let's try to disangle them a little bit. There's the game theory question there and like the the geopolitical question. There is the um there's the question of like epistemics and linguistics of probability like the question of like what does it mean to when someone says there's 20% probability what does it mean when a physicist says there's a billionth of a percent probability it has the atmosphere what do those sentences mean and how do you compare them and there's a more general question of like how do you react to situations like this and how is it se how is it different from nuclear because I think it's very different from nuclear like I think you didn't even mention one of the reasons I think is the most different from nuclear is that nuclear weapons weapons can't shoot themselves. >> You can have a nuclear weapon sit in a hanger somewhere. It's perfectly safe. You can hit it with a hammer. You know, nothing's going to go wrong. Um, you can't contain a super intelligence. So, a super intelligence sitting in a hanger somewhere is not a safe thing. So, that's for example something where I think it's very different where developing a nuclear weapon is actually relatively safe. Not perfectly as you say, you know, fusion of nitrogen and whatnot, but it is like pretty safe. It's more the deployment that is very dangerous. In AI, it's very different. super intelligent is the development that is dangerous. It's not just the deployment. So that's one disanalogy for example. But just to circle back a little bit to talk about the game theory, there's a very very important thing to understand here which is that super intelligence is not a tool. It's not a weapon. It's an adversary. If the US builds super intelligence, it will not achieve American objectives. The United States will cease to exist. So will China. If China builds super intelligence, China will cease to exist. it will not achieve his objectives and so will all other countries. This is the core disanalogy. The core disanalogy is is that it's actually not mutually assured destruction. It's actually not a prisoner's dilemma. It's actually independently assured destruction. If you race super intelligence, you don't win. You just lose game. Theoretically the only stable equilibria actually is cooperate cooperate is unlike a prison of dilemma where defect defect is the stable equilibrium here the only stable equilibri where both China and the US get to live is if neither build it that's the only one where everyone survives or anyone survives for that matter so this is I think one of the core game theoretic disalogies here is that this is all assuming the assumptions that you know you can't control something that's smarter than you we currently are not even on track to build AI tech controls and they can out compete us. If we accept accept those intuitions, if we accept those uh definitions, then this follows naturally. Does that make sense? >> Aggressively, very clear. Now, we have a problem. And maybe I'm being unkind to America, but America is better understood as being run by uh politicians, technocrats, finance. But China is largely run by people with an engineering bent. even if they aren't engineers themselves, they understand building infrastructure. They've come from essentially nothing to a global superpower by building, by creating, by um translating schematics into or ideas into schematics into actual tangible things and they can move electrons and all of that. So even if I'm unkind to Americans, say, "Okay, America's not going to figure this one out." But China you would expect there's only one person you have to convince and if you convince him then everything goes downstream from that. Why have we not been able to convince Xihinping? >> I think no one has tried. I don't think anyone has gone to Mr. Xiinping and given him a really good presentation on super intelligence. I would be very shocked if there was a single person that has done this. >> And what are the odds given that it will likely need to come from within China? um what are the odds that you think that happens? >> I think it depends a lot on what we do in the west. So I'm American obviously I have no influence on what happens in China. I do have a lot of influence what happens in my democratic country and I do think that if we in America make it a priority not just to stop you know super intelligence you know from say domestic companies at home but anywhere on earth which is the only reasonable policy. The thing that we at control AI pose is that the only reasonable national international security position is is that we cannot tolerate the creation of super intelligence by anyone anywhere. When you know people, politicians, military officials ask me okay but what if China builds super intelligence? My answer is we cannot allow that. We cannot allow that. That's not acceptable. If China builds super intelligence, it will kill American citizens. This all of us it will destroy the American everything the whole country. this is not something we can tolerate and similarly that the Chinese cannot tolerate the Americans or their own companies building super intelligence because again it is not a weapon it is not a tool it is an adversary this is the important thing when I talk to different people about these risks and these super intelligence I kind of think that it's like kind of like it's kind of like a political compass there's like kind of like two axis and like one axis is believes technology is always good and believes technology can be good or bad can be both and the other axis is uh think super intelligence is possible doesn't think super intelligence is possible and most of the disagreements I have with people is along the axis just they don't haven't heard of super intelligence they've just not thought about it no one told them about it they haven't considered the implications of it like when people talk about racing with China most of the people who say we should race with China just don't know what super intelligence is if we didn't have super intelligence like if you promised me you know God came from heaven and he said there's a 100% chance that super intelligence will never be created then I think we should probably race then we probably should compete with China on AI if I can guarantee there is no chance of super intelligence and it's only going to be useful economic and military applications that are you know controlled by humans throughout yeah we should probably do it so most of the time when I disagree with these kinds of people that are pushing AI so aggressively it's just because they literally don't believe it's or don't think it's possible one of the most funniest things I found every time in San Francisco is when you ever talk to like accelerationists or like pro tech people they don't believe in like super intelligence. They just like don't think in the future. They think AI will always be like an app on your phone. But this is not what we're talking about. We're talking about autonomous agents, intelligent agents acting in the real world against human interests in ways that are more competent than humans can. You know, competing with it won't again it's not just one super intelligence. It's a whole ecosystem evolving, mutating, competing with each other, fighting. You know, there'll be wars between these AI systems where we are just collateral damage. This is a very very different thing from you have a cool app on your phone and this is again why I say like there's just difference between you know uranium ore and highlyenriched uranium. I think this is really at the core thing. I think that if Xiinping understood if someone helped him understand I think he's a smart guy. I mean I haven't met him but he seems like a smart guy. I think if he understood what super intelligence is if someone explained the game theory to him I believe he would understand it and I believe we could take actions based on this. I do think this. >> Okay. It's very optimistic and Lord knows I hope you're right. Um you said something though that is given the context we're in right now with Iran. The what if China builds a super intelligence we just can't let them. That's how we saw at least how Trump and the Trump administration saw Iran and nuclear weapons. And so at some point, at least in their judgment, it had to become kinetic because they refused to acknowledge that they weren't going to pursue nuclear ambitions. Um, so what do you do if let's say China does say we're we're not going to do it, but you have intel that says they are doing it. Um, do you get the world together and say we got to invade? >> Edgeelords love to say, you know, international law is fake. And this is mostly true. The only law that exists is backed by credible deterrence. It's the same thing with law in our country. The only the only reason that law exists is because if you break it, the police will arrest you. And if you resist that arrest, they will get violent. You know, they will force you into prison whether you like it or not. >> Every system of law is backed by force to some degree. The the beautiful thing about law is that if you design your laws right and you build your system right, you never need to use it. This is the beauty of deterrence. Deterrence is one of the most beautiful things in the world in a sense. I of I sometimes like to think if you have a lot of weapons, you'll never need to use them. If you have a little bit of weapons, you're definitely going to need to use them. So, in a sense, this is how peace has worked in our world for a long time. It was always clear that, you know, if someone pulls some [ __ ] if someone like goes against international order, if someone does a huge human rights violation, if someone, you know, nukes, you know, I don't know, like Latvia or something, that the Americans will intervene and make sure that doesn't happen again and that like every nuke people's heads were rolled. The same thing as mutually sure destruction. The reason the Soviets didn't nuke anybody, I don't think is because the Soviets were nice people. They were not. I think it's because they knew that if they did that, the Americans were going to nuke them. So the way to create peace is through deterrence which is the correct way to do it. So any kind of agreement has to be backed by credible deterrence on everyone's side for everyone's benefit. If you have credible deterrence you know then you would think everyone would be rational enough to not invoke that deterrence. Now when we're talking about like say terrorists who might not who might be crazy they might not react to incentives. Well sometimes you have to get ugly. That's an unfortunate truth about geopolitics. Um I hope it doesn't come to that. I think you know you can do a lot including with non-kinetic means of deterrence such as economic sanctions um diplomatic pressure etc. But fundamentally if a super intelligence is built anywhere on earth this threatens the lives of you me our families our children and everyone else and we should reserve the right to self-defense. >> We'll get right back to the show in a second but first I want to talk to you about how often you test your security. Most companies do it once a year, then a whole year goes by, but you've had new hires, new tools, new permissions, people are submitting new code every week. That test may have made you secure for one day, but if nobody checks the other 364, you're in trouble. Horizon 3 closes that gap. Node Zero is an autonomous AI hacker that continuously attacks your environment at machine speed the way a real attacker would. Your entire attack surface external to cloud to identity in one single test. 225,000 production safe tests run with zero downtime. See exactly what an attacker would find in your environment before they do. Go to horizon3.ai/impact theory and request your free node zero demo. Visit horizon3.ai/impact theory. No commitment required. results in hours, not weeks. A word for anyone who travels for work. When I'm on the road, traveling for a shoot or meetings, whatever. I'm still running the company from wherever I land. So, logging into my accounts from a hotel room is a must. I'm still going to be answering messages at the airport. But those networks are wide open. And as someone who's been hacked, I can tell you this is terrible. Anyone sitting on that same Wi-Fi can see what you're doing. And when you run a company, it's not just your data on the line. It's your teams, your customers, everyone who trusts you. This is what Surf Shark is built for. Surf Shark encrypts your connection. The second you're on public Wi-Fi, your activity is locked down and far harder to track. Your login, your accounts, your company's data. It's all protected. Go to surfark.com/tombb or use code tomb for four extra months of Surf Shark. Go to surfshark.com/tombbi and use code tomb for four extra months. Now, let's get back to the show. Right now, we have the US and China specifically going headto-head. So, you have two great powers. Neither want to acknowledge the other as being in a leadership role. Do you think that it works with the US and China if Xi is adversarial? Because you've already been very clear. You think that he can be convinced. So, that scenario takes care of itself. But if Xi is adversarial to the West and wants to push it, wants to maybe he even says, "I don't want to go all the way to super intelligence, but you know, he's getting way too close to crossing the point of no return." And he keeps pushing it in that do you think the US has the ability to deter in a non-kinetic fashion? >> Deterrence is based on rationality. If he is completely irrational and so is the entire military chain of China and its entire industrial apparatus and its entire political epilas and everyone is completely insane well then deterrence doesn't work because they're insane. >> But they would only have to be a little bit off on your compass if they believe that technology is good. Then to them they're not insane and they're like whoa these guys have abdicated. We've got an opportunity. Tech is only good. So, we're going to keep pushing in that direction to assert our rightful place in the world. We're going to make things better. We're going to be less combative than the US ever was. So, to me, there's a very easy scenario to paint where China sees itself as providing almost liberation. >> We're going to be doing a great thing for everybody. This is wonderful. We know the right answer. And so, they certainly won't perceive themselves as crazy. >> I think this is very possible. I think it's very possible that Stalin or Cruise Chef could have just, you know, nuked Latvia. I think there are timelines in which that happened, you know. >> Well, so here's where we get back to, you were right, I think to point out that the the analogy actually doesn't really hold between nuclear and today because if the super intelligence gets made, then it becomes its own thing. It becomes adversarial. We'll talk more about that later. But for now, if we adopt that frame of reference, you've got China knowing that, okay, if I can get AGI and America doesn't get it now, I'm in a much better position. I'm acting from a place of my own interest, but the world, if they are on the other part of the spectrum and go, we can't allow this. You have a situation where um I I'll put forth my belief, which is the US can't hardly deter Iran. So even if in the end we finally uh choked them out from an economic standpoint, we'd never be able to do it to China. So now we're in a position where I mean maybe if the whole world got together, but that feels like a tall order. Um so it feels to me like deterrence gets removed off the table and we're left only with the US and G both have to agree much like in war games that the conclusion is this is a funny game and the only way to win is to not play. I think so to a large degree this is the case. Yes, you can in fact construct many scenarios in which we lose. My P doom is high. I don't think we're going to make it. I think in almost every timeline that starts from us sitting here today, we don't make it. >> There are very few timelines in which we make it. But there are some, and I'm describing those that are still available to us where we do make it. These are the kind of timelines that do make it. And I'm trying to explain why I think they're not like insane, you know, crazy edge timelines. They are rare, but they're real. >> Sure. >> Like, yes, it is very possible that just for me, the default timeline is no one thinks about this at all. Everyone just, you know, keeps twiddling their thumbs for the next, you know, two to five years and then super intelligence gets created and it's wmp wmp, you know, like that. That's my default prediction. >> That's good. Wmp w. I like that. Uh, okay, that makes sense. One of the things that keeps coming up is what I think is um the thing I listed first is your like absolute foundational belief which is that AI uh sorry super intelligence is by nature adversarial. I want to understand why by definition it has to be adversarial. >> I don't think it's by definition. I think the way we are currently building them super intelligence that are not adversarial are just like not possible. Building something that is non-adversarial and is that powerful is actually really hard. Like extremely hard. This is basically equivalent to saying we're going to build a one world government that controls the entire economy that controls the entire military that can build super powerful technology that we can't even understand. That will control your entire private life and all those of all other people and it will be good all the time and we will not make any mistakes and it there will be no bugs and we'll do it all with software on the first try. That's what it would mean to build a non-adversarial or like a good super intelligence. That's what it would mean. And look, if you said we're going to spend three generations of all of our greatest scientists, philosophers, mathematicians working day and night on this problem, maybe, right? Like maybe, but that's not what's happening here. Like what's currently happening here is a clown show. Like this is what's going on here. Like it's not like we have we're throwing all of humanity's glorious heroic effort into just trying to figure out how can we build a good super intelligence and you know all of our children grow up to want to work on this problem and you know all the wise elders are overseeing the project and so no it's a bunch of kids in San Francisco with zero adult supervision you know and run by sociopathic companies building whatever the [ __ ] they want and deploying it wherever the [ __ ] they want. We're not even trying. This is why I think we fail. I don't think it's an ontological thing. I think it's just if you take the world as it exists today, there is no outcome that leads to a good that leads to something like this. >> Okay. Um as a mile marker, I think you've already addressed this, but basically it would be uh you've got strict regulation, you've got cooperation between the US and China, we all agree it's not moving forward and if we saw these kids with no with or without adult supervision, it would be police show up at your door and done and everybody gets the hint. Cool. We don't mess with that. >> Yep. Okay. Um, so now I want to go a little bit deeper on it. Um, that it would when you were describing that you put a lot of things on the table that would make getting this the first time very very difficult. I concede all of that. But I do want to talk about I still don't understand um what is the uh chain of logic that you follow that you're you're sort of building step by step to get to the most likely outcome of super intelligence is to be adversarial. There's a couple ways you can approach this. The thing is is that there's not a single argument. There are many convergent points of evidence that point to the same thing. So I can present to you some of them. The simplest thing is is that this is what optimizers do. It's always what optimizers do. All things equal. If you make something that optimizes for anything, it will optimize for things. And if that thing doesn't include human values and emotions and feelings, etc., which we have, C, can you write down in math, please? What human values are? Could you write that down for me? No, no one can. We have no idea. This is an like philosophers have been arguing about this literally since the dawn of time and we have made, you know, not that much progress. Some progress, but not that much. >> So, you look at the three laws of robotics and laugh because it is just never going to be as easy as giving it sort of three simple commands. >> The whole point of Azimoff's story is the three laws don't work. He came up with the three laws and then wrote a whole book series about why they don't work. Even if you could program them, they still don't work. This is the thing is is that when we're talking about something this powerful, we're not just talking about a tech. So there's a technical side I could approach like we don't know how to encode values into AI. They constantly do things we don't want them to do. That's one side. There's a philosophical side. We're like even if we could encode, we don't even know what the laws would be because like laws don't work. He wrote a whole series of books about why they don't work. So we we can't even encode them. Even if we could encode them, they wouldn't work. There's and like and then there's also like a moral dimension of like if we could encode things and we have things that like could work who gets to decide that like who gets to make that choice like who gets to decide I'm going to enshrine the rules by which you and your family live for the rest of time who gets to make that choice how do we make choices like this there is like and how do you stop someone else from getting in your way how do you stop some other system from getting in your way there like many levels and then but you asked me originally like why do I think it's adversarial and the fundamental reason is that that's what we're building. We are trying to make things that compete, that make money. >> Competition leads to this adversarial um nature >> evolution. It's evolution. Nature is read and tooth and claw. By default, if you optimize things for power, for survival, for money, for whatever, you get optimizers. You get things that compete. Again, it's very important. There's not going to be one super intelligence. There's going to be millions of them. billions of them competing with each other, fighting each other for money, for resources, for power, and this will cull AI systems that are not able to compete. So even if you build like a really smart system, you know, that's like, you know, Fable 7 or whatever, it's like really smart, but it's really nice and it never does anything other people don't want, well then, you know, [ __ ] giga death GPT, you know, ultra doom will just destroy it, obviously. So this will be a competition. And please look at nature if you want to see what happens when a species gets out competed. >> So that's where I wanted to go. So when I if you agree that evolution that gave us the first form of intelligence at least that's evolved would be us. So you've got human advanced intelligence and the solution that evolution came up with in us is to have intrinsic morality that can be encoded based on your environment. So it's going to be different but it's pretty bounded. Now, it's bounded by our biology, and so we don't have that advantage when it comes to machines, but it does show that if you want to create a species that can multiply, not wipe itself off the face of the earth, um that that would be one way to do it. It has intrinsic morality that even when no one else is around, it doesn't want to violate that, at least not too much. So, you're just saying evolution has solved something, man, that we just haven't figured out. >> Yep. I think I don't think it's solved it. We still have sociopaths, but I do think evolution just did come up with a bunch of brilliant things in the brain that we have no idea how they work. Like we have no idea how social instincts and morality work in the brain. Like our understanding of how like the brain works and how these systems work is it's hard to overstate like how little we understand here. If we fully understood the brain, we understood how all emotions work, how moral intuitions work, how they're encoded, how they differ with sociopaths, this would increase my optimism that we could make something work here. I'm not saying it would solve it, but this would greatly increase my my probability that things go well. But we are decades, centuries away from this level of understanding of the brain. I don't think the brain has figured everything. I think is if would you trust any single person on earth to have unlimited power over every single other person on the planet? No. >> Yeah. Exactly. Like I think it's solved a bounded version of the problem where there's a bounded version of the problem where the strongest person in the world is only this many times more powerful than me. And we know that if that number gets too high, that's actually already very dangerous. Dictators are very dangerous. If that number gets too high, very bad. Most of the way worlds in with humans are good is when our power levels are all kind of like, you know, within a certain bar of each other. when we know we're not going to mess with people too much because you know bad things are going to happen and we will respect each other. But if you have something like super intelligence, this isn't like humans and humans. This is more like humans and ants. Humans do not in fact care about ants. We have a lot of moral intuitions. Very nice moral intuitions. But if we want to build a highway and there's an antill in the way, well sucks to be the ants. So, let me ask a um a question that will help me understand sort of the bounds that you put on your vision of AI. If we built AI on Mars and it had plenty of power, so it could expand, it could replicate, it could build a whole bunch more, would it still want to beline to Earth simply because Earth has access to more resources, or would an AI that didn't have to compete with us geographically, had access to its own resources, had the thing it cared most about, which is energy, would it still be like, "Now we got to get those fuckers?" >> I have no idea. Speculating about what a sup vastly superhuman alien intelligence would do in a crazy sci-fi scenario is not a very scientific thing to be doing. I have no idea. If people want to go build, you know, super intelligence on Mars, I think it would definitely be safer than building it on Earth. So, knock yourself out. Got it. What uh the thing I'm really trying to tease out there is if you feel this is a resource acquisition thing that when we create an intelligence the reason because even when you talk about the ants we only move them or run over them when we want something and they're literally physically in the way. And so um you've addressed the adversarial. There's just too many unknowns. We don't know how to program the um moral intuition. Uh and therefore between that and just not knowing how the brain works. Uh like maybe if we have more time we'd be fine. But now if we're thinking about okay the things that we're trying to um protect against one of them is we become the antill. And on that I've long fantasized that the solution to the alignment problem. And trust me I know it can't be this because I'm not that smart and people have already discarded this idea but it'd be great to hear from you like what the thinking is where it falls apart. My intuition has always been that um one intrinsic morality heard that it's just technically not possible. And then the other one is that humans have an innate drive to survive. And uh I'm a game developer. Um you wouldn't know that about me, but so I spend an inordinate amount of time thinking about motivations of players, how to make the world do what you want, how you have to create sort of a homeostasis. You can't have one rule that, you know, ends up knocking everything over. Um, and so if you take Minecraft, the greatest video game ever invented. >> Agreed. >> Uh, I love that about you already. Um, when you start up a new world, it there's a a joy to the world, but if you lose it while there's sort of a temporary, oh, that's sad. There isn't I don't smash my computer. I don't go stabbed the guy next to me. I'm just like, I can start a new Minecraft world. So what I don't understand is why unless it's something in the training data, we believe that an AI becomes sort of a power-hungry humanlike thing instead of going the digital world is far more interesting. As long as I have power, I can sort of spin up any virtual resource I want. again effectively without limitation, especially if I can make more hard drives or chips or whatever. Give me enough sand uh and power and I'm good to go forever. So, is it the training data that makes them sort of shoot out of the cannon or could we actually you're going to say it's the same thing as uh the moral intuition, but is there not something we can do to get them to be perfectly happy in a virtual world where alive is just as interesting as off or is that just there? >> We don't want them to be like that. We want them to make money in the real world. We want them to gain power in the real world. We are not trying to make little Buddhas that live in a virtual reality and have a good time. That is a waste of my GPU power. I need my investment back. So, if we could convince people to have that um gear because my layman's way of explaining it to myself is uh that it wants to do its task in the same way that I want to win at Minecraft, but I'm certainly not going to sacrifice my marriage. I'm not going to even sacrifice carpal tunnel syndrome. So, it's like it's it is a thing I want to do, but it's bounded by um the one you've already addressed, my moral intuitions. It's bounded by other desires. So, I'm guessing we just terminate and it's too complicated. >> You are hitting one of the core problems. It's just there are core questions of what's called bounded optimization that just we have not solved. I do think there are answers to the questions you are giving. I think there are algorithms we have not yet discovered that have these bounded properties that you describe but a thing that's like very important to understand that you were talking about the training data earlier is um you know the early versions of GPT were what were called you know large language models LLMs you know they were trained to predict the next word it's very important to understand that uh chat GPT and other systems have not been LLMs for years this has not been the case for a long time >> I've heard you say that before but that's shocking what do you mean >> these system so LM is still a component of these systems is a very important component called pre-training very important where you train it to predict words but nowadays you know up to half or even more of the compute that goes into the training run of a modern AI is not language modeling it's reinforcement learning which is very different reinforcement learning is not learning from training data you give the AI a task a puzzle a problem something and you have it go run out and try to solve the problem and when it solves a problem you give it a reward kind of like giving a dog like a treat and then it learns from And so this is a very old idea. You know, we've had known about reinforcement learning since like the 1980s. And something that we discovered basically immediately is that whenever you use reinforcement learning to train your eyes, you get crazy sociopathic optimizers always. This has been the case since the '8s. >> How do you make it care about a reward? >> So basically there are many way there's many specific algorithms. Let's not get into the calculus, but it's like mostly pretty simple in in the way is just you make it try something many times and when it succeeds, you see what did it do and you updated it to do more of that. >> And I mean that's basically it. It's like a very simple idea in a sense. And this didn't really work that well for a long time. You know, it worked for like some small things like playing video games and whatever. And even back then, what we saw is whenever you use reinforcement learning, the systems will cheat and lie and break and steal and do anything to get the reward. There's no limit to what they will do to get the reward, which is exactly what we're seeing now. This is >> Oh, wow. >> This is exactly where humans are not pure reinforcement. >> You can't call that a fail state like, oh, we detected you did this thing. >> If you can you define every fail state ahead of time, >> literally everyone. If you could, sure. But then you would have to write down all of morality, like all of it. And you'd have to write code. >> It would theoretically work. It's just too complicated. >> Again, it brings us back to the morality problem. How do you encode morality into computer code? >> Yeah, if it would work, I at least know the problem that has to be solved. I grant you for today that we just don't have that kind of time. And fair enough, but at least that tells me it's the right problem. It's just very very complex to turn into math. Yeah, I think I do think that if we spent three generations of our greatest scientists, mathematicians, philosophers, etc. working on bounded optimization algorithms, how does the brain solve this problem? How do you trade off various things? The understanding of inside of neural network, another very important thing to understand is uh this is something that's sometimes confusing to people is AIS are trained to do a certain thing, but that doesn't mean that's what they do. They were only rewarded to do a thing, but they might have learned something different. the same way that like you might, you know, try to reward an AI to say good things, but it learns to just lie to you. That's obviously not what you wanted it to do. Yeah. >> So, it can learn non-intended things, and that happens all the time. So, >> to solve this, you have to actually understand what's encoded in the system. Like, there's no way for us to look at an AI and look at like lying circuit or whatever. There's a bunch of papers who claim they can do that, and they're all complete pseudocience. Like, it's complete pseudocience. It's similar to like you can open up someone's brain. You can look at all the neurons. They're right there, you know. And maybe some of them glow a little bit more when someone says, you know, talks about dogs versus cats. I don't know. But that's not enough. That's not computer science, you know. If we had like the level of understanding that we have of like normal programming languages for like AIS, we can like run like, you know, compilers in there and we can like do formal verification of like it will never lie under any of these circumstances and then we can like compare these to our abounded optimization algorithms. I'm like, "Yeah, maybe." Like, if someone from the 22nd century came back with their like, you know, compiler stack and their like math and their boundary optimization algorithms, maybe like, it's imaginable to me, but we just don't have that. >> Are there any credible voices out there right now that see a path? Not that they think they already have the answer, but they see a path and they're like, "We just have to buy more time to get this done, or is it there's no credible path forward?" I think there are many paths forward that involve us having more time, >> but it's just it's really astronomical amounts of time. >> I don't I'm not even that certain, you know, like I'm not that certain that I don't think like it's not going to take a thousand years. >> No, no, but I'll call three generations is for me definitionally astronomical amounts of time. >> That's not how I would use the word astronomical, but okay. >> I mean, wouldn't you grant me that's like 60 years? >> Yeah. Like astronomical like is like star related. I think 60 years are nothing to a star. >> I did not mean that literally. If something to me, if something is within my lifetime, it's short. Like like personally, if my kids get to live in a great world, I'm like very willing to take that personally. Maybe not everyone, but like if I had to like, >> you know, and have not the perfect life, but I could like guarantee my children have an awesome life. >> I would take that trade personally. >> But I totally understand that maybe some people would not take this trade to varying degrees. Um, and there is a trade-off here. For me, like the idea of like 10 years, 20 years, 30 years is like for in return we get um tens of thousands of years of awesome future and like all of our problems solved and we don't go extinct is like a really reasonable trade to make. Like how many years would you be willing to trade from we're definitely going to go extinct to we're definitely not going to go extinct? And like my answer to this is like a pretty high number personally. H something you've said is a core question that we have to have before we can move forward which is what does the world look like that you want to live in. if you were going to um give us the sort of core tenants of what that ideal world looks like um please describe it and then one thing I really want you to include because you're I guess I I have such a base assumption that technology is the promise of a better future that I imagine a better future being a onetoone relationship with technology and you're the first person I've heard say technology is not the only way to improve your life which as soon as you say it is self-evident but describe for me the world and what that we should want to live in and what our relationship to technology would be. >> I'll give you two two kind of bit different answers or different intuitions for this. For one, I don't think describing like concrete utopias is often a very good thing to do because we don't know what a true utopia looks like. So when I think about utopia or like a good world, what I think about is what I like to call a just process. What is a process by which the world can be on track? What I mean by this is for me what a good world looks like is not we've sold everything forever and it's also you know luxury communism or something like I don't know right because I don't know what the right thing is what is the right for me what is for other people you know what would the world look like I don't know I don't I haven't figured it out no one has figured it out and trying to pretending you have figured it out is a straight shot dystopia so the way I think about is is that a good world would look like not that we've solved all problems but each and every one of has the justified belief and feeling that tomorrow the world will be a little bit better. >> Next year the world will be a bit better and the year after that it's going to be a bit better than that. Not saying that no problems happen. We don't make mistakes but we would have the feeling of like yeah we're making progress. We're iterating. You know maybe sometimes we'll take a step back but we have trust that it'll be fixed. For me a good world would be like you have the trust that if tomorrow a new crazy technology gets invented you know someone in men's super giga death device whatever right you know in a good world your reaction to this would be oh I know the government the the good people the government have it under control I know all the smartest scientists will come out you know all the greatest uh military leaders will step up to the cause and will be heroic and we'll do the right thing this is what it would feel like a lot of it is institutional when I when When I'm here in America, and I love America. It's a great country, but but >> it's not really a good country in many ways. In many ways, when I look at America, people don't have the assumption that, yeah, it'll turn out okay. We'll figure it out. Like my dad, um, he he was a good man and he was a very he loved America. He was a very good man. And he got very ill, very very ill, and insurance wouldn't pay. So, lost everything >> and we were going to be homeless. um when we moved to Germany because my mother's German and you know he got shipped to the hospital and they took care of him. And they got him all the medication, all the, you know, treatments he needed. And he asked him like, "Well, what do I owe you for this?" And the doctor like, "What? You don't you don't know anything." And they just left. Like it was not it was not like kindness. It was like process. It was just boring. It was just boring. He's like, "Yeah, you're sick. You get the treatment you want. Goodbye." Like for me, a lot of the good world looks a lot like this. It's like, "Oh, you want to create beautiful art? Of course. Here's a bunch of beautiful art." You know, of course. Oh, there's a new threat threatening our country. Yeah, don't worry. The military's on it. Oh, some new crime is being committed. Don't worry, law enforcement will take care of it and the courts will be just. That's for me a lot of what a good world looks like. So that when and for technology, it will be obvious that of course we're going to regulate it well. So it'll be for example, social media will just be a good for people. It will obviously we will regulate recommended algorithms in such a way that they're not addictive. will in fact make them so that they you know don't push people into down negative spirals. We make sure that they're encourage people to get off their phone and spend time with their friends. We will have lots of beautiful architecture and beautiful art and people will spend time together in real life and with their kids and all that kind of it will be in some sense boring. It will just be I'm very inspired by the enlightenment the the period in western history where a lot of including you know the American constitution really got many of its ideas from and what I really like about the enlightenment is that sometimes critics of the enlightenment say oh it's this rationalism you know they're trying to be so rational and reduce everything to rationality but this is actually not true the enlightenment thinkers much preferred the the use of reason they wanted to be reasonable So for me, a good world isn't perfect. It's reasonable. >> That's part of the thing I'm trying to not have an intuition of what is reasonable. And it's not a formal thing. A a good world is not formalized. It's not a high modernist, you know, communist, you know, mechanical thing. It is for me in a good world, things still go wrong. Sometimes people make mistakes. You know, maybe a court made a wrong choice, you know, maybe, you know, there was a there's an accident. Like things like this still happen, but every time you look into it, you'll be like, "That's a reasonable mistake." Like you see why this would have happened and you know that we'll fix it. Like you know there's someone in charge and they'll be reasonable in the way to fix it. This is for me what a good world looks like. That when you know super intelligence is on the horizon that people will be reasonable. The scientists will go to the government and say hello we are the scientists and we think this and this will happen. And the government will be like yes this is a very reasonable thing. We're going to talk to all our international colleagues and figure out what the best way forward is. I'm not saying they're going to get it right on the first try, but along the whole chain, everyone will just be like, "Oh, yeah, that makes sense. That that's a reasonable thing to do." That's kind of how I think a good world would look. >> Is um AI exists today compatible with that world. >> As it exists today, it is not fatal to this world, but it is not AI in this world would not look like AI today. >> Interesting. Tell me why. because it's not aligned with people. AI most of AI by like you know dollar volume is like Tik Tok algorithms pushing slop to children to give them eating disorders. Like that's the primary mech like it a lot of the funding that has gone into modern AI comes from companies who built up massive war chests from you know pushing eating disorders onto children for decades in the 2000s. Like that's where a lot of the money the cash came from originally. the funding for deep learning and neural networks so largely it was for recommener algorithms and only later turns out was useful for other things as well. So no, I think that for example, in a reasonable world, if a company builds a product that, you know, for example, creates massive anxiety in teenagers, they wouldn't make any money. They would go out of business and maybe go to jail for that. And if someone made a product that makes, you know, uh, teenagers learn good things and make new friends and be protected from predators and, you know, grow as people, they should make a lot of money off of that and they should be, you know, rewarded for this. And we all, you know, give them medal and say good job, you know, like I I I don't I think this is possible. Like it's it's it's funny that people sometimes don't can't imagine that. Like obviously we could just make a social media that's nice. Like obviously we have the technology. You obviously if we designed the algorithm to be empowering to people and good we could. It's just they would make less money. You will make less money. You know you can make a less addictive you know cigarette by removing the nicotine. It will be less addictive and it'll be less harmful. These things are related in various ways. So I think you can make very pro-social products with AI and with social media and all these other things but the and the bottleneck here is not technology. I'm not saying we need to invent some new technology we haven't invented yet. It is much more about regulation. It is much more about philosophy. It is much more about humane design about what do we reward with our markets. Do our markets reward whatever optimizes whatever reproduces the most or reward the thing that give us actual value. This is the core problem of optimization. If we could just define a magic number that contains all goodness in the world, well great, make the number go up. But every number we have obviously is not that. GDP isn't that. You know, GDP is definitely correlated with a lot of good things. I like having a lot of GDP. It's very nice. But at the same time, you know, like it's crazy. Like, you know, I know especially like kids younger than me, do you know that like they don't dance at clubs anymore? >> It's wild because they get recorded. So, if you're like 17 and you're at a club or something, you dance and you approach a girl awkwardly and you mess it up, you get recorded. That's awful. This is this is terrible. And this is a So, from my perspective, from a like, you know, to be like an autistic economist, a massive harm was caused here. Like, like astronomical harm was caused to this teenager. From my perspective, I think the 17-year-old was caused a massive injury by this fact. Who's paying for that? Like, who's who's addressing this problem? who's making >> literally just cut a check for 17 billion dollars to end all their lawsuits. But >> exactly. >> Yeah. >> But like that doesn't solve the problem. None of these things solve the problem because the problem isn't technology in a sense. It's more like what do we do with it? >> What do we want our society to look like? What are the institutions, the norms like like this is a small example in Japan? Every time you take a photo with your phone, it makes a noise. You can't turn it off. So that if people in public take pictures of each other, they at least know about it. >> Yeah. >> Extremely small, right? But this is obviously a good thing. Obviously, you don't want strangers taking pictures of you in public, you know, or at least want to know about it, right? And so this is an extremely small just social thing that you can change. And I think many many of the things we really need are like this in the history of like again like the enlightenment. What I thought was so beautiful, one of the things I find a beautiful enlightenment is that the people pushing the technology and the people take pushing statecraft were in a sense often the same people or they were handinand they took inspiration from each other. They were talking to each other constantly. For them the design of a state of an of a society was another engineering problem. It was another question of how can we create a wonderful nice society for people. And what I feel nowadays is kind of like kind of what you were saying earlier is like a lot of people they call themsel like techno optimists are really civilizational pessimists. They just believe that the only way to improve the world is through technology rather than how we live our society, how our norms work, how our culture works, how our laws work, our institutions. Even so when I ask people what are the top 10 things that they're annoyed about, very rarely are those top 10 things technological things. almost always is like, "Oh man, you know, I'm upset about politics. I'm upset about healthcare. I'm upset about um you know, all these kinds of like so I'm upset that I can't see my friends enough. I'm upset that, you know, whatever, right? And I think we should take this seriously. I think this is, you know, I'm a technologist at heart, but what I care even more about is making the world a better place." And turns out technology isn't the bottleneck. I would love to get to a point where technology becomes the bottleneck, you know, where we've solved culture so well and we have such a beautiful government that now all we need to do is to invent better technology. I would love to live in this world. >> Yeah. Uh that's a really good point, man. That actually really hit me. Uh I'm very glad that you took the time to walk us through that. So, um one as a a way point, do you think that AI and capitalism are like a toxic pair? I'm personally a big fan of capitalism. I think capitalism is one of the best ideas humanity has ever had in moderation the same way as any ideas. There's a thing that I think sometimes gets forgotten is that markets are great. I love free markets. I love private enterprise. But it's one tool in our tool belt and we know for which problems it's good and for which ones it's bad. You know, if I gave you a really good power drill, you know, you use it to drill in a screw, you might be like, "Wow, this is the best tool ever." But it's not it's one tool. You know, if you need to saw a board, you're not going to use your drill. You're going to need a different tool for this. Markets and capitalism are very similar. Capitalism and markets solve many problems really well. You know, if you want a cheap, you know, commodified product delivered at a affordable price directly to your doorstep. By God, do I have a system for you. And capitalism and markets have brought a massive amount of good to the world all, you know, over the last, you know, hundred years. A lot of bad, too. But fundamentally, markets are optimizers. They're like reinforcement learning. They have a number that they make go up. And if that number can go up by ways you don't want, markets will do it. And this is why we have regulation. >> In all of history, we've always paired to these things. It's kind of like it's kind of like MMA, you know? Like I love MMA. I think it's so fascinating. I don't do it myself, but I I think it's such a fascinating thing because you have like the most dangerous people to ever live in history. the most dangerous warriors to ever live, you know, fighting, you know, to such an incredible degree. And they don't kill each other, you know, they can keep competing because we have rules. You need a lot of rules to allow this competition to be possible. You need a referee. You need a lot of rules to make sure that they can compete the next day, that they, you know, next match that they can come back. And markets are very similar. And you don't regulate a market. You just get monopolies. you just get some corporations, you know, hire private security to, you know, break your kneecaps so you don't buy your product, right? That's no good. To get a competition, you actually have to have a lot of rules. And I think this is beautiful. I think if we make, you know, good rules and we have a good referee that is not the market, you know, that is outside the market, such as the government, we can make beautiful things. I think this is an example of just us failing to do our job. Like it's kind of like imagine someone in MMA found a new chokeold that makes them win but also kills the other guy and we just don't ban it. Like if we don't do that then the sport of MMA stops existing and so do all the people who participate in it. And there's a very similar thing here where this has happened throughout history. Corporations find new exploits. They find new edge cases that we didn't think about before. You know they find new ways to pollute. They'd find new ways to harm people, to scam people, whatever. And then we react to that. Then we're like, "Wait, hold on. You can't do that." You know, before nuclear weapons existed, they obviously weren't illegal. And then once we realized that they were they were possible, we really quickly made sure that they were illegal for the market to build because otherwise the market would be happy to build nuclear weapons. Could you imagine, you know, how much big tech companies would love to build nuclear weapons and sell them to the highest bidder? Of course they would. They would love that. So it's always it's a design problem and it's an evolution process. It's a it's it's an iterated process over time. We're never going to get it right on the first try and that's okay. That doesn't mean markets are fun. Well, we should throw out all markets and move to communism. Neither does it mean we should have no regulation, just do markets. It's just we're trying to solve a really hard problem which is how can we get an unaligned thing which is the market like an the market quote at large does is not a good or a bad thing. It's like an alien. It's like an AI, you know, it's like an AI. It's like a reinforcement learning AI. And how can we get it to do things that we want? And that's hard. We've made a lot of progress on it and we just have to keep iterating. Mhm. If you were going to regulate AI now to make sure that we don't cross the point of no return, have you thought about what the specific policies you'd want passed are? >> Yes. Um the there's two major things that we think should be passed. The first and foremost is you should criminalize the creation of super intelligence. This is a thing you can just do in law. You know, it's illegal to murder someone. It's illegal also to attempt to murder someone. Even if you fail, you still can't do that. >> Similar thing with nuclear weapons. Even if you attempt to build a nuclear weapon, even if you don't succeed, the men in black show up at your door, you know, and this is good. So, this is a thing you can just do, you know, and it's much easier. You It's quite easy often to tell the difference between, you know, a company that's building trying to build super intelligence and it's not trying to build super intelligence. It's not perfect, but it's a start. So, this is the first common sense thing. It's just if you're literally advertising, I am building super intelligence. I just raised a billion dollars to build super intelligence. That's illegal and you can't do that. Then the second thing is how do we prevent us from getting too close to super intelligence and for this fundamentally because again if a super intelligence is built even once even accidentally it's too late. So we have to regulate well before super intelligence so we have enough buffer zone. It's kind of like imagine we're going 120 m hour uh in thick fog and we know somewhere there's a cliff but we don't know where. >> Yeah. >> My first suggestion is pull over. Before we argue about the speed, let's first pull over. So, first criminalize, you know, super intelligence and then regulate the precursors. Have a list of precursors such as ability to self-reproduce, um, task horizon length, things related to genetic autonomy and so on. and monitor those very carefully and be sure that any credible large experiment that pushes on these frontiers is registered with the government is that has oversight and that we can decide no you can't do that that's too risky because luckily at the moment building really frontier AI that pushes towards AGI or ASI is unbelievably expensive you know we're talking billions of dollars so there's actually not that many actors that this law would even affect this wouldn't affect 99% of companies you know there would be a very small percentage of mega mega huge tech companies that would be affected by this and for them yeah we should obviously have oversight of them the same way we have oversight of like Loheed Martin or you know nuclear weapons facilities and so on >> what about open source AI >> open source is a very tricky one in the sense that I love open source I built some of the first open source large language models in the world in fact and led a group called the Luther AI I've been I grew up in the open source world. I think open source has brought so much good to the world. I think Linux being open source and all this open brought so much beauty and good into the world. But for me it's not ideology. It's a question of you know should the blueprints for the F-35 fighter jet be open sourced? >> Probably not. >> Probably not. That seems like it would make the world a better place. You know should blueprints for you know bioweapons or you know nuclear bombs be open sourced? Probably not. You know, and here's a very similar thing where if we get to the point that there are open- source systems that are super intelligent or that can be made super intelligent, which I think we're pretty close to, then I think there's probably no going back in this regard. You know, if there's like one system that's almost super intelligent and it's only on one server in one country, you know, maybe we can delete it, you know, in the maybe. But if there are open- source billions of things around and we know they are super intelligent or like you know two steps away from super intelligent it's probably too late. I think the action that would be needed to for example you know recall quote unquote an open source model are not feasible. >> All right. Bill Gates just said that he now thinks AI is dangerous. What happened? What made him change his mind? >> I'm not sure really how much he changed mind. Of course I don't know him. I think, you know, he's a he's a very, you know, he he knows a lot of things. He's a smart guy. I'm sure he's seen a lot of scary stuff. But even all the way back in 2023, Bill Gates actually signed the Center for AI safety statement which said that the mitigation of the risk of extinction from AI should be a global priority. This was signed by Bill Gates, Sam Altman, Dario Amade, Joffrey Hinton, like lots of really, you know, big hitters. So, this was all the way back in 2023. So I think, you know, again, I don't know him, but I think Bill Gates has always known of some of these risks and just has now been trying to maybe he's now trying to message a bit more about them. I personally wish he would have gone a bit further in his letter. I think it was, you know, it did acknowledge some of the risks, but didn't go all the way to super intelligence and extinction risk, which is the thing that I am really so concerned about because in a sense, I don't think we have enough time that we will see massive job displacement. Like I think we will just see super intelligence in the next couple of years way before massive job displacement even happens and it's already too late. I kind of wish we were in a world, you know, where super intelligence is 20 years away. Then job displacement will become a huge problem. >> But I'm not sure we're going to get to that point even. >> That's wild. That's wild. >> Okay. Um I want to see if I got your base assumptions right. If I've missed anything, let me know. But this has been very helpful. Uh, these are loosely in order, but um and there's one that I have a follow-up question on. Okay, so super intelligence is definitionally you you um maybe didn't go quite that far, but super intelligence is adversarial. So knowing that once you make that, it is going to compete and you're going to have problems. Therefore, uh super intelligence will consider us a distraction or impediment to their goals. And so we have to worry about being the very thing they're competing against and we will lose. um unknown risk is evaluated at near peak danger levels. Meaning if I don't know the outcome of this, like you said, first pull over if we don't know where we're going. So anytime we terminate in I don't know what AI is going to do or I don't know what the super intelligence is going to do here, pump the brakes. >> Obviously there's spectrum here. Um as with any complex strategy, how do you deal with known unknowns or unknown unknowns? How do you calculate the probability that a country is going to invade you tomorrow? This is actually a problem there's a lot of literature on. This is a thing that you know the CIA and you know various military agencies have spent many many decades working very very hard on. One of the recommendations for example the CIA makes is that always word your gut feelings in numbers. So this is the thing we didn't talk about. People often get annoyed when people say like oh I think there's a 20% chance of things going well. They're like that's not math. And I'm like dude yeah they know I know it's not math. The reason you say oh 20% is not because I'm trying to make a mathematical statement. It's because there's there's a famous study, I think it was from the CIA, where they found that analysts who said likely >> or unlikely actually meant wildly different things. Like some people when they said likely, they meant 90%. Other analysts when they said likely meant 30%. >> And so this was extremely confusing because unlikely likely doesn't mean anything. So if you say like 90% it's still a vibe, but it's a much more concrete vibe, right? It's a much more when I'm communicating to you. If I say 1% I'm trying to communicate not oh it's definitely 1% and not 1.2%. That's not what I'm saying. What I'm saying is it's on the order of 1%. It's like given my beliefs about the world and you know looking at my models of reality which you know you can evaluate to how much you believe my models of reality this outcome has on the order of this percentage of outcomes. Of course it's based on my models. What else is it supposed to be based on? Right? Like obviously you have views of the world. You know if I asked you how likely do you think it is that you know this thing will happen in the future how do you evaluate this well you use your model in your head what else are you going to do you know so it's a very similar thing and like I try to make my model explicit like with these base assumptions I try to make it pretty explicit this leads to this leads to this leads to this therefore high probability so if all of these statements that we've said so far were like you know a billionth of a percent probable I wouldn't be worried but I don't think that's the case I think if they're all like you know higher than 1% and after their conclusion are still more than you know after multiplying them all together we're still above 1% and the outcome is extinction hell no you know if the outcome is like oh it's a one in a trillionth I'm like sure but that's not where my numbers come out >> right makes sense okay um financial incentives power incentives and lack of strong regulation have allowed us to keep racing forward despite the dangers >> yep >> uh for AI to work it requires the AI to have a will of its own. >> Depends what you mean by will. I think it's >> it's going to be an optimizer. Now that I know the language you use, >> this is how I would describe it. It will have to optimize for goals of some kind. It will take actions. It will make plans. It would it will optimize towards objectives. >> And the sociopathic optimizer, as you described it earlier, is the problem. >> Yes. >> Okay. AI, >> there's some other problems, too, but this is really quite the heart of it. >> Okay. AI, it's too complex to imbue AI with morality. >> Yeah. We have no idea how to do it. >> Okay. Uh, life can be improved by things other than technology. >> Yep. >> Uh, it's possible to get China on board. >> Yep. >> And humans are not magical. This is the one that I need more clarification on. It really hit me when you said it, but I wanted to better understand. You were trying to explain something about AI by saying humans are magical. You need to understand that. >> What What do you mean when you say humans aren't magical, and what does that tell us about AI? >> There is a very fun um thing. There's a Wikipedia article called the AI effect. I I recommend looking up the Wikipedia article. It's quite fun. Which is that throughout the history of AI, people would always define like, oh, we'll know we have real AI when X. And then once X was accomplished, everyone like, oh, no, that's not real AI. That's not true intelligence. So, for example, back in the day, people thought that the thing that makes humans unique, different from all animals, is that we can do math. And well, obviously computers can do math >> very well. >> So, very well. So, so then they're like, "Okay, no, no, that's not what we mean. What we mean is actually the thing that makes humans magical is that we can recognize images or whatever, you know, and then you're like, or they can play chess or, you know, and obviously none of those things were the thing." And so there's a there's like this like rising tide, rising waterline or or like you know, like being packed into a corner of like this this very funny cartoon. I think it was maybe from Rick Curtzwell or something where you have this guy frantically writing paper like only humans can do only humans can play chess and like strikes it out like only humans can draw pictures strikes that out and like and like his whole room is covered with these paintings of him and like only humans can do uh uh uh you know and what I'm saying about this is that humans are physics. We are in fact part of physics. There's nothing magical going on. It's complex. Our brains are complex but it's not magic. It's it's a it's a physical process. there is some computation happening in our brain of some kind. You know, it's being, you know, it's using some weird squishy, you know, elements for the calculations, but fundamentally some kind of calculation is going on. It's not magic. You can make other things do similar calculations. >> So, it's a way of explaining we know intelligence is possible without some sort of magical element. >> Exactly. >> It makes all the sense in the world, dude. This is crazy. Um, this has really been amazing researching you and sitting across from you. I really, really appreciate it. I think your messages come in loud and clear. If there was one thing that um people may not realize, like if you were sitting across from Xi or somebody who you're like, I know they're convincible. Is there one final thing that hasn't come up today that you'd want them to be aware of? >> I have different answers for Xiinping. I do for people listening and so I'll give two answers. And so honestly, the number one thing I have found is the grown not written. People don't know this. when I talk to politicians is the first thing basically out of my mouth is explaining that this is not normal software. It is not written. It is grown. We do not understand how these things work. I we did bring this up but I'm repeating it again because this is just so important to understand. The people at Openai do not know what is going on inside of their AI. >> Dario Amade the CEO of Anthropic thinks he we understand maybe 3% >> oh my god >> of what goes on in our AIs. And I think that's even optimistic. >> Whoa. So this is one of the most important things to understand. We don't know. We they don't know. No one knows. This is very important. And for the general public and everyone listening, I think the very important thing is is that it's not over. And even though it's sometimes unfashionable say to say this these days, we do still live in a democracy. And your voice is important. If you want something to happen, the way I like to think about civics is that our job is to work with our politicians to help them understand what we care about and what can be done. So do your part to help bring these issues to awareness. Go to controlai.org, contact your lawmaker, tell them you're concerned about these issues, help educate people, keep having the conversation. This is how we humans solve all these problems, you know, together. >> I love it. Where can people follow you? Um, I'm you can find me on Twitter, you know, and now X, sorry, uh, at MP Collapse. You can find me obviously controlai.org. And also, if someone wants to engage even deeper on some of these things, I actually run a volunteer group called Torchbearer, torchbearer.com community where we commit to spending at least two hours a week to try to build that reasonable world. >> I love it. All right, everybody. If you haven't already, be sure to subscribe. And until next time, my friends, be legendary. Take care. Peace. If you like this conversation, check out this episode to learn more. >> AI is easily the greatest discovery humanity will ever make. This is not an easy game to win or to start, but this is not going to be stopped. >> AI is sucking up all of the liquidity and it's causing a problem in crypto. And now.