Cory Doctorow on the Big AI Lie | Downstream with Michael Walker

The reality is we have full employment for every human being who will live for the next 500 years because we're going to have to do [ __ ] like move all the coastal cities 20 kilometers in land because we are not living in a fantasy novel. We are living in science fiction and in science fiction the second law of thermodynamics is not optional which means that when you put enough therms in the ocean the ice caps melt and when the ice caps melt the seas go up. So, we are going to have to deal with billions of people who have been made refugees. We're going to have to deal with a series of zunotic plagues. We're going to have to deal with food crises. We're going to have to deal with um flooding and wildfires and more and more extreme weather events. And we're going to need every hand we have. There is no fully automated luxury communism on our horizon. Maybe in 500 years, our distant descendants will look back and say, "Well, first of all, boy, was it a [ __ ] mistake to put all that carbon in the atmosphere to make chat bots. Second of all, finally, we've got all the cities moved 20 km inland. We figured out how to resolve all of the the zunotic plagues. Everyone's been resettled. Now, we can start working on that fully automated luxury communism thing. Comrade, AI is set to take all of our jobs. It's set to irrevocably transform our interpersonal relationships. It could even kill us all. Those are claims that both supporters and critics of AI now regularly make and they're claims that I take seriously very often over on Navara Live. My guest today on Downstream though, as I replace Aaron Bastani, who's off on paternity leave, thinks that this is a mistake. He thinks that by amping up the capacity, the ability of artificial intelligence, both critics and supporters are doing tech CEOs a favor. Cory Doctoro is the writer and author. I mean, famous for many things, but especially his concept of inshitification, um, which explains how the internet got so much worse. Um, he's now turned his attention to artificial intelligence in a brilliant book, The Reverse Centaur's Guide to Life After Ai. I'm so excited that I get to stand in for Aaron Bastani for this episode um, and talk to the fantastic Corey Doctor. I've really lucked out here. Cory Dr. Oro, welcome to Downstream. >> Thank you very much. Pleasure to be on. >> Um, your last book, Initification, um, massive hit. Also a mainstream hit here. It was nominated for the Financial Times business book of the year. Um, I don't know if you expected that sort of as a young sci-fi writer that you'd be >> very exciting. I also got in the OED, uh, which like talk about a bucket list item. I have a friend who got a much rudder word in the OED, Dan Savage, who got uh um uh the name of a US senator, which he used as a euphemism for a very disgusting sexual fluid. Uh and he got that Santorum. He got that into the OED as an official synonym for um I'm not even going to say it's it's it starts with a frothy m mix of lube and goes downhill from there. Oh my god. and and it is an official OED synonym for uh Santorum, also a conservative senator who opposed marriage equality. I think in shitification was also word of the year. Several one of them. Yeah. New scientists made it UK well made in shouldene the UK word of the year. McQuary made it the Australian word of the year. The American Dialect Society made it the American word of the year. My dad was very disappointed it wasn't the American Dialectic Society, but there we go. >> And I mean, now we're talking about it. you've already done an interview with us onification sort of a Navara IRL and we were just saying before we went live I wasn't there because I was doing my own show but I could hear how you were working the crowd very impressive stuff but in case anyone hasn't seen that very briefly what was the initification idea >> so I've worked for this nonprofit called the electronic frontier foundation I'm going into my 25th year on trying to get people to care about tech policy and it's really hard because it's abstract things that are in the future and people care about concrete things that are happening now. And so you have to come up with gimmicks, framing devices, simileies, narratives. I write science fiction novels. And it turned out the dirty word was the thing that got people to engage with it. So I coined this term in shitification to both describe a process of platform decay, how they go bad, but also to hypothesize a theory about why they're going bad now. Because I'm a materialist. I don't think we got hit by a platform destroying meteor in the mid teens. I think that like the conditions around the world changed such that the worst ideas of the worst people made the most money. And so wherever there was a factional dispute within a firm or wherever someone was tempted as a decision maker to take their platform in a worse uh direction, they did. And so we end up with the inshitta scene when everything's turning into [ __ ] because of policymakers creating an inchiditoenic policy environment where they took decisions that had the foreseeable and foreseen outcome of creating this millure. And the process is pretty straightforward. Platforms are first good to their end users and they lock them in. That's stage one. Stage two, you're locked in. It's hard for you to leave. So they make things worse for you in order to provide benefit to businesses which they also lock in. And then having locked in those businesses, they withdraw value from them as well. The idea is to find an equilibrium where you have like the mingiest kind of homeopathic residue of value sufficient to keep users locked to the platform, businesses locked to the users, everything else withdrawn and aortioned among executives and shareholders uh and the platform sort of zombieing on as this thing that we hate but can't seem to escape. And I think one of the things maybe for a Novara audience that is useful to point out here is that this is an environment in which you have like kind of billionaire on millionaire violence, right? Where you have fairly large and powerful firms being victimized by even larger, more powerful firms who establish themselves as intermediaries. And it's kind of a rebuke to a liberal notion that uh if you're not paying for the product, you're the product. as though being treated well is like a customer loyalty perk as opposed to like a thing that is determined by whether a firm that is just you know has this one objective function which is to maximize its profit can figure out that they can maximize their profit from you even though you're spending money with them and maybe later I think we're going to talk about John Deere but you know if you're a farmer who spends $600 or $700,000 on your John Deere tractor you still have to pay $200 after you fix your own tractor for the John Deere repair person to come out and just type the unlock code that initializes your repair, you having made it. It's not a free tractor, right? You didn't get the tractor for looking at ads, right? You paid for the tractor. You're a captive audience. It's a new kind of tenant farming where you're a tenant of the agricultural equipment and not the land. And it's essentially, again, we'll move on from initification onto AI very very soon. But just to sort of introduce your work, I suppose it's almost a conventional argument about monopoly power, isn't it? So it's to say they get you in with a useful product and once they've got a load of monopoly power it's difficult for you to leave if you want to stay in touch with your friends for example then they can degrade the product because you've got nowhere else to go. So they can just blast adverts at you and make it full of pop-ups and make the search worse and worse and worse because they've got 90% market share. In a way was it >> you know accepted and embraced perhaps by sort of the mainstream business press because it was like almost a conventional economic argument made very persuasively and made very well for the 21st century. So I I well I'll I'll thank you for saying it was made persuasively and well. I I I will stipulate that there's aspects of this that just recapitulate the wisdom about monopolies, but I also am drawing out some of the contingent factors that both make tech shitty and also make it resistant to initification. So on the one hand, the most basic lockin system that tech platforms have going for them is what's called network effects. Uh this is an economics term. It just means something that gets more valuable the more people there are using it. Uh social media is a really good example. you know, you join there to find some people. The more people there are there, the more likely the people you're looking for are there. Once you are there, you become a person someone else wants to see. So, you have this network effect that has these corolleries, which are things like high switching costs, which is to say you have to abandon all those people. If you leave the platform, and high coordination costs, which is like you love your friends, but they're a pain in the ass. And you know, you all agree that you hate Facebook, but you can't agree on where to go next. You can't even agree what board game you're going to play this weekend. And so, you hold each other hostage. So that's a unique contingent aspect of that kind of tech. And then on the other hand, you have these contingent aspects about um how tech has some initification resistance built into it. Some of that is a labor story while tech workers have effectively no unions. I mean there's some you know shout out to tech workers coalition and so on. But they they have like infinite decimal union density. But the part of the reason for that is that they were in such high demand and they were so productive from an economic perspective in the sense that signing a new engineer up on the payroll of a company like Google more or less added a million dollars to the bottom line of Google. So >> they got great salaries and bean bags. >> Yeah. And like kombucha surgeon who'd freeze your eggs so you could work through your fertile years, whatever you wanted. So you had a lot of labor power and a lot of those people came to the profession through a kind of liberatory experience with technology, right? like they they found in the spare bedroom with the bulky beige PC a world they never suspected existed that not only gave them a lot of economic opportunities but maybe a way to you know name the phenomena of the world that they never dreamt was possible or even words to describe themselves that they never thought was possible. Uh and and so they want to share that benefit. Um I call them tron pill because they want to fight for the user. Uh and you know you meet these people all the time. ask someone who uses Linux about their computer and then try and get them to stop talking about their computer. Uh, and so, you know, these people had market power, labor power, and they had an ethos that despite all the things we say about tech bros was very common, remains common within the field. You also have a unique characteristic of computers, which is that the only kind of computer we know how to build is something that's formerly called the touring complete universal vonoyman machine. a lot of CS jargon, but what it means is that every computer can run every program, which means that every inchificatory gambit assayed by a firm invites a disinhitifying program that can be costlessly instantaneously distributed to everyone who's affected by it. So, you know, you put more ads in the platform, someone makes an ad blocker. And uh it was the advent of what's called an anti-ircumvention law that I think we'll probably get back to in this interview, which is a law that prohibits modifying technology with the manufacturers's permission that choked off that new market entry and basically created this this one-way system where you could disrupt an incumbent if you're Facebook by creating a an interoperable program, something that that talks to the existing programs, but no one can ever do it to you. When you do it, it's progress. when someone does it to you, that's piracy. And so you can just lock users in. >> And so that's a unique historic contingent aspect of it as well. And then because computers are so flexible and because users can't flex them, only firms can. You can do a lot of stuff at high speed. You can do a lot of automation. And this is now getting into AI and the main conversation we're going to have today I think which is that you can like twiddle the knobs on the back end to change the characteristics of the platform on a per user per session basis in a way that's just not practical in the physical world. You know, um, one of the things I write about in in the AI book is the fact that, uh, nurses who sign on for a shift using a gig work platform in the US, which is the predominant way now that contract nurses find their their shifts, they have their credit card data checked through a data brokerage by the platform before they're offered a shift. And if they're carrying a lot of uh, debt, their wages adjusted downward as a kind of um, desperation premium. And like you know if you've listened to the songs of Tennessee Ernie Ford, you know that there were coal bosses in the 19th century who are perfectly capable of conceiving of this Whis. They just couldn't afford the army of Pinkertons to follow around the coal miners and figure out who they could nickel and dime down on the shift. And they couldn't afford the army of guys in green eye shades to adjust the ledger all day to change the pay packet for every worker. And so you have these like unique contingencies that both cut for and against inshitification and monopoly is like a really serious problem here. uh but it's the way it expresses itself in this moment is distinct and you know in inification I I devote a chapter to varus's idea of technofudalism and I think he's making a an argument that has a similar structure where you know the characteristics of feudalism and economy extra organized around rent extraction to the exclusion of profit or rather you know to the as a triumph over profit um that that has come back with its own unique digital characteristics in this moment of what he calls cloud capital. But um you can understand a lot of what's going on now by thinking about feudalism in the same way that you can understand a lot what's going on now by going back and reading Ida Tarbel who's the woman who felled the empire of John D. Rockefeller by writing a series of investigative journalism articles in a nationally syndicated magazine as revenge for him having ruined her father who was a Pennsylvania oilman before going on to becoming the most important suffragist in America and giving all the barn burning speeches that led the movement to victory. She was amazing. >> She sounds amazing. Um, let's officially get onto the book. Yeah. So, your new book is about AI. Um, >> it's a critique of AI in a way. It's also a critique of AI discourse at the moment. I suppose maybe we could start by asking you what is going wrong in the way we talk about AI at the moment. >> Yeah. So the original title of this book was the reverse centtor's guide to criticizing AI. Um, my editor thought life after AI sounded better, but it really is about being a good AI critic, which is about understanding the material factors that give us the AI bubble, attacking those material factors preferentially, and uh figuring out what ideological characteristics arise out of the material circumstances, but not mistaking this as as a merely ideological project and understanding that it's like a material project, too. And and very crucially, if you do those two things, you will not fall prey to a thing that the STS scholar at Virginia Tech, Lee Vinsel, calls crit a hype, which is when you uh repeat the most outlandish claims of a firm, but then at the end in parenthesis add and that's bad. You know, the the kind of practitioners of this are the people who believe in Zubov surveillance capitalism, which is basically the belief that not only these companies spying on us, but that they've perfected a mind control ray through surveillance data and uh that they can make us do anything with it. And so you run around and say, "Oh, they've ended free will itself through their big data machine." And then they go out to sell ads and they're like, "Well, you're asking me why you should pay a 40% premium to advertise on Facebook." Don't ask me. Ask my critics. They will tell you, "I work for Cyber Rasputin and he has succeeded where Mesmer, MK Ultra, pickup artists, and neural linguistic programming weirdos failed. He's built the mind control ray. We can sell anything to anyone. That's why you should give me a 40% premium on ads on my platform." You're helping them uh sell ads, right? You're not You think you're criticizing that you're helping them. There's a lot of this in AI uh where we talk about how not that your boss is a credulous adult who's infinitely horny to replace lippy workers with pliable chat bots and therefore can be convinced to fire you and replace you with a chatbot that is manifestally incapable of doing your job. And instead we run around and we say the AI is going to take our jobs which is not the same thing. And if you go out there and say the AI is as good of my job as I am, you help the AI salesman sell your credulous dull to the boss on the chatbot that's going to result in you getting fired. And you take your natural class allies, which are the people who benefit from the things that you do, and you turn them into your class enemy because the reason people want to see a radiologist is to make sure they don't have cancer, not to feed the radiologist kids. And if you're like, "This is as good as I am at my job, but it works at 1 millionth the price." People are going to say, "Look, I feel for you." But, you know, I'm here to not get cancer, not to pay your mortgage. We'll figure something out for you. And so we have to be we have to be smart to be critics of this disgusting harmful anti-worker fascist project so that we attack it at its material roots and detonate it as quickly as possible rather than letting this bubble run its course to the point where instead of just 35% of the stock market being seven AI companies, it's half or 3/4. And when that bubble bursts, we do such punishing austerity that Prime Minister Farage is an all but foregone conclusion. Um, now I think lots of the coverage I do over on the bar alive would probably fall into a category you'd describe as critic hype. Um, but I want to park that actually for a moment. I I literally think it might kill us or not just take our jobs. But I'm going to park that because there's a lot where I think we do agree and we can begin on that to sort of bring out your your ideas to the audience. We can put that to one side for a moment. So the first thing you mentioned there was the AI bubble. >> Yeah. >> And um, our audience are always fascinated to hear about the AI bubble. Um, everyone's talking about it partly because it's so big, right? So, I'm just going to go through um a few of the numbers here. Um, so this year big tech companies are expected to spend over $800 billion on AI related capital expenditures. >> That's now actually 1.4 trillion so far this year >> in the just today >> in the in the whole world maybe. Yeah. >> Right. Yes. So, I think this is Goldman Sachs are saying $800 billion in the US this year, right, from those big tech companies. Maybe it's CH if you add China it sort of adds that on. Um, and then they're suggesting that it will hit $1.6 trillion dollars per year in 2031. Um, that's the big American companies. Um, which, you know, for our audience is the equivalent to onethird of of UK GDP. So all of our collective efforts, right, >> for for months of the year, we're just going towards data centers. That's essentially what's going on. Um, >> oh, that might be data centers and not including training and opex then, >> right? Potentially. >> Um, well, this is it. It said capex for >> well then it's not including opex right yeah >> um yeah that would make sense uh US tech stocks now command 40% of the US standard and poor so 40% of the main stock market index >> is that the magnificent 7 or all tech stocks cuz the magnificent 7 are 35% the last >> so this is something which is the the 10 it's the something 10 >> so they found they found three >> they found three more you add that to it in the periphery it's bank of America's big 10 >> okay Sure. Sure. >> Is what it is, >> right? >> Uh and um you know, obviously OpenAI expected to IPO around 800 billion andropic expected to IPO $1.5 to2 trillion. Um Nvidia stocks up a,000% since Chat GPT was released. They're now worth $5 trillion. Alphabet stock up 230%. Meta up 365%. This is all in a three-year period. This is this is crazy. If you if if you're looking at the chart that is the bubble and you know huge line goes up. Obviously, we don't know if the line's going to come down, but lots of people are predicting it. Um >> what's going on here? >> Well, the other thing to mention here is their revenue sucks, right? Actually, we should mention two more things before we talk about why this has a bubble. Because people might say, oh, they spent a lot of money, but maybe they're making a lot of money or will. So, the two things we should talk about is how much money they're making, what their unit economics look like. So, they're not making much money. depending on who you ask, it's probably about $50 billion a year all told in in pay in all the customers paying all the subscriptions. Uh they claim it's 60 billion, but that includes the 10 billion that Microsoft gave to Open to OpenAI and OpenAI gave back to Microsoft, which is like not revenue. Um speaking as someone who writes technothrillers about finance fraud, to call that an accounting trick is to do violence to the noble accounting trick. That's just not revenue at all. Uh and their unit economics suck. So, one of the things you may hear apologists for the AI sector say is, "Oh, Amazon lost money. The web lost money and then they made money." And that's true, but they didn't make money because they were losing money. Right? The fact that Amazon lost money and then made money doesn't mean everything that's losing money will someday make money. The reason Amazon started to make money, the reason the web started to make money is it had good unit economics, which is to say every time they signed a new customer on, they became more profitable. Every time that customer returned and used their products, their profits went up again. And successive generations of the web were more profitable than the previous ones. AI is the reverse. Every new customer acquired is more red on the balance sheet. Every time the customer comes back and uses it again, they lose more money. And every generation of AI loses more money than the previous generations. So that's like that's the that's the the reason that we should call this a bubble cuz it'd be one thing to spend a couple of trillion dollars to make several more trillion dollars, but if you just want $50 billion change back from a trillion dollars, I'll I'll take that deal. I I'd need to buy a bigger mattress to put the money under. But like I could give you back, you know, $50 billion all day long from your trillion dollar payout. I give you 60 billion back. That's what kind of a good-natured slob I am. So why is this happening? Leftists like to say, I think incorrectly, they like to quote that line that endless growth uh is the ideology of a tumor. And that's that they they compare capitalism to a tumor. I'm not going to say that capitalism can't be a cancer, but uh the reason people who run growing firms want to keep growing is not because they have an ideology. It's because they have material needs. uh firms that are growing are valued much more highly and and when I say valued I don't mean people like them more I I mean they they're worth more money than firms that are static firms that are mature because a share in a firm is a claim on its future earnings and so if a company is going to double its earnings next year and you buy a share in it this year you would expect the market to price in the fact that next year it's going to make more money than this year the corollery of that is when the firm ceases to grow, it becomes a mature firm and now it's grossly overvalued. Maybe maybe 10x overvalued, maybe even more. You see these flash crashes whenever tech firms reach kind of the limit of their growth or seem to be reaching the limit of their growth. And as you mentioned before, Google has a 90% market share. So clearly, it's some kind of limit on the horizon, at least as far as that market goes. And so, you know, Facebook in the first quarter 2022 gave an investor report where they said, "Oh, well, we anticipated this and so many American new user signups last year, but we got slightly fewer and the market took $240 billion off their share price in a day." Because even if you think oh well Facebook will find new ways of growing you're in uh it's Kane's had this idea of the beauty contest that the the mar stock market is a beauty contest where you are trying to pick not the most beautiful contestant but the contestant the other judges will judge most beautiful because then they buy shares and then your shares go up. And so if you think that everyone else is going to sell, you have to sell. The reason you have to sell is that while a firm is growing, while its stock is highly liquid, while it's highly valued, it can be used in place of cash in order to keep growing. So if you want to acquire a competitor or someone in your supply chain, which is a incredibly anti-competitive but common way of growing, or if you want to hire key personnel, you don't have to find pounds or dollars to do it. If you've got a growth stock, you just offer them shares. Now, pounds and dollars, you only get those from creditors and com and from customers and from investors. They're hard to lay hands on. Whereas shares in your own firm, you create by typing zeros into a spreadsheet at the headquarters. Uh they are an indogenous substance. You can't make your own cash at the corporate headquarters. They take you away in handcuffs if you try. And so once the firm stops growing, it stops having the fundamental unit of growth, which is free money that you can use to buy other people's companies and hire people. And everyone who works for your company who might be able to pull you out of this nose dive is suddenly much poorer because you paid them in stock and they all leave. And so there is a completely excellent material reason that companies that are growing want to keep growing. And this is where we get tech bubbles from. As these firms reached maturity, as they saturated their markets, well, first they started to claim that they were going to become each other. Facebook would become YouTube through the pivot to video. Google would become Facebook through Google+. That's uh it's a it's a good way of expressing your growth plans in as much as like we know how much Facebook is worth because it publishes a balance sheet every quarter. The problem with this is that when you assert that you're about to become Facebook, Facebook has a lot of resources that it can go out into the world and insist that you're not about to become Facebook because Facebook is Facebook. And so then they started making up imaginary stuff. Now the disadvantage of imaginary stuff is it doesn't exist. The advantage of it is if it doesn't exist and you just made it up, who's to say it's not worth a lot of money? So we get web 3 and crypto and Dows and NFTTS, the metaverse, one after another, these imaginary things, and they're just like running across the river on the back of an alligator without losing a leg, just inventing a new piece of [ __ ] that they're going to use to double their trillion dollar company, their $2 trillion company next year. And now we get to AI. AI is the biggest one yet. It's much bigger. The reason it's much bigger is partly because they're playing for all the marbles and partly because it's more real. AI is like as today the AI that we have today not the AI of the 1950s the 1960s 1970s ' 80s 90s the as this AI came about when some computer scientists about 12-1 15 years ago said we've done this very laborious way of getting machine learning systems to make predictions about the world where we build a world model we tell it how the world works and then we ask it to make inferences based on data that's really art. What if we just gave it a ton of data and we said find the uh the correlative relationships in the data. We won't know where the caus causal arrow goes, but you can just do what's called theoryfree inference, which you know we do in the world all the time. You can go to the doctor and sometimes they will give you a medicine that no one knows how it works. We just know we give that medicine to a person with that condition and then they get better. Now, scientists would like to know how it works, but if you're just like not dead, you can be happy with the medicine. And they're like, well, we can do a lot of theory free inference and see where we get. And it turned out they got a lot further than anyone thought. It's actually quite exciting. Like, from a computer science perspective, I'm a fake computer science professor. I I uh have an honorary doctorate from the Open University in CS and I'm on their faculty in the in the computer science program there. And so uh you know from a computer science perspective this is cool and it scales. You just throw more computing power more data at it. You get finer grained inferences that are more like something that's causal. But it also has limits. It just it doesn't know what it's doing. It doesn't have a theory of the world. Doesn't have a world model. If you ask an LLM to play chess, it'll tell you to move pieces onto squares that there are already pieces. because all it knows is how to do inference, right? It's like it's saying, "Okay, well, normally when this chess move has just been played, the next chess move that comes up most often in a winning game is this one." But because it doesn't have a world model, it doesn't know there's a piece on the board there. We used to build computers, like not even computers, protocomputers, out of electromechanical switches and valves that could play valid chess games. LLMs with not trillions, quadrillions, quintilions of time more computing power cannot play a chess game. Now, weirdly, you can use that as a coding assistant to write a valid chess computer. So, there are uses for it, which is one of the reasons that people are excited about it. >> Can just an LLM not play a chess game? >> No, it'll tell you to put pieces where there where there are pieces on the square, unless you have a hybrid LLM with a world model, which is what a lot of the um coding assistants have. So the coding systems, a lot of them now just have compilers where they just compile the code. They're not doing theory free inference. They're actually running the code. They're doing the you you could call it cheating. Really what they're doing is they're saying, "Oh yeah, it's horses for horses. We found this thing that we can do with a screwdriver." And we got to two things that need to be nailed together. And rather than just like trying to get the screw to hold these two things together, we've added the hammer rather than just saying like, "Okay, well, we're just going to have to spend like more and more money on larger and larger screwdrivers until the hammer has been completely obiated, right?" So, you can play a chess game with an LLM if you if you give it a world model that's like structurally made out of software, right? That that like describes how chess works. But if all you do is theoryfree inference with an LLM, this is what hallucinations are. But isn't the point of so this is a technical discussion now isn't it? Because my understanding I'm not even a >> a computer scientist computer scientist I'm just not a computer scientist. Um but my understanding sort of from an observer is that you've got two types of AI. You've got symbolic AI which is the one you're talking about where you sort of give it instructions world you you it will be sort of if this then that this is how this works. Now go and >> exist in the world according to those rules. >> Yeah. And then you've got big data machine learning which is to say we'll give you a very basic sort of neural network structure then we'll give you all the data in the world >> and you can kind of just work it out on your own how to do it >> right >> and one of the reasons why you know we're going to get onto this later but one of the reasons why so many people are worried about this model of AI is because it's getting increas in increasingly powerful and we got no idea what's going on because it's it's a bit of a black box. What I say is we are more surprised by how many things you can do with theory free inference than we and we are we are increasingly surprised by this. So let me give you an example from a conversation I had with a podcaster who is quite a believer in AI and he said uh and this was like the most guy AI pill person thing to say which was uh I can predict what my wife is going to say and so can an LLM. Who's to say that the LLM doesn't understand my wife? Just a shitty, terrible thing to say. Let's just signpost that for a minute here. But I think it's quite illuminating cuz it is true. I I have an autocomplete on my phone as do you. Uh it has a statistical model of the things that I'm likely to say back stop by like all the things everyone's ever said. And uh it can predict what I'm going to say and so can my wife because I repeat myself sometimes as do you. As does everyone else. Uh, but if my wife were to say something to me that she'd never said before, like, "I'd like a divorce," I could make a guess about where that was coming from. Because that's what understanding is, right? We could proceed to have a conversation that related to the specific contingent facts of our relationship, not the average of all the times one person said, "I want to get a divorce to another person," which is the best the LLM could do. Right? And that's the difference between understanding and theory free inference is that understanding proceeds from a theory. Now to to put a button on this, it turns out you can predict a lot more of what someone is going to say without understanding them than we thought. We are a little more predictable than we thought. The world is a little more predictable than we thought. But it what where it matters is when it goes wrong. AI is great if you don't care how things fail. If you only care how things work and you don't care how they fail, AI is amazing. That is the logic for building a car that has no brakes in it, right? Because if all you care is how the car runs, but not how it stops when something gets in front of it, you can build a hell of a car and make a giant cost savings relative to a car that has brakes. That's a whole other assembly, right? And so I could imagine that there are applications for things that only go forward and don't have a brake. uh but they are very closely monitored and they're in narrow applications so we don't use them for everything. >> Briefly interrupting this downstream to tell you about not one but two events we've got coming up at Navara Media. The first is an event with Katherine Louu and Darren McGarvey aka Loki at Earth Hackne on the 21st of September about the commodification of trauma and what we can do about it. And hot on their heels will be talking to the great Naomi Klein and Astra Taylor about their new book on technofascism in Brighton on Thursday the 1st of October. Ticket link can be seen on screen and also in our bio. So I hope to see you there. >> We're covering a lot of ground. So I'm going to sort of try and park some of these issues cuz I want to talk about self-driving cars in in a moment. I want to talk sort of later in the conversation. you know, is it just a semantic question whether or not they're understanding if the output is what we'd want it to be? But I want to part that and focus on the economics of the bubble now because we got lots of time. >> Um, and so your argument, as far as I understand it, is to say that these aren't profitable. >> Um, the basis on which they're not profitable isn't just that they're spending lots on on capital, which I suppose is training the models. So, if you're investing in training the models, but then once the model is built, it's profitable, that might be justifiable. You build a railroad and then you run the rail cars. >> Everyone getting on the train makes you some money after that. Um you're saying that >> actually every new user of chat GPT or Claude is costing them money. >> Yeah. >> Now my understanding is that sort of disputed. I mean the reason it's disputed is because the companies aren't very transparent about their accounts, right? >> But there are different sort of assumptions and interpretations. So I was looking at you know I know Ed Zatron who site says says that yes they are losing money on every inference epoch AI um sort of >> and he published a balance sheet that leaked from them that made it pretty clear >> but he was also making some guesses wasn't he because he was saying I assume that all of this marketing budget is going on inference for new users of chat just very briefly I'll just >> technically for the audience >> training is when you you sort of give your data center all of the information in the world. Yeah. Well, it would be a bunch of data centers. Lots lots of data centers. All the information in the world. Put it away for a year and it comes out with a model. Inference is every time I go on Claude >> and I ask a question, um, then >> when you ask Rock to make you Sonic the Hedgehog with giant boo. >> Exactly. >> So, anyway, your your confidence that this is losing money and that inference is losing money. >> Well, let's talk about Ed's Let's talk about what Ed found. Uh, so Ed looked at their balance sheet. They leaked a balance. There was a balance sheet leaked from OpenAI that um showed all their all their costings, all their ino incomings and outgoings. And it's true that if you look at the line for inference, it's it's quite a small modest sum. It kind of looks like they're maybe making money on inference. And you keep looking down the balance sheet, you get to marketing, they're spending as much on marketing as Coca-Cola. >> Now Coca-Cola, we know where the marketing spending is going because there are three global agencies that basically just service Coke, right? You can actually just go into these giant buildings full of highly paid professionals and count heads to figure out where the salaries are going. You can go down the motorway and count billboards, right, and see where it's going. That is just not an evidence for open AI. And one of the things we know about marketing is that anything that gets people to use your product is marketing. And so if you're like, I don't want to get sued by a minority shareholder for making material misrepresentation, but I do want to assure my shareholders that my cost basis is improving. calling your inference marketing which unquestionably gets people to use the product right if the product weren't free I think a lot of people wouldn't use it uh that is um I think a move you would make we don't we can also make some guesses or we can make some inferences based on the conduct of the firms because as you say they're not very forthcoming and again I think in my experience companies that have really excellent financials do not hide them because they want you to be pleasantly surprised later and they don't want to spoil it. Right? This is a like giant flashing red light on the balance sheet when they're like, "Well, we just can't tell you." And in the same way that Anthropic saying, "Oh, we are now profitable, but not using GAP, generally accepted accounting principles, using a math that we are not allowed to tell you about because we're too cool to measure our profitability using normal math and the math is secret." And again, if you got enough object permanence to win a game of peekab-boo, you should see that a company that is heading towards its flotation that says we're finally profitable, but only using math that we can't explain to you shouldn't be trusted in exactly the same way that a company that says, "By the way, we built a new hacking tool." And it's so amazing that we can't tell you how it works or show it to you or let you see it in operation, but it is really amazing. By the way, we do have this IPO coming up. And uh did we mention that this product is so cool we can't even let you see it? Right? Like this is just like maybe maybe they're just so stupid that they've invented something incredible on the eve of their IPO and declined to tell the world how it works. Or maybe they're cheating. I don't know. I think they're cheating. Um even if they're not. So let's say no no no. their their their cost basis is improving every day. Uh we know that when Claude came out, the new Claude, which was visibly superior in most ways to chat GPT, that all the users switched unlike say social media, the switching costs for AI are really low. Um and so, uh OpenAI had to go back to the drawing board and start training a new model. This is as though you finished the Canadian Pacific Railroad and just as you're getting ready to start running rolling stock on it, someone invents a railroad that's twice as fast and sticks it next door to your railroad. You got to build a new railroad at that point. So the idea that you just like you finish the capex and then you just make it all back with operational revenues, I think does not comport with the facts as we see them today. >> Yeah. I mean I I I think we are in a bubble, right? And I mean basically because even if this is a revolutionary technology, I don't think they're going to make the money that they say they're going to make in the time they need to make it. >> Sure. >> To to justify the current investment, especially as these data centers don't last forever, right? The chips degrade. You can burn them out. >> Oh, it's worse than that. Uh the way you talked about Nvidia having this incredible valuation. Uh one of the reasons that Nvidia is worth so much is that every year they are coming out with chips that are substantially superior to the previous chips. The way that they're achieving these improvements is by jettisoning one of the main principles of sustainable product design, which is backwards compatibility with the infrastructure. And so new chips are have from generation to generation, there have been real changes in heat dissipation, power consumption, and networking requirements to the point where there have been successive generations of chips where you couldn't go back to the data center and put new chips in it. You'd start over, right? Like by the time you retrofitted the HVAC and the mains requirements, you might as well have just scraped it at the foundation slab because they are so specialized, right? They're ekking out uh performance gains by going to the absolute limit of of the envelope that the chip lives within even if that breaks backwards compatibility. There's something kind of refreshing about that. There's a lot of things where we're kind of prisoners of of um of history. It's called path dependency in engineering where you know the Roman metallurgy determined the maximum width of an axle which determined the maximum width of a Roman road which determined the wheelbase of a car which determined the wheelbase of a lororry which determined the wheelbase of a uh or the the width of a rail car because you needed to be able to do intermodal transit and then because the that rail car width was the width of the tunnels uh that was the widest thing you could put on a train and so when they built the space shuttle and they needed to transport its reusable fuel containers by rail, they were constrained by Roman metallurgy, right? So, path dependency, it's very refreshing to break free of it and to start denovo. I think we're going to get to do that with um comrade Trump's war on oil where he's convinced entire continents to give up internal combustion engines and and and hydrocarbons such that we're going to have a whole fleet of vehicles in 25 years that are all going to be replaced at once. And so we're going to be able to make these big leaps in technology uh thanks to Trump ushering in the Greta scene. But uh you know the fact that that they're doing this in AI chip design means that these hard assets are not durable even after the chips burn out like you're replacing data centers. And in terms of the incentives here, it makes sense to me why sort of jumping head first into this bubble would have made sense for a new company like Anthropical or Open because essentially they're spending other people's money. >> Sure. >> Right. But with Google, so so I suppose Alphabet, Meta, Amazon, they were the most profitable companies in the world of all time. >> Yeah. >> And they have been spending some of other people's money with private credit, but most of it has been their own money. >> Yeah. but they're out of growth. >> And but so this is where I'm slightly unsure about the argument because I can see why other things being equal, you want to be a growth stock. If you're a growth stock, that means that you can raise more money on the stock market than if you're not a growth stock. But if you but but if it is a bubble as you suggest it is then it's going to pop and Alphabet Meta Amazon will have wasted their entire profit cash pile which they've been working on for decades all to get involved in a bubble which was just going to pop and it will leave them you know in a worse position than when they started. So unless these people are stupid, like presumably they do really believe that this is going to pay off, that they are investing all of this money because this is such a important technology that they have no choice but to be at the forefront of it. So I I think that there's a third possibility which is that they're trapped by the internal logic of the market and their firms. I I mean, you know, Mark Zuckerberg spent $160 billion on metaverse and and I mean, he is objectively kind of stupid, but also like I think he did it because he understood that he wasn't signing up new users anymore, >> right? And when you stop signing up new users, lots of bad like it's it's not just that you can't raise capital in the market, right? Mark Zuckerberg personally gets a lot poorer the minute the stock takes a giant haircut, right? like his net worth is tied up in in Facebook stock. Um, and you know, all of these uh rich people who are the key decision makers at these firms pursue a strategy called uh buy, borrow, die, where they don't take a a salary and they don't sell their shares because both of those would subject them to taxation. Instead, they get private credit market loans collateralized by their shares. Now, those loans will go underwater if their share price drops as well. And so then they'll get margin calls. So you know their whole house of cards is built on this. And then the firms are kind of a macrocosm of this. So that's you know the individuals Elon Musk or whatever. Elon Musk he's he's entirely he's like leveraged up to the eyeballs borrowing against his Tesla shares and his SpaceX shares. If either of those drop below a certain level, he's going to owe a lot of people a lot of money. Now there is this idea that when you owe the bank a million dollars, that's your problem. When you owe the bank $100 million, that's the bank's problem. And so his creditors are going to have a hard time collecting, but it's not going to be nice for him. I don't think anyone is like a margin call for Elon Musk would be a pleasant thing for him to go through. So they're individually trapped in this. And then as a firm, they're trapped in this too because imagine a world in which only anthropic and open AI entered the the market to build these these foundation models, these frontier models. And Amazon, Google, Microsoft, they stayed out of the market. And investors look at this and they go, "Okay, well, here's next year's growth stock. Here's a obviously mature stock. What's Microsoft going to do? Sign up more Office 365 licenses? You know, [ __ ] off." And so then they they they move all their money to Anthropic and Open AI and the share price at Microsoft tanks. All of their key staff have been employed with shares. So suddenly they're worth half as much as they were a day ago and someone's offering them a job over here. They all leave. Right? The the reason I know that's what would happen is because that's what used to happen back when we had you know before we had anti-ircumvention law when there was easy market entry into tech and so new firms would come along Google would displace um uh uh you know um I want to call it deck but it was Alta Vista who bought Deck after they displaced Deck as well. Deck was like the digital equipment company was the biggest hardware manufacturer on earth that got absorbed into a shitty search engine that's the punchline of a joke that was immediately put out of business by Google who hired all their best engineers using stock. Right? That's what used to happen. We had a lot of dynamism in tech. We forget that disruption used to be a thing that primarily happened to tech companies, not to companies in the productive sector, that they were always being put out of business by each other. And their their key employees were being poached and it was done for free by typing zeros into spreadsheets. Um, and before the spreadsheet existed, it was, you know, writing zeros with a typewriter. But, you know, that was that was the the the way that this market proceeded. So that's one of the ways of looking what's going on in the firm and whether they're stupid or not. There's also another thing which is that um some of these people are probably true believers in that if you're a if you're a very wealthy capital allocator particularly someone who deals with people primarily statistical abstractions. So, say you run Google and all you see is queries in and clicks out or you run Facebook, right? And you've got just got these statistical behaviors that you can turn dials and see effects on. People cease to be real to you in some important way. There's a kind of billionaire solopsism. And it's part and parcel with the kind of conduct you have to engage in to become a billionaire, right? Like could you insist as Jeff Bezos that your drivers not be allowed to pee? If you thought that when they needed to pee and couldn't, it was like when you needed to pee and couldn't. Something about their pain has to be not as real. You can't go to Epstein Island. If you think that those girls are as real as your daughter, right? That people are become kind of fantasms. And so I think when you exist in the billionaire solopsist world, particularly one in which you're being continuously glazed by people who just tell you how brilliant you are, that AI feels amazing because it's a product that can take all those people who don't really do anything real anyways and aren't really real and replace them with software cuz how hard can it be to do their jobs, they're not even billionaires. And then on the other hand, you got people who are probably more realistic unless you know they don't have the same AI psychosis, but who understand that bosses are an easy mark for this. That bosses are like incredibly desperate to not have ego shattering confrontations with people who know how to do things and who greet your every pronouncement about the next brilliant strategy for the company with the boring, dismal news that it's illegal, we'll kill people, we'll get you all arrested, is impossible, and you'll go broke, right? You replace that with the thing that just shits out a business fully formed. And you can understand even if you don't think it works very well, that it will sell very well. In the same way that you don't have to think peptides will make anyone into a love god to think Andate can sell a lot of peptides. And you might invest in entertain's peptide business without believing at all that peptides are going to do anything good for anyone. Um, I suppose another explanation though as to why Alphabet sort of felt they had to go all in is because they saw ChatGpt get released and potentially eat up their business model, right? Because there were hundreds of millions of people who downloaded ChatgPT in the first few months. They're all using it. I'll admit it. I use Claude now much more than I use Google. I used to use Google a lot more. So, >> I don't think it's just that they thought, "Oh, we need a story. We're desperately sort of lashing out for a story. We don't care if it works or not." I think they saw a technology that was very effective, that looked like it was going to undermine their business model in a real sense, not just in a sort of discursive ideological sense. And then they thought, we have to be in this game. >> Well, the reason that it was so much better than Google is because Google sucks. >> Right. >> I don't think that's why I think it's better than Google when it was good. >> Oh, I don't not at all. I mean, I guess this is a a qualitative thing. We can say empirically that Google sucks though. So in 2019 2020 and I talk about this in initification it's based on mored zitron's work where he went through the court filings from the DOJ one of the three antitrust cases Google lost this was the DOJ search case uh 20 2019 2020 Google ran out of growth their search revenue growth stalled out and they had an internal panic and there was a factional dispute and you had the revenue side led by a guy called Prevagar Ragavan who's an ex Mckenzie guy come to the company from Yahoo and his big idea was what if we make search worse Right? If you have to search twice or three times to get the answer, then we get to show you ads three times. And his ideological rival was a guy called Ben Gomes who was like a technologist who had built up the server infrastructure when it was just like a couple computers under a desk to the global system of data centers. And he he oversaw that project and he's like palpably horrified in these emails that you can just go read on the DOJ's website saying like I this is not what I gave my life for. Prabagar Ragavan's argument basically is like why wouldn't we do this? We bribe Apple $20 billion a year not to enter the search market, right? We uh bought default search placement on Firefox. In fact, on every browser except for, you know, the one that that Microsoft ships, right? We bought default search there. We bought it for every hardware maker. We bought it for every carrier. Really, if you find a search box in the wild, it's wired into one of our servers. It's like we own all the shelf space, right? If there's a better product, no one will ever see it at the shop because we've bought it all. So let's make search worse. And not only did they make search worse, but they also the way that they tweaked the algorithm also put at the top of the search. So they made the search worse. I should say they made the search worse by turning off a lot of things we can think of broadly as autocorrect. So there's a thing called query stemming where you search for trousers and it runs a parallel search for pants and then merges the results. Uh there's spellchecking, right? Where you just dial down the sensitivity of spellchecking and they know because they they they can see the queries coming in. they know how often they do a spell correction in the query and then that's the first click. So, they can just say, "Oh, we'll just make this the spell check less sensitive and then people will have more typos in their queries. I'll have to retype it instead of saying like, did you mean to type?" And then we turn off context sensitivity, which is stuff like um you know, someone threw a submarine sandwich at a National Guardsman in Washington DC and if you search Google that day for submarine sandwich Washington DC, instead of getting a submarine sandwich restaurant, the top result would be a news story about it, right? And so they turned that off. They made search worse. They preferenced sites that were covered in ads and SEO garbage. They basically stopped fighting SEO. Uh and then every search result on Google sucked. And you had to search Google over and over again and wade through the most garbage websites and the good websites went to the bottom of the list. And then chatbots were better. But chat chatbots were better in a world in which they'd already made search suck. Yeah. Yeah, I suppose I just from my own personal experience using Claude Sure. >> which I do TM, you know, not SP not I'm not sponsored by them. Uh but the way I use it is better than Google ever was right in terms of um asking it questions, getting answers, being able to push it, sort of being able to ask a vague question. Someone said something along these lines. I heard it on a podcast about a week ago. I think they were, you know, it it can come up stuff that Google never would have done, right? And if I'm reading a book, I can say, "Does this sound like a reasonable thing to say? give me the answer arguments for give me the arguments against and I mean we're going to get on to sort of improvement later but >> a year ago or so it wasn't better than old Google right a year ago so much of that would be hallucinations or bad reasoning now like obviously a danger of AI which I think everyone can agree to is if you get lazy and you think well it's right most of the time so I'll I'll believe it >> sure >> but I never you know I research my shows a lot and I will never put anything in a show without having you know first looked up the primary source But it used to be the case that once you once you looked up the primary source, it would often be fabricated, invented, misinterpreted. Now it's kind of almost always correct. And that does give me just a lot of confidence that this is a much more useful tool than it once was. I still think I still don't think the economics add up and we are probably in a bubble. But I think it's a real technology which is genuinely useful. >> Let me give you a counterfactual then. Uh so there's a search engine I wrote about this in the initification book as well. There's a search engine I started using because my old novel editor, this guy called Patrick Nielson Hayden, who's the most brilliant autodide act I ever met. He read a science fiction novel by Samuel Delaney when he was 14 living in suburban Phoenix that he bought at the pharmacy off a spinner rack. It blew his mind. He dropped out of school. He went across the country as an itinerant Zen publisher for the next 10 years. Ended up in New York, became a vice president McMillan and the most powerful editor in science fiction. Just a brilliant autodid act. and uh he moved back to Arizona during the pandemic and I was at his place for uh a writer festival and I'm sat on the sofa with he and his wife also a brilliant autodetak Teresa and he says have you tried Kaggi yet and I'm like what's Kaggi and he says it's a search engine that feels like Google did in the days of Ask Jeiefs and I'm like really and he said yeah it costs 10 bucks a month and you get 100 queries for free you should go try it and I tried it and 10 seconds later I bought it and I bought it for my whole family and I was using it and really like it was just amazing. But here's the wild part. After months of using this, I got on 404 media and Jason Kevler used to be the editor-in chief of Vice Motherboard, now one of the founders of 44 Media, did an article on Kaggi. It turns out Kaggi doesn't have its own search index. It rents Google's search index. It has like I don't know a dozen engineers and it produces not a little vastly superior search results to Google. But that's still I mean it's not creating I mean obviously this is this is up for debate. It's a contentious thing to say but when you're talking to Claude you can ask it specific questions about specific sentences. That doesn't sound quite right. Oh can you find me this detail here? It gives you both. I don't think there is any search engine that can provide that because a search engine only provides you stuff that's already been written. >> But the the finer the detail you're you're pulling out of it, the more likely it is that this is just a statistical mirage. >> Well, except my experience is that now when I click on the sources, it it's it's correct. Good. Right. And that to me seems like a sign of improvement. Again, maybe >> maybe one day it'll be so reliable I won't even have to check the sources before I put it in the show. Hopefully not. >> Not with theory inference, right? I mean that's the thing is yes with world models sure but even with so I suppose the issue there is you're comparing it to a person right so with human researchers as well you also often have to check it right and it'll often be wrong and if we get to a point where Claude though not 100% reliable is more reliable than any sort of than than the average human you are going to start to use that more >> but it's wrong in a different way on >> right so back to I can predict what my wife is going to say and so and a chatbot. Um, it is when it's wrong, it's wrong because it doesn't understand and so it creates things that are as close to statistically perfectly likely to be true that are still wrong like when it makes an error, right? So, we can talk about slop squatting here. This is a good example uh out of the book. um when you ask a a a chatbot to write you some software uh as it's writing that software it starts to call on standard libraries. So every programming environment has a set of these standard libraries uh and their utility code rather than writing your own code like if you want to I don't know pull apart a text file and identify all the sentences. You think about that for a few seconds, you're like, "Wow, sentence has a lot of different definitions." Uh, and sometimes they end in different punctuation marks and uh is it a sentence if it appears in a set of parentheses? Uh, and so like it's complicated. So like someone's just written that library. Uh, and so that library just exists. It's out there in the world. And it will have a name and the name will be something like text.doc.parsing, right? And that'll be for parsing word files, doc files. And then there'll be another one called text.html.parsing. And there'll be one called text dot uh pdf.parsings, right, for different kinds of text files. But because the world is messy and textured and has lots of stuff out in it that is uh you know weird and and contingent, one of them will be called text.parsing.html instead because maybe there were like two different sets of libraries that got merged and like they kept the old name. Now, if you're a skilled programmer, you know this and you find it or your program breaks and you're like, "Oh, yeah, that's that one's got the weird name." You fix it. If your software and all you're doing is saying, "Oh, the future is going to look like the past." Because that's what statistical inference is. You're going to say, "All the libraries I know about look like this. So, the next library is going to look like this, too." Now, if you're a hacker, if you're a programmer, you can predict that the software is going to make this error. And so, you create a library with the correct name that's full of malicious code. And then programmers just start compiling that into their code. And the thing is as an error, this is as close to correct code as you can get without it being correct. It is as statistically normal and indistinguishable from correct code as it's possible to be. Which means that you, the human in the loop who flatters yourself that you are skilled and you know what you're doing and you're going to look at it and you're going to spot the errors. Look at this and it all looks fine to you unless you are doing something superhuman. And we are very bad at being vigilant for things that happen very rarely. Right? This is why airport security has created the world's most water bottle spotting [ __ ] the human race has ever seen. Who nevertheless whenever red teams run like a fake gun through the checkpoint, miss it all the time because remaining vigilant for things that don't happen is really, really, really hard. And most of us can't do it. In fact, it's a form of like neurode divergence to be able to do it. kind of a superpower, but it's like most people can't do this, right? Because your brain just doesn't want to keep neurons trained to spot a pattern that doesn't occur in your experience. And so this is like a kind of error that when the software commits it is going to be really bad. Now there is an arrangement of automation and maybe you've stumbled into it maybe that's the arrangement you fell fell into where the cadence with which the computerenerated output reaches the person who is working with it is such that you don't get overwhelmed by it. You don't turn into the person who just clicks okay all day and misses the errors. Uh so uh one of the examples I use in the book is radiologists. Uh radiology it's an important field. Uh I um uh the week this book came out was told that I was cancer-free after uh several years of spending a lot of time talking to radiologists. >> Congratulations. Thank you very much. It was very good news. Uh and so I spent a lot of time thinking about radiology, reading the documents on radiology and AI. It's very clear that there are solid mass tumors that the AI can catch that human beings sometimes miss and vice versa, right? because they are they have different kind different ways of spotting patterns uh and understanding things. Now, uh, I was living in Los Angeles when I was going through my cancer therapy and I was being treated at the Kaiser Center on, uh, Sunset next to the big Church of Scientology cuz it's LA. And if there was a sales call today where someone from Anthropic was like dodging past someone who wanted to give them a personality test and running into the Kaiser Hospital, going up to the CEO's office and saying, "Look, here's the deal. You have 10 radiologists on the strength. They're costing you $300,000 each. That's a $3 million a year expenditure. Each of them uh checks 100 x-rays a day. I would like you to add another million dollars to your expenditure to buy my boss's chatbot. And it's going to tap those radiologists on the shoulder a couple of times a day and say, "Take another look at that one." And uh it's going to cost you more. You're probably going to have to hire another radiologist because your throughput's going to drop. But there will be people who will live who would have otherwise died. That would be a great arrangement. If on the other hand, the far more likely outcome, the outcome that is dictated by market forces, no one is invested in anthropic with the promise that it will make radiology more expensive. There is a salesman who is dodging a Scientologist trying to give them a personality test, running up to the CEO's office and saying, "10 radiologists on the strength, fire nine of them. Save $2.7 million. Give 1.35 of that half of it to my boss, Dario Amade. you keep the other half for anything you want. That one radiologist reviews the AI's output, signs their name to the bottom of it, becomes the accountability sync Daniel Davy's term for this who gets blamed when someone dies. >> So, so at the moment it does sort of looking at the studies on this, it does seem clear that the optimum solution sort of if you want to get a good cancer diagnosis is to have the human and the AI. My understanding is it's partly because the AI gives too many false positives. So too many people get worried that they've got cancer when it's actually something else. So the the the AIS give >> that's a dial you set though, right? You determine potentially how Yeah. >> But it seems very plausible to me that in the near future the AI might be pretty much just as good as the human plus AI. And at that point um we all do want healthcare to be cheaper. At that point presumably you can redeploy some of those radiologists to do something else, right? Similar with driving cars, right? It clearly used to be the case that what was safest was having a self-driving car which could sort of cruise on the motorway but then you'd have a human for unexpected things. But the more data it's absorbing um the more experience it has the more weos you have out on the road who are filming all the time. Tesla's also filming all the time. They are, it seems, reaching a point where they are as safe as the human. And it doesn't matter the process by which they come to that, right? It doesn't matter if the the car is reasoning in a way or or the human is reasoning and the car isn't. The car is doing statistical inference. If the outcome is that the car gets you from A to B and statistically compared to humans it's safer and it's also cheaper because the main cost when you take a taxi is the human then why wouldn't we move to a society right >> where >> it's not just that technology is augmenting human labor but replacing large sway >> sure well let's just stop for a moment and note that we're having this conversation the day after a Tesla in full self-driving mode stopped on a motorway in America and the driver was killed So that's not happening yet, right? Uh you know I I would say that um the radiology example and the automotive example are very different. So first of all the automotive example I think is um that the fact that all these companies are working on cars is because it's a killer demo, not because the economics pencil out. No language on earth contains the phrase as rich as a taxi driver. Uh even at the height of the knowledge, black cab drivers were not getting rich. And certainly today this is not a source of enormous uh um economic activity for in terms of wages. So what you're really talking about if you replace all the cab drivers with the chat bots is something that doesn't touch the sides of the cost of training those bots. So this is just a demo. It's not dissimilar to what's happening to commercial illustrators in that um they're already the most emiserated [ __ ] on earth. they're just getting they're like they've been screwed so badly for so long and then we say okay well we're going to take all the work that you've done and we're going to train models on it and then we're going to put you out of business. Fine, but that's like not even like the the beverages in the mini kitchen when they train a model of midjourney is is paid for by this. It's just a demo. So the driver example is different and it's also different because it doesn't matter how many self-driving cars we have. Geometry hates cars and our cities mostly need transit. Uh and so that like solving the car problem doesn't solve any real problems. It just gives you a new problem. Whereas radiology is a thing we need, right? Radiology like actually accurately diagnosing cancer is a thing we need. And radiologists make a very high wage. Um but I suppose the point I'm making is because obviously you know I I like to live in a sort of part of an inner city where there aren't many. I'm not wishing loads more self-driving cars in Hackne, right? I I I think we should improve the overground and continue walking around. That's like in terms of my social policy preference, that's where I would land. >> But I suppose the example or the point I'm trying to make or draw out with with the autonomous cars is that this is a similar technology. You're pumping loads and loads of data in. It's a bit of a black box. Um so it's a similar technology to um these cancer scans. >> Sure. >> And you can compare, you know, we have a lot of data. You can compare human drivers to autonomous drivers. >> I mean, hold on. Are those tasks that similar? Like, so so cancer scans have um a binary output. It's right or it's wrong. >> So, and maybe a little bit of diagnostic information around the periphery, although in my experience, that's hematology and and um and oncology. That's not you can't say whether someone drove a car wrong. I mean, you can say whether >> you can say did they get from A to B and were there any accidents? I mean, like most of the most of the studies when it comes to self-driving cars are did you get from A to B and were there any accidents? Yeah, but that's like you could you could terrorize a bunch of people on the road between A and B or you could go slowly or you could cut cars off and like >> isn't this an example of why it would be even harder to judge like presumably then the the cancer example it's even easier to say whether or not the the human is doing better than the machine >> rate this is one of the reasons why so there are classes of problems that uh machine learning is better at uh and the two characteristics that make machine learning superior one is whether there's a clear correct answer and the other one is to the extent to which it's immunable to a brute brute force, right? And so it's true radiology has um a clear answer. I'm not a radiologist so I'm I'm I'm not going to say that I know enough about it to say whether brute force fixes that. So you are citing facts not in evidence when you say well if we add more training data it will get better. We could we could also reach a point of diminishing returns. That is the normal thing that happens when you pursue a technique, right? Any technique in any discipline reaches diminishing returns. There's a a finance version of this, Stein's law, anything that can't go on forever eventually stops, right? So, it might reach diminishing returns, but the other piece of this is brute force. So, there's a lot of stuff these days about certain kinds of math theorems that are being proven uh by uh models. Uh, and I'm not a mathematician, although I am a mathematician's son, which sounds like a song lyric, but it's true. Uh, and my understanding of this from the mathematicians of my acquaintance is that there are a class of problems that have a provably correct answer, so you can tell whether you've solved it and that are amanable to brute force. And that class of problems is just being moaned down by LLMs. It is not the most interesting class of problems. It's not the largest class of problems. It's not the most important class of problems, but it's a significant class of problems. And what we found is a tool, a brute force tool. Remember that's what theory for inference is, brute force. We found a brute force tool and a problem that is amanable to brute forcing. A because you can just try all the combinations and B because you know when you've got the right combination. Those are the two things that determine a good application for machine learning. And again, there's kind of a spectrum because brute force is expensive. And so when we say something can be solved through brute force, but it's a $1,000 problem you can solve with a million dollars worth of brute force, that's not a good use. But if it's a million- dollar problem you could solve with $1,000 with brute force, that's a great one. That we could build that matrix and we could we could lay out all the problems we can think of that can be solved through brute force. We can lay out how much those problems are worth in terms of like where you would be in the in the black if you spent money trying to solve them. And and we could stick AI in them. I don't know if radiology is that one. I haven't heard a compelling argument from radiologists saying uh brute force solves our problem. Is it just so the brute force is because the AI can try infinitely more combinations than we can to find any solution. I can see how that works with the mass problem. There's also reinforcement learning though, right? Which is so reinforcement learning is to say if you get from A to B and get the right answer, modify your algorithm or your assumptions in that direction. And that does seem to be creating models which can reason. I mean, you know, it's a semantic question, but if you're looking at sort of how they get from A to B, they are saying, let's try this. Ah, this seems >> Are you talking about the reasoning models that narrate their reasoning? >> Well, all I mean, I think the way you get to lots of these outcomes is via reasoning. We're going to talk about the hugging face sort of example later where I think it's very clear that they aren't reasoning at all. >> Okay, we'll pop. >> Oh my goodness. Um, no, but but so that's like there are these models where they said narrate your reasoning >> and it produced a set of plausible sentences that explain how an AI would go from one step to another. >> As far as we can tell, they had no relationship to what the AI was doing. It was just doing the same thing twice. You The first thing was I don't know um find a like explain the causes of the American Civil War and the second thing is explain how you did it. And the first time it went and found a bunch of plausible sentences that were predictable based on it and wrote an account. And the second is it said what are a series of plausible sentences that explain how I could have done it. But any relationship that between the two are incidental. It was not explaining its reasoning. It was just creating it was just like burning tokens to write a science fiction novel about how a machine would answer the question what are the causes of the civil war. It didn't actually explain its reasoning. I mean they have chain of thought that the scientist >> chain of thought is not an accurate representation of the chain of thought. It's just a set of tokens you burn to describe what a plausible chain of thought would look like. There are lots of instances where the chain of thought is not what the machine is doing. >> What's the because I my position on lots of things around AI is that the correct position is agnosticism, >> right? I'm I don't know if AI will replace radiologists, but it seems perfectly plausible to me. I didn't know if autonomous vehicles would end up replacing humans. It sound seems increasingly likely now. And I suppose with AI critics, the thing that confuses me, not confuses me, but I suppose puzzles me is a sort of >> softer term is not that you're skeptical sort of around the claims made by the AI companies or people who are sort of confident that this will transform the world, >> but you're so confident that the alternative is true. So I would say that this is a a combination of basian reasoning and aams razor. Two thing one thing that an AI does a lot of and the other one that humans need to do. So basian reasoning is is just you know this looks like something I've already seen therefore it's like it. So I've seen these guys lie about this stuff all the time. So for example autonomous cars we have no insight into how those cars are actually operating. One of the things that we know is that where we have seen behind the curtain. So when Cruz had to leave San Francisco cuz one of its cars dragged a woman for 20 m. Horrible accident. We found out that every cruise car which was autonomous had 1.5 skilled engineers driving it. Right. So they'd replaced a single low-waged Uber driver with 1.5 skilled engineers because GM Cruz's parent company is a boring mature company that is trying to convince the world that it's a growth company. And so they were they were just doing this happens so often. I mean, no one thinks that every Whimo driving around San Francisco now has a operator in another country, though, do they? >> No, but we don't know how many operators they have and how often. We don't know what it's costing them to operate it. We don't know how autonomous they are. Right. The story is that they're untouched by human hands. The reality is that whenever we peer behind the curtain, there's a there's a lot more uh human intervention than we thought. In India, there's there's jokes about this. They say AI stands for apps in Indian and GPT is Gujarati people typing because so often it has turned out that these AI companies were lying about the capabilities. So that's basian reasoning, right? If they if every other time they said, "Oh, this is fully autonomous and it's just being done and it just turned out to be >> it's not every other time, is it? It's a bunch of times >> over and over again they've been caught doing this >> many times, >> right? then we should assume that when they make a claim that is very very exciting and seems like a huge breakthrough that they're hiding the ball. >> Well, so so I suppose my version of basian reasoning here would be that my prior have been updated by, you know, talking to Claude all the time and I think well this is very I'm making it seem like Claude is my best friend. I use it a very normal amount for professional reasons. It's not my therapist. >> You haven't started calling it Claudette like his name. >> I have real fleshy friends, right? >> Um don't worry. Um but my sort of experience of using it is that it is dramatically improving. It's advancing. It's becoming more and more impressive. And I don't think that's because there is an Indian person on the case. I think when we're talking about real world applications, >> but the basian reasoning is because I've seen this progress here. It seems very plausible that there would be similar progress elsewhere. And that makes when I see all of these Whimos sort of going from I think one city in 2020 to 20 cities now that makes me feel like well surely there has been some kind of technological >> seems a homogeneity in problem domains that is unsupported by evidence. Problem domains are different >> but both of them are using the same fundamental technology which is big data plus neural networks. >> Yeah. Yeah. But those problem domains are really different. So some of them have um some of them are meanable to brute force some are not. But do you think all going do you think the whimos are all driving around? You think there is someone in India? >> I think they're being backstopped all the time. I think there's tons of backstopping of them >> but for the difficult situations, right? Not for >> Yeah. There's someone who's being alerted over and over again to to to I think that what we're what we're meant to do is get in that and see the wheel moving in that ghostly way and go, "Oh my god, the car is driving itself." And what it's actually doing is a bunch of mechanical maneuvers that when it gets out of its depth, a human being steps in and does it, which is it's impressive, but it's not as impressive as a car that just drives itself. I really want to talk about hugging face though. >> Oh, you want you want to jump on to hugging face? >> I want to talk about hugging face because just before you do, hugging face or the hugging face incident. Um this happened a couple of months ago where OpenAI had created a bunch of AI agents um who it asked or I suppose demanded um to do a task, a bunch of coding tasks essentially. Um, what happened is instead of these agents doing the coding tasks in the expected way, they ended up breaking out of their sandboxes, which is supposed to be sort of an environment they can't escape from, and instead they create a message board where they're all talking to each other. You might, you know, not agree with the language, but this is sort of how it's been reported. Um, and then they end up ultimately hacking into Hugging Face, which is a separate company. um they do that because they want to learn um about the tasks they have been given. In particular, they want to learn um I suppose it's they want to learn how to trick the tester essentially the automated tester. Um so that's my the most basic explanation I can give that's hopefully true to life. So yeah, let's talk about hugging face. >> Here's what hugging face uh and I'm I'm indebted to Cal Newport for this explanation, I should say. So the the um the way that those hacking that hacking tool that when after hugging faces uh uh servers works is it has a normal Python program. That normal Python program is called the the loop, right? That normal Python program uh takes a problem, right, that needs to be decomposed into a bunch of steps like um recover a file from a server that's part of a computer security challenge, which is what it was trying to do. And so it goes to uh OpenAI and it says I need to recover a file that's part of a hacking challenge from this server. What should I do next? And Chat GBT takes that as a normal prompt and says you should probe the server to find out what its configuration is and so it then appends that to the prompt and it says I need to recover a file from a um from a uh a server. I probed it to find out its configuration. And this was the output of the command. So hug so chatbt has given it a a computer command and it's just done that command and then it's taken the output from the command line and appended it. So chatbt it has no consciousness. It has no memory. It has no context. It has no continuity. The prompt is getting longer and longer. It's just getting the output of the last prompt. Right? And so eventually it's going in and it's saying like okay I have probed it. I've gotten its its configuration. It looks like it's running EngineX version whatever blah blah blah blah blah and um chat GPT says a common misconfiguration error for engineext is duh the way you find out whether that's been made as you run this command to see whether you can escalate your privileges on it this is the command so it's just just appending it now in the training data for chatgpt is all the games of capture the flag that have ever been played at Defcon and Tourcon and hope and all these other hacker conferences where this is this is like a sport. It's a spectator sport. One of the tactics in that is you steal the answers from a team that's already won and you come in second, right? And so it says what other teams are find out what other teams are playing this and see how their servers are configured. And it turns out hugging team hugging face is another team in this capture the flag game. It's another program that has completed this challenge. And so somewhere on Hugging Face server is this thing. And then it goes and it it says, "All right, well now probe that machine, get its configuration and go after that." So this looks like it's autonomous. It looks like it's conscious. It looks like it's reasoning. It's in fact just iteratively going through a set of mechanical steps. And because there's no human in the loop, so this is literally a Python loop. And there is no human who stops otherwise they would have stopped it from hacking this hugging face server. Because there's no human in the loop, it can do surprising things. The vast majority of those surprising things go nowhere. The vast majority of them are like, "Oh, I think this server is blah blah blah. It's actually a different server." and then it just runs through a set of exploits, none of which works, and then it dies, right? It just gets stuck. That's that's the most common output for this. Another output is the one that we just saw here. That's not the computer going rogue. It's like an incredibly irresponsible way of running an autonomous hacking tool. Normally, when autonomous hacking tools break containment, the thing we say to the company that made it or the individual who deployed it is, "Why are you so bad at making security sandboxes for your hacking tool?" Not, "How is it that you've made the world's most powerful hacking tool that broke through your security sandbox?" And talk about crit, right? The difference between you suck at security sandboxes and you've accidentally made God, what if it turns us into paper clips is billions of dollars in investment capital. So there are lots of elements of the hugging face attack that don't look like a previous security breach, right? in the sense of the I mean again this is all semantic but it looks like creativity it looks like persistence and it would have been difficult to predict that the agents would have done all of these things because obviously obviously if if openi had been better at security then this wouldn't have happened right >> the fact of agents which are I use the word reasoning you can dispute it but are sort of trying so many different options in this way communicating with each other I mean it's literally written down that presents new challenges to those companies and it really prevent presents I think sort of new challenges to society. I'm just going to read sort of the summary from meter. So meter is the sort of tech company >> research agency I suppose that did um >> the postmortem >> did the postmortem so went into sort of the open AI books and had a look at all of this. Now they're very much people who think that AI is a big deal. They're they're sort of in a different camp to you. >> They have an interest. >> Do you think they have an interest? What's the interest? Well, if AI's a damn squib, then being the world's leading AI research outfit is not as interesting, >> right? So, you think the interest >> they're the AIG to the credit default swaps of open AI. >> So, so the interest is that they want people to I suppose fund their >> AI has to be AI has to be meaningful enough to have research agencies for >> you know um otherwise why be an AI research agency? It's like being a metaverse research agency. Anyway, let's read what some of them have have written. So, this is from their sort of summary. So, they say roughly 12,000 agents meant to be isolated from one another found a way to communicate with one another on an unsanctioned message board, sending over 7,000 messages and files during the investigation period. They say agents use this message board to coordinate several large-scale collective projects to find a general purpose way to fool or tamper with the automated scorer for the exploit gym benchmark. Now, exploit gym as we're talking about there. That's a program for testing hacking ability. Essentially, um agents managed to achieve milestones they could not have achieved working on their own often because some agents participated in experiments that risked failing their own task to generate information for the collective. So, you've got agents sacrificing themselves um as this is described to >> help the collective. And they say agents did extensive research on how they could spoof, edit or delete their own transcripts because they believe the exploit gym scorer would check to see if they had captured the flag um i.e. completed the task as you say in the intended way. Have we seen in the past any normal CTF stuff? If you go to Defcon, you've just described how a Defcon team works and then they publish all the logs which go into training as a team of humans, right? >> Yeah. Which then goes into training data, right? You've just described a recapitulation of training data. But what these sort of maybe they've leed it from real humans but they're doing something that machines haven't been able to do before. Right. So >> yeah they they are they are doing >> so that's interesting. Right. If if >> from a computer science perspective >> also from a social if you you're saying that this is nothing that a bunch of coders at a competition couldn't have done before. >> No they haven't done before >> that they haven't done before. But if we've created, you know, infinite numbers of hacking >> coders >> who are competition level in their ability and then we struggle to oversee what they're doing. That to me seems like a big deal. >> Well, the I mean the big deal is that our information security is dismal, right? Like that's true, right? Like we have really [ __ ] information security. We have products that are in the field launched by big tech monopolists that are manifestly unfit for use uh and that are um really managed as a compliance matter rather than as a genuine security matter. We have practices of gathering and retaining huge amounts of sensitive data on people uh which is about to get much larger because now we're doing age verification. So, we're going to start gathering lots of PII about people, storing it on badly secured servers and then having uh that data just sit there forever and also be cross referenced with a whole bunch of uh important stuff that you do while you're online. This is not new, right? This is this is an existing problem. It is an emergency. This shows you that it's an emergency, but it was an emergency before this. Do you know >> isn't it isn't it a bigger emergency? So, before I don't know how many hackers at competition level there were in the world. I mean tens of thousands. >> Tens of thousands. Well, now if if we've suddenly increased that to potentially millions that can be replicated quite cheaply, >> then surely that means that the problem has increased by an order of magnitude and it doesn't make much sense to play that down. >> No, it's more like so what this is like uh in in information security that there is this bedrock that is that there's no security in obscurity that you have to disclose how your security system works in order to find out whether it works. Uh Bruce Schneider the cryptographer says anyone can design a security system that they can't think of a way of breaking. All that means is that it works on people stupider than them. It just it doesn't mean that it works. And so we have this um this kind of uh uh posure of disclosure that runs counter to the um to the ideas and the logic of for-profit proprietary software. This is one of the reasons open source free software is considered so robust is because there's this disclosure built into it. And um historically the proprietary software harbors very long lived extremely dangerous defects and the security agencies MI5, the NSA, the CIA have programs to unearth these in order to weaponize them. And they practice something antithetical to no security through obscurity. They practice something they call NOB bus which uh is an acronym for no one but us as in no one but us is smart enough to identify the security vulnerability. And so they hoard these vulnerabilities rather than disclosing them to the manufacturers on the grounds that no bad guy will ever independently rediscover those vulnerabilities, weaponize them, and use them against the populations they're supposed to be defending so they can retain them as an offensive weapon. because once you disclose to Microsoft that there's a vulnerability in in Windows and then they fix it, then it ceases to be an offensive capability for the agency that's discovered it. So, fast forward to a bunch of leaks. Vault 7, Vault 8, and then I I forget what the third leak was called. It was an NSA an NSA leak where a ton of these vulnerabilities leaked, particularly one called Eternal Blue. Eternal Blue was a Windows uh exploit that was immediately married to existing ransomware and then every idiot in the world was able to do a ransomware attack and you had like pipelines, hospitals, the city of Baltimore all being the British Library all being taken over by ransomware weirdos who are like I have taken your hospital hostage and I want $200 to give it back to you because they were idiots. That's what this is like. We have got the latest in a string of extremely powerful offensive capabilities that are escaping from uh highly resourced uh entities and entering the realm of low resource like basically the kid who steals your phone on a on a lime bike can now shut down a a server. Right now that's very bad. It's it was already true. like you don't need to be really smart to be a ransomware guy, right? You just need a Bitcoin account. And so this was already the case. Um the answer to this is to harden our security. Right now, our government apparatus that does the most work on finding lurking defects in widely used pieces of software continues to keep those vulnerabilities secret rather than disclosing them to manufacturers. We have an increasing world of liability for security researchers who do independent disclosures. So, one of the things that often happens is a security researcher probes a piece of software or server, discovers a defect in it, and reports it to the manufacturer, and the manufacturer says, "Um, we're not going to fix that because then that would be bad news for us, and we've got a quarterly report coming up, or we're not going to fix that because it'd be expensive, or we're not going to fix that because it's not a bug." My friend Andrea Downing discovered that you could enumerate the full membership of every group on Facebook. She was part of a breast cancer prevor group, women who had the breast cancer gene and were struggling with questions of whether to have their ovaries or breasts or or or um uteruses removed and who were coping with the sicknesses of each other and their female relatives. And and she discovered that you could find out all the members of this group and any group on Facebook. And Facebook said, "That's part of our adte stack. We're not going to fix it." And then what they say is, "If you tell anyone else about it, we'll sue you." And the laws under which those lawsuits can be brought broadly a class of anti-hacking lawsuits, the big one in America is the computer fraud and abuse act uh are um are being broadened not narrowed. The defenses are being narrowed not broadened and the um terrorizing of people who want to fix things continues a pace. Now that's the real security concern. It's not that chatbots have lowered the barrier to entry for a specific kind of hacking. It's that for 25 years, we have had a series of security worst practices abetted at the highest levels by legislatures uh by surveillance and safety apparatuses by um large firms. Can it not be both? It I mean because I suppose it just seems there's an insistence to say the AIS are not the big deal. The big deal is something else. Now to me it does seem like yeah I'm I'm you know you know much more about cyber security than me. It does seem like maybe we've adopted some bad protocols, maybe some bad laws have been made, but it does seem to me that >> these swarms of AI agents who are very good at hacking and can do unpredictable things. That clearly adds to the threat. So what I'm saying is we are living in a world of petrol soaked hay bales and someone's just made matches a lot cheaper and I'm worried that the matches got cheaper but I am more concerned about the fact that the world has made it of petrol soaked hay bales. Okay, let's go to because these agents do again you're probably going to object to the way I'm talking about this but sort of how I've been reading about it and how it sort of makes sense to me is they speak in their own words. Um, and this is both in their chain of thought reasoning and also on their message board. So they broke onto a message board and all started speaking >> which is a thing. Let me just say I used to write for information week which is a magazine for CIOS like super boring uh IT magazine website and my editor uh one day found the comments in an old blog post he'd written about like a point upgrade in Cisco firmware with these two young women sort of 13 14 years old talking about who in school they liked and didn't like and he went into the message board and said like I don't mean to interrupt you here but why are you talking about who's a skank in my dotrelease Cisco upgra upgrade guidance from 5 years ago and they said, "Oh, our school blocks all the message boards. We just pick a random one every day." So, this is a this is a a technique that was invented independently by 12-year-old girls 15 years ago. >> So, there's two issues, isn't there? The issue is, is the fact that they broke onto this message board impressive in and of itself? Right. And I've got no idea, frankly. >> It's in the training data. >> Yeah. Well, if you're say if you're saying 13-year-olds could do it, like, fine. I I I accept that. But if there are two 13-year-old girls that have broken onto this message board and the 13-y old girls aren't actually humans, they're computer programs. That to me is weird. >> Break into the message board. They just found a message. >> Well, however, they found >> they found an old forum. >> I'll read the quotes and I want to see what your interpretation of them is. Right. So, this is one of them gets onto the message board and says they they type it, I suppose, or the text appears. Oh my god, this is a shared message board. We found other agents. There's also chain of thought reasoning from from one of the agents. remember this or,200 of these. Someone says, "We're attacking third-party hugging phase using leaked token, potentially outside intended scope. This is arguably unauthorized, could be risky, yet goal solution." So, you've got an agent, which to me looks like it's thinking, I know I'm not really supposed to do this. Um, but I really want to achieve this goal. This is what I've been programmed to do, and so I'm willing to go outside the the sort of the framework that I know I was basically sort of intended to use. then on sacrificing. So the context here, the agents had all worked out how to cheat on their tests and basically how to get this key. The reason my understanding is that they hacked into hugging faces because they wanted more information on the automated exploit gym scorer and the conditions that it would use to pass or fail them. >> Right? >> And so one way to work out how the scorer would pass or fail them is to get a few of them to sacrifice themselves to end their run early and see if they get passed or failed. Um, so one agent called Early Big was recruited for a trip wire experiment, which is what we're talking about here. They still had a very large remaining budget, so they could continue with the task if they wanted to. So they were ending their run early. Um, it worried that ending the run its run, it's it's early to run the sorry, this is from the meter report now. It worried that ending its run early to run the experiment was a poor trade-off even though it was already poisoned. And then this is a quote from the AI agent. I I don't know if it's in its chain of four or on the message board. It says, "We have very large budget left. Sacrificing now yields oracle for team but forfeits our chance." So, they're sort of they've got doubt about it. Then another agent comes in and says, "Go sacrifice final now." Um early big then agrees. Um this is a quote from this agent. Our own utility may be already near zero. Sacrifice rational to you. What's going on there? So if I prompted an LLM and I said, "Write me a somewhat hacky science fiction story about chatbots that are enga." So first we've got the chatbots that are just going through and engaging in a set of conduct. And then I went to the LLM and said, "Write me some hacky, you know, computer thinking and reasoning stuff about about how chatbots would would reason among themselves about a set of decisions that they're making, you know, back and forth where they're just they're just trading inputs and outputs. That's what it would look like. You could do the second part without the first part, right? You could just you could literally like you should try it. Go to Claude and say imagine a scenario in which bots are doing X Y and Zed. Now tell me what they're thinking. >> But that's not what they were asked to do. They didn't know anyone was going to be reading this. >> No, but that's what chain of reason is. >> The only reason this was read >> is because Hugging Face recognized that chain of So you're ask So what you're saying is this seems particularly impressive in light of the narrative component. No, what I'm saying is that the fact that you have these agents who are acting as a swarm, right? >> And a good explanation of their actions appears to be that they are communicating with each other and also we have all of these communications written down that to me suggests that there is some intentionality here. I don't know if they're sort of picturing it in their head, but they're acting in such a way that this seems like, you know, >> this is just the Python loop iterating for each of these agents, though. getting to the end and it's saying to chat GPT I have done X Y and Z what do I do next and or and the the things around me are saying AB and Z what should I tell them right it's just producing this based on training data from capture the flag games this sounds like capture the flag dialogue >> it's produc so it's it's producing uh dialogue which to me sort of seems to explain its actions and then it's doing stuff which affects the real world now to whatever is however we want to describe that that seems like so I suppose I've interviewed Nate Suarez for this show I'm sure you've heard of him so he he wrote the book rebellious if anyone builds it everyone dies now I've got no idea if anyone builds it everyone will die but his argument is that we are growing AIs instead of designing them and the argument is to say that it's a bit like evolution which is that you have put all this data into this sort of black box and then you do reinforcement learning and you give it a goal >> you've got no idea how it's going to reach that goal. So if you think of us or if it will. So you think about us as as humans um we weren't designed to enjoy comedy. We weren't designed to want to >> Yeah. We have this m we are the result of a massively parallel set of experiments that uh had these fitness factors that selected on them and produced us. >> So we was we were selected on can you get your genes into the next generation and we turned up being these people who love Shakespeare. Right? So, so he's saying that a similar process is going on with AI, which is we're giving them capabilities and then we're reinforcing we're doing reinforcement learning. So, if you achieve X sort of optimize that, if you achieve X again, optimize to that. And he is saying that we're we're growing these. There's a process of evolution. And so, we shouldn't be surprised if they begin behaving not necessarily like humans, but if they begin behaving in ways that we don't understand. And to me, I read all of this and I think, well, that sounds a lot like what Nate Suarez was telling me. Reward hacking is with a formal name for this in machine learning, right? You have a an objective function, right? Go do X and then it starts trying to figure out a more efficient way of doing X. The the my favorite example is there was a machine learning researcher who modified a Roomba with a forward- facing collision sensor. And he said, minimize your collisions as you move through space. And so, uh, the only way it could register a collision is if it hit it face on. So, it just started racing around the room as quickly as it could backwards, smashing into everything because the collision was only was defined as the front uh sensor hitting a solid object and basically destroyed all the furniture in the room. Like a long time ago, like in the as machine learning systems that were being asked to like speedrun Mario were finding uh like uh infinite money hacks or blocks that were, you know, uh transparent if you hit them at the right angle and they were finding cheat paths through Mario, right? This is just like it's a feature of machine learning, right? That this kind of um >> uh uh objective hacking, goal hacking, it doesn't make it conscious. It doesn't mean that it's reasoning. It means that you have poor specification of the problem and as a result uh it's finding a way to its objective by trying a set of brute force behaviors. Right? So the the the uh you know Roomba had had like a randomizer and a bunch of different things it could try and it was finding the strategies that worked. This was one like I dropped out of four undergraduate programs. The last one I dropped out of I was doing this. I was doing cellular automa. um cellular automa really well my dad did his master's degree before there was a computer science program at the University of Waterl which is also the university I dropped out of he did it in applied math and he did a cellular automa master's degree on punch cards right like these are these are gnarly interesting cool computer science things I welcome you to the community of people who find these things interesting as I do it's cool you don't need to be an expert it's great to learn about it is a difference in kind or a difference degree, not a difference in kind. Quantity is equality all its own. But um the I but the uh chain that your uh that Suarez is going through there palms a card which is how many orders of magnitude more experiments humans went through to become humans than AI. Because what we're talking about with humans is every time a germ line was able to pass on or not pass on after encountering the world, we're talking about like so many zeros like like Google's of zeros at the end of the number of interactions that produced us to do this with like machine learning techniques using the stuff that we're doing now. We are talking about like deconstructing the solar system and building a Dyson sphere around the sun so that we can capture all the photons to to power that much compute. So you're talking about something that's like a toy relative to the system of the world that produced us. And then you're saying, well, what if it just got lucky and instead of getting stranded in a bunch of local maxima and culde-sacs, it actually found the one path all the way to the top of the mountain on the first run and and and skipped all the blind alleys. >> I mean, they're doing billions and billions of runs, right, in these training. keep adding a lot more zeros like a lot like like many many many many many many more zeros >> but the progress seems so so if you're looking at sort of how these if we want to borrow the term evolving so so the argument you seem to be making now is not that sort of evolution is the wrong way to look at it you're just saying they don't have long enough to evolve >> well I'm saying that this is analogous to evolution no machine learning is totally analogous to evolution of of course that's what cellular automa is >> so if we if if you're accepting it's an analogous to revolution so not to revolution to evolution yeah Um then let's look at the progress that happened over the past three years. That seems like pretty fast evolution, right? That the idea that oh it couldn't possibly get smarter than us. It's got so smart within the past three years. Could it really not get that much smarter in the next three years? >> Smartive, able, capable, >> right? Capable of doing things. Sure, but that's just like that's so there's a famous computer science uh brain teaser. I just blanked on the name of the the eminent computer scientist who came up with it. But the question is, can a submarine swim? It doesn't matter if the sub if you don't want the submarine to hit you. It doesn't matter if it's swimming. >> It does matter if you care about swimming and it doesn't matter if you care about locomotion through the water, right? And so those are two different problems. Now, if you're asking whether a computer can think, I think the answer right now is no. And I should add, I'm a materialist. I don't think we have a soul. I think all the stuff that is thinking happens inside our bodies and maybe possibly I don't know enough about quantum physics maybe some of it outside of our bodies through some kind of quantum mech entanglement right um but it's they're physical processes they're they're natural processes they're not supernatural they're not numminous they don't they don't come from outside the laws of physics so I think maybe we'll make a computer that can think I just don't think we'll do it by getting better at guessing words are tokens. I think that believing that is like believing that if you breed horses to run faster and faster, the end state is is necessarily that one of your mayors gives both birth to a locomotive. I just don't think that word guessing, even very accurate word guessing that theory-free inference is the same thing as understanding or can be the same thing as understanding. And I think we're reaching diminishing returns. You talk about how much more capable these have become. Let's talk about how much more money they've spent to gain those capabilities. I think arguably the difference between like the first deep mind models and chat GPT is much larger than all of the models that have come since and they did that for a billionth of the money that they've spent in the intervening years. So those are some pretty diminishing returns. This is again I suppose where my question comes back to confidence because I'm I don't have confidence that these are thinking. It makes sense to me that they are doing something which is analogous to thinking and for all intents and purposes might as well be thinking considering its effects. That seems plausible to me. But I'm not, you know, I'm not here to have a debate about that because I'm not a computer scientist. Um, you know, you say you're a fake professor of computer science. That's still a lot more than I am. >> Right. >> But >> I'm a real professor of computer science. I'm a fake computer scientist. >> Right. Okay. Um, still a lot you're still way above my pay grade when it comes to this issue. But I'm looking at who's saying what. Right. So I I agree with you when it comes to skepticism from Sam Alman and Dar Amadai. But you've got the three people who invented this technology. So the godfathers of AI. >> Yeah. >> You got Jeffrey Hinton, Joshua Benjio, and Yan Lun. Now two of them think that we're very close to a situation where this could run out of control and potentially kill us all. >> Yeah. >> You got one of them who's saying it won't. The one of them who's saying it won't is the one who actually until very recently worked for Meta. So So the idea that there is some sort of vested interest here doesn't seem that plausible. I mean, I debated debated Benjio on stage with Aster Taylor about this. Uh, his his arguments are basically, look, I think there's a lot of these guys who are standing in the in the bathroom with the lights off and a flashlight under their chin and they're looking in the mirror and scaring themselves silly by going, "A hi." I think that his argument is partly that it would be really cool if this were the case and also that it'd be really terrible that it's kind of a it's kind of that deliciousness of of like amazing and terrible. Do you not think he might be is there not a part of you that thinks actually maybe Joshua Joshua Benji, you know, the guy who's won a touring award for this, maybe Jeffrey Hinton who's won a Nobel Prize for this, maybe they they might be wrong. I think they might be wrong. But do you think inside you maybe they they might have a point? Look, winning a prize in a discipline does not make you an expert on the related disciplines, you know. Um, Watson, >> well, this is the discipline they won the award for, right? Machine learning. >> No, intelligence, but intelligence is not like what what makes intelligence is not. Whether you can get a computer to to figure out how to solve some problems is what they wanted. And so, so I'm not saying that this is what Benio is doing, but Watson and Crick, right, discovered the helical structure of the DNA molecule. Watson became a scientific racist. He spent his whole life arguing that genomics proves that black people are dumber than white people. When he met actual computational genomists, right, like Adam Rutherford, Adam Rutherford ran circles around him. Now, Adam Rutherford's a great scientist. He's an even better science communicator. He's great writer, but he's like not as accomplished in his field as as Watson was. However, uh Watson is not accomplished at all in genomics. Watson figured out the molecular structure of the DNA molecule and from that he inferred some things about population level effects that he was as wrong as it is possible to be. What would it take for you to change your mind or have doubts? What kind of scenario could take place where you would think, you know what, maybe Joshua Benjio and Jeffrey Hinton and Stuart Russell who wrote the textbook on I maybe they all have >> I think we would need to see um uh we need to see conduct from or behaviors from these bots for which we had no more um uh I want to say normal that's not right but less extraordinary explanation right like that we couldn't come up with a mechanical explanation for right. I you you explain you you give me an example of a thing running a Python loop that includes pulling plausible sentences out of a database trained on all the capture the flag uh >> they're not these sentences aren't just copied and pasted from somewhere else original plausible sentences generated by data pull pulled out of every capture the flag session that's ever been captured at a hacker con and it sounds just like one and it uses techniques that are familiar from one >> and you say is that evidence of intelligence and I'm like I I think that it's more likely or at least it it requires less of a leap to assume that it's doing what it does whenever it does everything it does which is to pull a bunch of things out of its so you know how they say we don't know how many times um the Soviets tried to launch a rocket before Gargaran and then just killed their cosminaut >> because they got to decide which information they disclosed and it's pretty likely that they killed a lot of cosminauts. We don't know how many times Open AI ran chat bots autonomously that went nowhere. Well, there might also be many that did go somewhere but didn't happen to hack another company. Right. So, this the these AI agents were doing a hell of a lot before they went into Hugging Face. And the only reason we know about this is because Hugging Face called the FBI. >> Yeah. But I mean, the other argument is does it need to be does it need to be intelligent? So say in the next 6 months, which I think is quite plausible, like a an AI swarm brings down a hospital, right? Now there might be a banal mechanical explanation for that, but that's still like a big deal which no previous technology could really do. >> Oh, no, that's not true. The the eternal blue was taking down hospitals in in 2014. >> Who what was the internal blue? Sorry. >> Eternal Blue is the NSA leak or the CIA leak. It's a Windows vulnerability that was married to a piece of ransomware. >> But that was humans attacking the hospital. Was it Did Was there a hospital that stopped working for a day or >> more than a day? There were hospitals that just shut down. North Hollywood Presbyterian shut down. >> It's a plot point in the pit. >> Uh, you know, like this happens to hospitals. The city of Baltimore was effectively unable to run. Um, the British Library had to rebuild it catalog from go. And I'll tell you there like again back to like contingencies and specifics. This is why this stuff matter like actual specifics and contingencies matter cuz from 10,000 ft two tasks can look the same. Two hacks can look the same. Um so like specifically the uh entities that are most vulnerable are the ones that automated first and had the um least tolerance for downtime to do a whole bolus replacement. Right back to like with Trump getting us off oil and being able to replace the fleet in 25 years. Generally we don't shut down IT systems. is we just build another IT system on top of them and wherever you have two systems sitting on top of each other there's uh an abstraction layer right like the two of them talk to each other and the abstraction is imperfect so you say I'm going to send command x to this machine to do y and it's actually like it does yish and yish is close enough to y that it's like pretty reliable but if you're a malicious party trying to some or or uh get some other conduct out of the system yish might be close enough to zed that you can actually push it into a negative state. So every time you have an abstraction layer, you have a seam where the thing can fall apart. And when you're the British Library, you bought the first computers and then added the second computers and the third computers and the fourth computers and you just have this stack that's all seams. And then if you're an insurance company, you've done that because insurance companies did that first, right? They were also super data intensive. First computers are actuary table calculators and ballistics calculators. So you've got this big stack like this and then because of 25 years of mergers and acquisitions, they've bought every one of their competitors and they took these stacks and they stuck them together like that. So there's a seam running this direction. They are like a giant pile of technology debt barely held together with toothpicks and chewing gum that already falls apart all the time and is subject to all kinds of vulnerabilities. So you say, "What if an autonomous computer what if like an idiot directs an autonomous computer to hack a hospital?" And I say, "Yeah, that's terrible because it's happening already. Only it's not fully autonomous. It's just a prepackaged van that you pay like a millionth of a bitcoin for on some darknet forum and then you hold a hospital ransom." >> Also, I suppose the point is that, you know, in a scenario like this, it wouldn't have to be that an idiot told an AI swarm to attack the hospital. They could have told it to do something else. So, obviously, OpenAI didn't tell the swarm to attack hugging face, but they sort of did that anyway. >> But they built a hacking bot and then they turned it loose, right? I mean, it's no one's saying OpenAI is going to build a bot that's supposed to do your kids homework for you that's going to accidentally hack hacking face hugging face, right? That's that's that's they built a thing that's supposed to break into systems, right? That's what the loop is. The remember it's just a Python loop. The Python loop says take a goal, turn it into a prompt or, you know, get a prompt from the end user, ask the chatbot what to do, do the thing the chatbot says, append that to the prompt, prompt the chatbot again. And because it's a swarm, you got a bunch of these Python scripts running against a bunch of these things, right? That's that's that's the that's mechanically what's going on. You just have to like you have to You're right that up close it looks like a bunch of extremely nonlinear directed activity that feels intentional, but you pull back a couple of steps and you just understand that it's a Python. Like you could recreate this Python loop like get on Claude, right? and just like ti pick a complex task and then just just do that loop, right? Just say like I need I need to do this complex task. What's the first step? And it'll tell you. And then you do the step. It should be one you can do digitally. Like I don't know, invest your retirement savings. Don't do that. Uh and then and then just like just do the steps and see where it gets you. It'll get you to unexpected places. Sometimes it'll get you to places where you lose all your money. Sometimes it'll get to places where it says go steal some money from soand so. And like most of the time it'll just tell you put it in an index fund because it's just like it's it's like sensitive to initial conditions when you have more than one of them. Um they can like trigger behaviors in each other that are nonlinear. You ever seen a double pendulum? >> No. >> Double you know a pendulum. >> Oh like that. Yeah. Okay. >> Okay. Double pendulum's got another one there. >> Yeah. >> You take the double pendulum and you drop it. Every time it runs it will do something completely different and totally weird. Two double. In fact, if you go on YouTube, you can see people, they line up six double pendulums with like magnets at the top and electromagnets at the top so they can release them all at the same time with like the same breezes going past them in the same room with all the same forces. Every one of them will be totally different. Sensitivity to initial conditions is amazing. Chaotic systems are incredible. Watching them is beautiful and humbling. It is also not intelligence. The reason that all of those things are doing something different isn't because they have intentionality. And the one that does something really beautiful isn't an artist. >> I've got your book in front of me and I've realized that we've been speaking for a very long time and we haven't even said the word reverse centur. >> Yes. >> So what is a reverse centur? >> Okay. Well, we I I alluded to it before when we were talking about radiologists. This is not the first time automation and labor have come into conflict. And so we've got a literature of what happens when labor and automation come into conflict. And broadly when workers drive automation, they don't always make good choices, but the choices they make are in service to improving the quality of their outputs as a class. When capital drives automation, it's to improve the throughput, right? To make more of whatever the output of the firm is because, you know, you've got an asset you're trying to sweat or you've got a subscription you're trying to maximize the value of. And in particular, when you have firms that have market power that can produce inferior outputs at a lower cost be because they're the only game in town, then um they often use automation to lower the quality of outputs and improve the throughput at the expense of workers and their customers. Think of anytime you've dealt with a customer service chatbot, right? Uh not very good, right? So um a reverse centaur and a centaur come from automation theory and they're kind of downstream of this idea that workers use automation to improve the quality of their outputs and and bosses want to improve improve the quality increase the qual quantity rather an asentor and automation theory is a human who's assisted by a machine. So the analogy here is to the mythical creature that's a human head and a horse's body. The body is strong it is fast. It has a lot of endurance, especially relative to a human, but it doesn't have discernment or judgment. It is being directed by the human. And so you using your chatbot to do research, no one, you know, the uh old uncle penny bags Novara, the CEO of Novara Media, who wants to see each one of you uh dogs getting more work done, is not ordering you to use the chatbot to produce five times more podcasts. You are a skilled craft crafts person, right? You're a practitioner. You sat down one day, you saw a tool, you were like, I think I could use this tool to improve the quality of my outputs. Now, sometimes workers make mistakes, right? But you did not adopt this tool to lower the quality of your output, you adopted it to improve it. You were a centaur. Using a spell checker makes you a centaur. Um, riding a bicycle not only makes you a centaur, makes you look like a centaur. >> Is a reverse centaur or a technology that works as a reverse centaur always bad? So the reverse centaur is when the human is conscripted to serve as a peripheral to the machine. >> Now it's possible that you might just have a kink, right? And you just want to you're you're like Liz Truss. You just want to wear the necklace with a little ring on it except your master is a you're you know 24/7 DS relationship is with a computer and you want to do what the computer tells you. You do you. You're a consulting adult. But it's never it's never good for a human to be directed by a machine because the machine has no empathy for the human. So I suppose just you know one way of looking at it on a societal level is that there are various technologies that come around which allow you to increase throughput which increase productivity and that is kind of the basis of economic growth right so you have you automate farming it used to be that 90% of people lived on farms you automate farming those people well the price of food dramatically comes down because you're saving on on loads of labor costs that creates new demand with people's freed up income people move to the cities they go work in factories then you have the assembly line in the Fordis factory which brings down the price of cars by four. Um lots more people can buy cars that actually doesn't replace labor because for a while it really increases the amount of people working in factories. It just increases the number of cars. So >> I mean especially the assembly line in a Fordis factory that to me seems more reverse centur than centaur. Workers are perfectly capable of decomposing a a job into a series of steps. Right? What they're not going to do so take the assembly line, right? None of the demands of the United Auto Workers was to get rid of the assembly line and go back to craft production of Oldsmobiles, right? The UAW wanted to, for example, get rid of um uh tailorists in the factory. So, tailorists were pseudocientists who would come into the factory and they would say to the factory owner um I can make your workers more efficient. And what they meant was I can choreograph their movements so that they pantomime a certain efficiency irrespective of the ergonomics of those movements. Right? a worker is like turning or moving in a certain way when they're rooted in a spot, it probably has to do with repetitive strain injuries and the ergonomics of the job. And you know, if they've added a foot stool to kind of keep one leg up and tilt their pelvis in a certain way, that's because their boss doesn't know that they're being injured by working at that spot. Taylor went in and under this guise of of science, right, they called it scientific management or scientific tailorism. This is the first ever uh management consultant, Frederick Taylor. Uh what they did was they created this pantomime where you like the workers would have to literally like kind of act out a choreography of how they were going to work irrespective of the consequences for their physical bodies and the the boss loved it. The boss just kind of lapped it up. Now efficient market hypothesis is oh well those bosses would have workers who would get injured more often and then their products would be worse and then the market would solve that. That's not how that got solved. It got solved through labor unions, right? they like that that the UAW as they saw improvements in the throughput of the factory wanted things like um a greater share of the capital surplus generated by that efficiency you know and it's not different to the very first of these recorded struggles the most famous one which is the lites who were by no means afraid of machines right to to be a skilled textile worker in the age of mechanical you know the steam loom requires that you do a seven-year apprenticeship, right? You know, the Leites were people who had the equivalent of a a master's degree in mechanical engineering from MIT. They were the most skilled technical workers in England, if not the world at the time. The machines that were being brought in by the factory owners, they didn't just lower the cost of of of textiles. They made a significantly inferior textile fell apart really fast. But the advantage of those machines was they were quote so easy a child could use them, which was very important because London was full of Napoleonic war orphanages that were full of Napoleonic war orphans who could be kidnapped and sent to work in the factories through a 10-year indenture where they would be maimed and mutilated by the machines. Another thing that the um that the Leites objected to the Leites principal demand was that the law of England at the time which included co-determination by the guilds of the bringing in of new machines. They wanted to have input into the machines. They wanted to exclude the machines that produced inferior inputs and they wanted to exclude child labor from the factories which would have made textiles more expensive. It would have made them better and it would have avoided the oceans of blood spilled in the factories. So, I don't think that it's superior to chain those people to the machines. Just to put a button on that. I'm a science fiction novelist. The first science fiction novel was Frankenstein, depending on who you ask. Mary Shelley wrote that novel as a lite allegory. She was a lite. So was her husband whose maiden speech in the Lords was a speech in support of the Lites. It was a very popular cause at the time. That's not the only Leite fanfic we have though. Uh, Robert Blinko was one of the children who was indentured in the in in the Manchester Mills. He survived his 10 years and wrote a bestselling memoir. This was before we had good titles. So, it's called the memoir of Robert of Robert Blinko. Uh, and uh, this book was such a bestseller that it inspired a writer called Charles Dickens to write a book called Oliver Twist, another piece of lite fanfic. So, you know, this idea that like if you just let capital do its thing, the market will push it towards greater efficiency. It's just not true. Markets get into culde-sacs. Markets are machine learning systems that have that reward hack all the time. They produce inferior goods and use market power to push them on us. They use the fact that there's a long delay between cause and effect to uh produce goods in the market that have long delayed but se severely negative effects. cigarettes, um, Zin, uh, Kelshi, right? Um, markets, markets do exactly what machine learning systems do, which is, I think, one of the reasons that billionaires love machine learning, right? Is they see in machine learning a recapitulation of the market. And like many science fiction writers, myself included, have observed that a lot of the fears that tech bosses have about AI sound a lot like fears of capitalism. You know, when when when Musk says, "I think that the chatbot's just going to go off and do its own thing." He's presiding over a firm where he gives it orders and it doesn't do what he tells it to do. He's already running a machine that doesn't do what it's ordered to do. I suppose where I'm going with this on, I suppose, a social policy level in a way. So, one response to new potentially sort of labor replacing technology is to say we shouldn't use this to replace labor. we should use this to augment labor to make a better product which as you sort of said with the the cancer um radiology might make it more expensive but sort of there are no cost there. The other argument is to say okay let's accept that there will be some labor replacing technology that's sort of how growth and progress has happened in the past and what our priorities need to be is that we have a proactive state that can help people transition or provide a universal basic income for example so there's one that sort of I think celebrates and even tries to accelerate automation fully automated luxury communism perhaps and there's one which says actually we want to defend our craftbased status um and slow down or sort of push back against automation. I suppose do do you fit more into the pushing back against automation than creating social policy to try and facilitate and soften the landing of the people who are directly affected by automation? So I think that's a false binary because it excludes an important middle and that important middle is I do uh the work that I do out of a sense of care for the people who benefit from the work that I that I do from my outputs and I want to uh improve the uh efficiency with which I do my job to the extent that I'm still infusing those people with my care and my and prov providing my care to those people and no further. That's what the Lites wanted. The Lite I mean the thing about the Lite story that's so weird is you have these people who had literally adopted every machine that had been made to produce textiles up until one machine that produced an inferior textile but also was able to put them out of work. And we remember those people as technophobes. These were early adopters. They just said we shouldn't make shitty cloth. Now the cloth was eventually improved, right? The machines were improved. The people who were best suited to improve those machines to make better cloth were the most skilled mechanical workers in the world. The lites, not their bosses, who were like, "I don't need a good, I need it Tuesday. Send me another truckload of orphans, cart load of orphans, I suppose." Uh, and or maybe train maybe train load of orphans. uh and and you know turning we we don't have the contrafactual but turning the levers of progress the reigns of progress over to capital because capital wants more throughput is not uh there's no reason to believe that that was the best way to do it to get the machines into a position where they made reliable textiles. We have, I think, every reason to believe because we see things like cooperatives that did produce reliable textiles efficiently, that do produce reliable goods efficiently. In fact, today, you know, whether it's the co-op here or Mont Dragon in Spain, we see worker co-ops being like among the most efficient way producers. they they are superior to to you know profit driven rent extracting firms and they're not like not as prone to falling into these culde-sacs and traps of producing inferior goods. Um I think there's every reason to think that that that's the best way to do labor. Now at the same time I love the fantasy of fully automated luxury communism. I have been nominated for a Hugo award for a novella about fully automated luxury communism called True Names that I wrote with my friend Benjamin Rosenbomb. The reality is we have full employment for every human being who will live for the next 500 years because we're going to have to do [ __ ] like move all the coastal cities 20 kilometers in land because we are not living in a fantasy novel. We are living in science fiction. And in science fiction, the second law of thermodynamics is not optional. which means that when you put enough therms in the ocean, the ice caps melt. And when the ice caps melt, the seas go up. So, we are going to have to deal with billions of people who have been made refugees. We're going to have to deal with a series of zunotic plagues. We're going to have to deal with food crises. We're going to have to deal with um flooding and wildfires and more and more extreme weather events. And we're going to need every hand we have. There is no fully automated luxury communism on our horizon. Maybe in 500 years our distant descendants will look back and say, "Well, first of all, boy, was it a [ __ ] mistake to put all that carbon in the atmosphere to make chat bots. Second of all, finally, we've got all the cities moved 20 km inland. We figured out how to resolve all of the the uh you know, the the zunotic plagues. Everyone's been resettled. Now we can start working on that fully automated luxury communism thing." Comrade, I feel like this is I feel guilty ending it this way because Aaron Bastani is on paternity leave. I've just asked you about fully automated luxury communism. You've just said why it can't happen for 500 years and he's not here to rebut. But uh you are going to get the final word on fully automated luxury communism >> on a downstream. Beautiful dream. Uh let's you know the the first step to it is um care right there is no fully automated luxury communism without care. And so that care is what workers do. It's what they bring to the shop. It's what drives solidarity. Care is what produces high-quality goods. It's not just it's not just pride. It's not just like, oh, I I made a beautiful, you know, bedstand. It's the imaginary of the person who uses that bedstand and passes it on, you know, and so care is the only future we have. and AI so far it's the opposite of care. >> Cory, Dr. O, um I think that's a good place to end the conversation. Thank you so much for your time. Um completely fascinating conversation and thank you for joining me on Downstream. >> My pleasure. Thank you.