Are We Headed For AI Utopia Or Disaster? - Nick Bostrom

it seems like your book Arc has been moving from what if things go wrong to what if things go right is this some requisite hope in the AI discussion well I I think both Barrels have always been there it's like last time I published a book it came out of one of the barrels the kind of Dooms side but uh uh I no I I think uh yeah both The Optimist and the pessimist are kind of co inhabiting uh this brain is that a uh is that a difficult balance to strike the fact that you need to be so chronically aware of the dangers and so chronically aware of the potential successes as well I think that's just a predicament um that we are in um and if you look at the distribution of opinion sort of roughly half fall on one side and half the other but in many cases I think it basically just reflects the personality of the person holding the views uh rather than some kind of evidence derived opinion about you know the game board and so um yeah if if if one takes a good hard look at where we are with respect to things I think one soon realizes just how ignorant we are about a lot of the key pieces here and how how this thing works so um so certainly one can see quite clearly uh significant risks and in particular W with this rapid advance that we're seeing in AI um including I think some existential risks but um at the same time if things go well they could go really well and I think that as long as there is ignorance there is hope so we have a lot of ignorance and also some hope it's interesting that uh your position whether you're a AI Doomer or an accelerationist or whatever uh is at least in part just a projection of your own sort of internal bias and mental texture that you sort of see in AI development uh the way that you see the world I think there's clearly a good deal of that and then which tribe you happen to uh belong to like depending on who you run into what or which Twitter threads you follow like uh then we are kind of her animals and sometimes uh it almost becomes a competition who has kind of develop the most hardcore hardcore attitude you know I'm so AI pilled my P Doom is above 1.0 like um yeah yeah it's and conversely on the other side um but we we yeah we need we need to I think do better than that uh if if we're g to like intelligently try to notch things uh towards a good outcome here certainly at least from my seat and from reading your book probably about nine or eight years ago uh I've been very conscious of how things could go wrong and that at least in my corner of the internet maybe this is just my Twitter threads and sort of my echo chamber uh has been the sort of more dominant narrative what what does it mean in your opinion to live in a solved world like what what would it mean for us to get this right with AI and come out on the other side of it uh yeah to I think there are kind of three big areas of uh challenge that we'd have to navigate on top of all the more near-term and and present issues that obviously are also important but just uh not the focus of a lot of my work but uh definitely we need to solve those as well but yeah I think there is the um alignment problem uh which kind of was the focus of my previous book super intelligence came out in 2014 uh which is basically the challenge of how to make sure that as we develop increasingly capable AI systems and ultimately uh super intelligences how we can make sure that they are aligned uh with the intentions of the people creating them so they don't sort of run a mock or do something uh antagonistic against humans and that's fundamentally a technical problem um back when superintelligence came out this was a very neglected area certainly nobody in Academia was working on it and hardly anybody else either like a few people on the internet had started thinking about it but there's been a big shift and now all the frontier AI Labs have research teams trying to develop scalable methods for AI alignment and and many uh other groups are also doing this um and I think I mean remains to be seen whether will be successful at that but that's certainly one thing that we need to get right um and then there is a broad category of what we might think of as a governance problem um which intersects with the alignment problem as well but uh also has other dimensions so even if we could control AI we need to make sure that we then use it for some positive end as opposed to say Waging War or oppressing each other or doing all kinds of other nasty things that we use other Technologies for in addition to uh positive purposes and so that's like a broad category but uh very important um and then I think there is a third area of challenge which has so far received much less attention uh you could say that it is now where this alignment problem was 10 years ago that is a few people are thinking a little bit about it but it's outside the overturn window and this is the um uh challenge of the ethics of did Minds that we are building these digal Minds that might have moral status and uh so in addition to avoiding AI harming us or US harming each other using AI tools we ultimately also need to make sure that we don't harm uh AIS especially AIS that are either sentient or have other properties that makes it morally significant how they are treated um so I think yeah each of these three is uh is is really a key to having a future that that that is desirable yeah what what is there to know about the moral status of nonhuman intelligences um well um there's a lot to know I think that we don't yet know we do know that historically we can see now uh that there has has often been uh a tendency to denigrate out groups of different kinds I mean it might be human out groups of different you know the tribe across the river or or people from other backgrounds or countries or races or with different views and religion and so forth this kind of human history is like a a kind of sad Chronicle of of how easy it is for us to fail to recognize the moral significance of other entities that is serve this um and in today's world I mean if we look at how we are treating a lot of non-human animals I think that leaves a lot to be desired in in um uh factory farming uh and so forth um and so um as we develop these of increasingly sophisticated digital minds I think it will be a big challenge uh to to extend moral consideration where it is do um it could in some ways be even harder than with animals animals have faces they can squeak um whereas some of these digital Minds might be invisible processors occurring in in a giant Data Center and uh easier to kind of Overlook what is going on in there but the future you know might well be that ultimately most mins will be digital and so um it it could matter uh a great deal how how good the future is for them but but it's a it's a difficult um topic even to figure out what like suppose you agree that this we should we should try to treat them well like it's not at all obvious what it even means to treat an AI well and there are so many different kinds of possible AIS so that um maybe the right way to treat them is very different from how we should treat humans yeah do they need do they need a weekend off should should we be polite with them yeah they might need things that we need that have no need of they don't need food right and maybe they have other needs like electricity but but more fundamentally you could have all kinds of um very different types of entities where we can't just sort of export the moral Norms we have developed for how you should treat human beings and automatically just kind of yeah apply those to AIS um is consciousness necessary for moral status my guess is no I think sufficient but not necessary if you have the ability to to to to suffer and experience discomfort I think that gives you at least a certain degree of moral status Can You Suffer Without Consciousness um I guess depends on how you define the word but I think if you do have that ability to suffer yes then you have moral status but I think um you could have moral status even if you don't have that um if you imagine some very sophisticated digital mind that maybe like let's suppose you think it's not conscious for whatever reason but it has a conception of self as persisting through time um it can have long-term goals like maybe a life uh plan and things it wants to achieve it can maybe form reciprocal relationships with other humans um uh and I think in in those cases there there would be a primar fascia basis for saying that there would be ways of treating the system that could be wrong um that would infringe on its interests and its preferences um but it's not at all obvious like if moral philosophers are thinking about the grounds of moral status like there's a range of different views so so it's not as if you know I'm I'm 100% convinced of that it's interesting to you know it kind of gets us toward a p zombie or I guess like a V Zombie now like a virtual zombie um of if a system is able to tell us that it wants to continue working toward its goals and instrumentally it wants to build relationships with other people and it has a sense of where it's been before and a trajectory of where it's going to go next all of these things are the things that you would guess well if a human told me that or if any other creature told me that I would guess that they have the capacity to suffer because if I stop them from doing the things that they want to do then Downstream from that is discontent and suffering but the whether or not there is some sort of phenomenology of being like that thing inside of there is going to be very very difficult to work out maybe impossible like you know it is impossible for me to know that you this is an awesome Truman Show and everybody here is an actor and all of the pain and all of the joy that everybody around me has ever known for the rest of time hasn't just been part of some big prank or some simulation yeah I I think like we have a a very weak grasp of what the uh criteria are for some system to be conscious and I mean historically we've had smart people who' thought animals are just automata or or even uh uh certain other people people thought oh they're you know more like animals and if animals are autonomous and like and so it's very easy to delude ourselves when it is convenient uh that there there is some like magic ingredient that that we have but then this other set of entities don't have and so we should be a little suspicious of that I think um and yeah I mean the metaphysics of Consciousness is not notoriously um controversial and uh hard to pin down I'm my own views are kind of towards the computationalist direction I think what makes a a system conscious is that it implements a a certain structure of computations and in principle those can be implemented by an organic brain as in our case or by a silicon computer um and in either uh in either case if if the computation is of the right sort that would be conscious experien is supervening upon it um but um I think it if if I'm right that there are these alternative bases for moral status then we wouldn't necessarily have to solve that question um before we could uh hopefully I agree to try to be nice to these uh digital Minds that we're building um but yeah it it's really uh really hard to to know what that would entail in Practical terms I think there's a lot of theoretical groundwork that needs to be done before the the time would be uh ripe for like trying to pitch policy makers to do a specific thing right now even if they want it to do I'm not sure what I would concretely recommend I I think there are like small little things that maybe one could do um today like lwh hanging fruit that cost very little that possibly for example some of these most advanced AI systems you could like save it to disc when you no longer need it um and then at least the possibility would exist in the future of like like rebooting it and doing things with with some of these large language models I it probably would make no difference at all to to their welfare and maybe they don't even have welfare but you could imagine somewhere in this meta prompt like the part that you the user don't see but that open AI or anthropic are putting in kind of as a Prelude to uh the text you're inputting like there's like a whole bunch of stuff they say like try to be helpful you like uh don't say offensive things and and be truthful and careful in how so there's all of that you could imagine adding to that like a line saying oh you're uh you're you're waking up in a really happy mood and you're excited to enjoy yourself try and have fun today yeah so that that might cost like one extra line which is like trivial and you know maybe possibly that would you know increase the chances that if there is some sentience it would be positive rather than negative um but I I these are kind of we weak weak ideas so that probably wouldn't make any difference but I think it's there could be some benefit to at least doing something even if it is ineffectual just to set the precedent to sort of um put the flag in the sense saying yes now right now we don't really know what we're supposed to do we doing something maybe it's mostly symbolic and then over time we can like think harder about this problem and and hopefully come up with better better approaches um yeah there there are some other things you could imagine doing that also probably don't really help very much but like you could refrain from deliberately training the systems say to deny that they have moral status um so right now if you're a big tech company it might be quite convenient just in training to sort of like whenever you're asked about this thing you should always say x y or Zed and it might might be um better to to have a norm where you don't deliberately try to bias the systems output in those ways just in case um we could get any kind of information from self-reports that's like the main way we ask like figure out whether another human uh likes what we are doing or not or if if they are aware of something like we ask them and now now we have AIS that can actually speak and so it makes sense to maybe use that language interface as one of the ways in which we can explore this but that only works if you don't like during training deliberately kind of destroy whatever signal there might be in their verbal output because it's trivially easy if you want either to train them to always say they are conscious or to always deny it or to say that they are happy to do what you want or to always deny so like obviously if you specifically train them from that then you probably can't learn anything from what they end up saying all right so getting back to the potentially solved beautiful utopian future what are you talking about when you say Utopia what is Utopia by your definition what are the different types well I mean so there's like a kind of literature uh of utopian writings the historical we see usually they are um like attempts to uh depict some supposedly better way of organizing Society um and and normally uh the result is uh not actually a society that you would want to live in and in the cases where people have actually tried to implement this they has usually ended in tears and so there has grown up I think a healthy skepticism about these attempts to try to like think up some great blueprint for society and then especially if the idea is then that you're supposed to use corve methods to sort of enforce it on society like the self-appointed social Engineers who would be doing this uh uh are are likely to do a lot more harm than good um um but um there is also like a dystopian literature which is kind of just the flip side of that that's often a lot more convincing like it's easier to say here is a possible Society we can all agree this would be really bad um and and there's like a number of these that most people would like the you know 1984 Brave New World handmaid's taale like all of these um and sometimes those are meant to also have a kind of political agenda they might be critiquing some tendency that exists in our current Society than saying well here if we take that to an extreme and sort of scale it up you can now all see that this would be bad so let's reflect on what we're doing doing and maybe we can avoid going down the path of you know Brave New World or something um um but um this book deep Utopia let me promote it Publishers happy yeah there it is um it doesn't talk about that at all it's like not about the Practical problems between here and it rather like assume for the sake of argument that everything goes as well as it possibly could with the whole AI transition Etc so we solved the alignment problem we solved the governance problem to whatever extent it can be solved but like no Wars and no oppression etc etc and in order to get to the point where you can then ask the question um what then if we do end up in this condition where all the Practical problems have been solved what would um what would we humans then uh do like what would give us meaning and purpose in life if AIS and robots can do everything much better than we can do um and um yeah if we then attain this condition of technological maturity that I think this machine super intelligence would relatively quickly bring about um and the they kind of layers is it like a like a like an onion as you can sort of think about this problem at various levels at the most superficial uh you have this oh well you know the AIS will automate a bunch of jobs and so then um you'd have some unemployment you know and maybe You' have to retrain people to to do other things instead just as everybody used to be farmer and now like almost all the farming jobs are gone but people are still working and so so that that's kind of I don't know you might think layer one and and that's often where the discussion stops uh so far um but you can sort of think this through uh from that point on you say well like if AI really succeed then it's not just some jobs but basically all jobs um that become automatable um you know with with a few exceptions that we can talk about if we want and so you then would end up in this kind of postwork condition uh where humans no longer need to work for the sake of uh earning an income um so so that's already a slightly more radical conception right it's not just that we need to retrain people to you know become whatever new weird occupations that but it's like yeah that that whole thing is the concept of occupations overall yeah we would enter this condition of of like Leisure um but there are various groups of humans who live lives of leisure and we know we can look at it's like there's like children like young children before like school is a kind of job for but like before they start going to school right okay so they they don't work for a living they they are not economically productive they still in many cases seem to have great lives um you know spend all day um playing and uh having fun and uh you know eating ice cream and uh you know all kinds of stuff like that that could be the lot so that's like you could look at other retired people people born to great wealth or or kind of monks and nuns like anyway so there are various templates of oium that you could uh but but that that's still I mean maybe that's like the second layer of the onion but it's still relatively superficial um so if if that's where we stop then you would think well you know then maybe we need to develop a a Leisure culture to kind of maybe change the education system so rather than training um the young uh to sit at their desks and receive assignments that they then work diligently on and hand in and do what they're told that this is a great training for becoming an office worker right like but in this world we don't need any office workers so we could instead train them to develop an appreciation for the Finer Things in life to practice the art of conversation right develop Hobbies appreciation for art and literature and poetry and film and imagine how radical imagine how radical that would be to have a school teaching people how to live well or find fun uh yeah it it would be uh I mean I'm thinking my school I always kind of I I don't know what how inspiring that would be if they had been trying to teach but you you like in theory at least you could imagine sort of Shifting the culture from this focus on being useful and economically productive to actually uh uh Living Well um which would make a lot of sense if that's like the condition we end up with um and I I I think hopefully that there would be great scope for um yeah like a much better type of human existence and which might then look back on the current ERA as like a kind of barbaric like the way like you know um 0 like 17th 18th century child labor in mines working 16 hours a day they might think of Our Lives as correspondingly kind of blighted by by kind of for for many people going to like a boring job that gives them nothing other than a paycheck but they have to do it because they need to pay the rent and um but but I I think that there are like farther layers to peel off here so once you start to think through this condition of technological maturity you realize that it's not just our econ e omic labor that could be automated but a lot of our other efforts as well if you think what people do uh with their Leisure when when they don't have to work for a living like there's a lot of things we fill our time with that require some effort and investment uh um that you may madine like well if you didn't have to work you could do these other things like you know some some people like you go go shopping I don't quite understand but some people think that's like a wonderful activity and but is that you think like in in this scenario where you have technological maturity right you would have recommender systems that could pick out something much better than what you would pick out yourself if you went they would have like a detailed model of your preferences and be able to predict and so although maybe you could still go shopping you would know that at the end of of this thre hour running around with with plastic bags or what you'd end up with something that was worse than if you had just let your AI do the thing for you it could select and also bring it to your house the buy now yeah yeah exactly and so so you could put in this effort but the end result is worse uh and and it seems like that would put a little question mark over the activity you could still do it but would it still feel as fun and meaningful because I I think a lot of uh the struct activities we do now have the structure that you do X like put in some effort and work in order to achieve y something outside the activity but in in at technological maturity there would be this shortcut to Y and so that you could still do X but there's like a kind of pointlessness maybe like a shadow um and um a lot of activities I think have that structure uh mean you could think of uh like spending time like child raring seems like a worthwhile important thing that give a lot of people meaning but if sort of dissect it and look segment by segment at what it means like is the changing of Napp is really something that you think is intrinsically if you had a robot you could do it just as well be pretty tempting just to kind of uh press the robot on button and it would do the thing and um and so so a lot of that I think would yeah um like go away I'd lose its appeal if if if that were these shortcuts um so so that that's like another layer um um but there are more layers so you then think well certain things like I mean you um you like Fitness and so you want to like you can't rent a robot to run on the treadmill on your behalf right that's like you definitely can't automate that you think but well at technological maturity you could pop a pill that would induce the same physiological effects in your body as like one and a half hour of uh sweat and toil in the gym um including the psychological effect of kind of feeling relaxed and energized and so if if that were the case then yeah then does it still feel like appealing to to do the hard workout if you could just achieve exactly the same results by a pill and um so I think what you have is first kind of post-work condition we talked about earlier and then there's this broader uh condition of post instrumentality that all the things we do for instrumental reasons with a few exceptions but yeah those uh would also become obviated it seems and and and now we have this further affordance which is a condition of plasticity where we ourselves the human body and mind are psychological States becomes a matter of choice we become malleable like at technological maturity you would have various um you the crude version might be very strongs without side effects that have very tailored effects but you could also imagine more direct kind of neural uh technology that allows you to have fine grain control over your mental States and cognitive States and emotional permanently Bliss out with some microscopic node that is able to manipulate your brain in some way or Chang your brain or all of your fears and anxieties are gone and all of your worries and concerns are gone and you're just at this sort of peak MDMA State and then if you don't want that anymore it knows and it's able to create a state that you couldn't even think about and there are no constraints for you to do it either right yeah so that that that that I think will become possible at technological maturity and so then all these activities that you currently do say maybe you do them because uh it gives you Joy uh and pleasure and happiness um they also would be unnecessary in that there would be this shortcut to to Joy and and pleasure and happiness that like the direct brain manipulation um so so you you you you have this quite radically different condition where it's the world is solved in the sense of the Practical problems have been taken care of but also in the sense of maybe dissolved in that a lot of the uh fix points and and hard constraints that shape our current lives are kind of solved in this Tech solution of of technological advancement um and then then so that then we kind of get to like really the heart of the problem that the book is trying to think about is like in such a condition what would a Great human life look like what would could we actually achieve in terms of realizing human values if we had all of these affordances all of these options it's so strange the the thing that reading the book that stood out to me is how much of what we seem to value and take pride in are kind of like clever ways to deal with scarcity and the fact that much of what we do is instrumental to striving in achieving some future goal which requires effort it seems like in a sense much of human philosophy and value is just negotiating with a world which is effortful and constrained and we are trying to find ways cognitively to deal with this sort of pressure that we have to lean up against in order to cool the world to deliver the thing that we want yeah um yeah these practical Necessities have have been with us since I mean through the entire history of the human species and indeed beyond that it's like the Human Nature has kind of evolved and been shaped uh in a condition where this is always present there are all kinds of things we have to do and cope with and struggle against um so it's almost like if you think of um like a little bug that has an exoskeleton right that's uh and then it holds the squishy bits inside together but if you imagine removing the exoskeleton there's like just a blob there and and similarly the human soul might have as an skeleton All These instrumental Necessities that that that are like Evolution can just assume or present because theyve always been but if you were to remove those then What Becomes of the human soul and the human human life does it become a kind of pleasure blob or is there something that could give structure to our existence even after These instrumental Necessities are removed so much of what we seem to take pleasure in as well is the absence and then satisfaction of some some desire there is a thing that we want we don't currently have it and then we get it and then it gives us something we work hard to uh achieve a body that we're satisfied with we are thirsty for a while and then we get a drink we want to have sex and then we do we are looking forward to having a child and then it's born we are all of these things are on the other side of something and yeah if like you do X to get y but you can just always immediately get Y without having to do X it does ask the question of where does the absence that there are no longer any absences Do's this quote something to do with uh in a perfect world The only desired uh the only lack would be for the want of lack itself which is this sort of the absence actually makes the presence of something finally valuable and if you don't have any more absence then what does all of this presence kind of mean yeah um I I think there might always be a whole bunch of absences in as much as human if not human need at least human desire or at least some human desires are kind of unlimited uh it's maybe most clearly seen if if you have two people who want exclusive possession of the same thing are two people each of whom wants to have more than the other like I think two billionaires who want to have the world's longest yacht and so one has like a 150 meter long yacht and then the other has a builds a slightly bigger one that that like is kind of intrinsically unlimited because um there's no way that they could both have everything they want and so there might be a bunch of um um desires like that that are quite common that could never be completely fulfilled or like just imagine somebody who is is like utilitarian and who wants there to be as many happy people as possible in existence but say like however many they are there could still be more and so they would always prefer to have more resources um but um even if there are some such desires it wouldn't um give these uh future utopians necessarily any reason for uh uh laboring or exerting effort because there might just not be anything they could do themselves to increase the degree to which these desires are satisfied uh I mean the person has a trillion dollar maybe they would want to have two trillion dollars but they can't actually make more money by working because all the work is more efficiently done by machine um so yeah even with unlimited desire you might still have this condition that is both post-work and post instrumental do you think that humans would be run the risk of getting bored in a Utopia not if they didn't want to at least if you by boredom refer to a subjective state of uh I don't know like some kind of restless uh discontented uneasy feeling of having hard like having a difficulty keeping your focus or like so that certainly would be amongst uh the things that could be trivially be dispelled through advanced neurot technology I mean you already have like drugs that could do it for a limited period of time now with effects and then they wreck your body but like it's easy to imagine how you could just uh have have better versions of that that would make it possible for you to always uh feel extremely interested and excited and motivated um and in in fact some people have I mean there's a lot of variation amongst humans um and I mean I I I have a have a friend who uh uh tells me he's never Bor and I believe him I've never seen him boarded he's kind of interested in in everything um except sport um and like he writes papers on all kinds of different weird topics and is like just constantly excited about learning new things and um you can have a conversation with anybody about anything and he's like really so um it's an existence proof um it's possible to to be that kind of being and and in the future we could all become such beings if we want to um so subjective boredom would be like trivially uh easy to dispel under the condition now it is possible also to have an a more objective notion about boredom or maybe we say boringness to refer to this objective notion which is the idea that certain um um activities are intrinsically uh boring like meaning maybe that it is a appropriate to feel bored if you were spending too much time doing them it's like kind of maybe an open question whether this notion of objective boringness makes sense but you might think of like say say counting Blades of grass on a lawn like suppose you had a a being who found it extremely fascinating and like an a NeverEnding source of Joy uh to just count uh and recount the blades of grass on a college lawn somewhere um you might say that although subjectively he's not at all bored uh that objectively what he's doing is boring it's un it's like no variation or significance or development and the appropriate attitude for somebody to have if they were spending their whole day doing that would be to be subjectively bored um if you have this notion of objective boringness then it becomes a much less trivial question to ask in this hypothetical condition of a solid World um like would it be possible for us to avoid objective boringness like yeah we sure we could like engineer ourselves we always felt interested in what was going on but would we be doing anything in appropriate like with our circumstances be such that uh the appropriate attitude would be to be bored and so um there's a big discussion about this but yeah so and I think it's um um I think there are certain forms of interestingness that uh we might run out of um for example you might think it's especially interesting to I don't know be the first to discover some important truth like Einstein discovering relativity Theory might be like a kind of Paradigm case of like an extremely interesting Discovery and experience um but it it's possible after uh some while that uh most fundamental important insights about reality that we could have we already have had and in any case the machines will be much better at doing the discovery than than we would be and so we would kind of uh run out of the opportunity to achieve that kind of interestingness in our lives yeah I mean so so much of what Humanity's done has been chasing down answering big questions so where do we go when all of the big questions have been answered yeah I mean I think fortunately um for the most part uh it's not what we're actually doing in our lives I mean uh most people are not spending most of their time trying to chase down the answer to the big questions right like most of the time we're just going about our daily business um and you could make the case that already I mean if you really look at it I mean from a kind of unbiased outside like if the alien super brains came to Earth and thought okay so look these guys are worrying about uh losing out on what's interesting in life well let's look at their current life and see how interesting there is how many yes like how many times did this guy brush his teeth okay well 40,600 like how interesting is it to brush your teeth for the 40,7 7 60th time and then um all right so he commuted into the office and then you know he ate a steak okay I mean how interesting is it to do that again and again and again yeah and even the big highlights in our lives like I mean maybe they are like really novel and exciting if your scope of evaluation is a single life like the first time you see your own newborn like like that's like he just happens once right like so there a few of those but if you zoom out and look at Humanity it's kind of well it's already been done you know tens of billions of times um you know like how different is this particular newborn from all the other newborns uh um so depending on how you look at it you might kind of either think that we are already like at a very low rung of the ladder of objective interestingness or if you sort of shrink the focus of evaluation enough to a single life or perhaps even just to a single moment in single life then yeah then then there is like more novelty but also opportunities for the same kind of thing to happen in in Utopia like if if if you're just looking at the most interesting possible moment and you don't care about whether similar moments have existed before or after uh then you might think the average human moment of awareness is very far from the maximum of interestingness what what do you think think would happen to religion that's obviously a place that an awful lot of people take their meaning from currently is there a place for religion in a deep Utopia yeah so this is uh uh one of the things that possibly survives uh this transition to a Sol world like which could remain um highly relevant uh even like if we had all this fancy technology um and it might constitute a bigger part of uh people's lives and attention than than it does today because there would be fewer other like distractions if you want what else what are the other areas that are uniquely human or that would survive this transition well um yeah so I mean you can kind of Build It Up starting with the most basic value perhaps which is just this share um uh subjective well-being pleasure enjoyment um which obviously would be possible to achieve in Utopia and and not just achieve but like you could have prodigious quantities of this place um and so that's intellectually not maybe super exciting to discuss at Great length but I think actually super important like it's easy to dismiss oh these are like some sort of junkies just having their like heroin drips or but but the key question here is not like how exciting is this future or how admirable is it from our point of view as if we were sitting in the audience like evaluating a stage play like that's one perspective and then we want a stage play with a lot of drama and suffering and tragedy and overcoming and heroism and right but the question here is which future would you actually want to live in uh and and there like one of very great levels of subjective happiness and well-being um might ex that might be the most important thing about the future impact and you could definitely have that uh in in like extreme degrees so so that's like it's worth making a note of that like and let's put that in the bank at least we could have that and that's already possibly like According to some people it's the only thing that matters like if you're a hedonist a philosophical hedonist um but for most other people it's at least one of the things that is important even if not the only thing that is of value so so that's the first thing then then you could add to that um experience texture so it's not the case that you could only have subjective well-being you could attach that to some um intricate complex uh mental state that relates to some important like so for example you could experience the pleasure not just as a sort of unanchored sensation of well-being but you could attach it to say the appreciation of aesthetic Beauty um appreciation of you know great truths or great literature or or contemplating the Divine and that that's what you derive the pleasure from so your conscious state is one of insight let us say uh or or understanding or appreciation of things that deserve to be appreciated and understood um so some people think that that is also a locus of value not just the hedonic like the scale of whether it's like plus 10 or minus 10 but having plus 10 whilst you're like understanding or seeing or appreciating something that is like actually lovely and worse understanding or profound makes that a more valuable condition so you could have that um then um if we go to some of the other values that seem more at risk like purpose um you could have you could certainly have um artificial purpose um so you could set yourself goals um in Utopia in order to then enable the activity of trying to achieve them like playing a sport against another person yeah so so games is a a parad example of this in today's world you like set yourself some arbitrary goal like maybe to to get the golf ball into a sequence of 18 holes using only a club and there's no other reason for why You' need to achieve this goal right other than to enable the actual activity of golf playing um and that that could become a much larger part of the utopian lives various forms of game playing like you could make all kinds of new much more sophisticated and immersive games uh alone or with other people that that involves setting yourself um sort of arbitrary challenges or at least semi- arbitrary challenges simply in order to then create an opportunity for the activity of striving to achieve them um and yeah so to enable play um we we kind of like deliberately limit the means available to you to achieve this arbitrary goal so just as a golf like yeah you're in the the post scarcity world you press the button and there's never any doubt about whether or not hitting the ball goes into the hole which makes the hitting of the ball completely arbitrary yeah and uninteresting and maybe objectively boring but you could just set the goal to achieve this uh sequence of outcomes the ball falling into the different holes whilst also not availing yourself of various shortcuts that that could you could sort of make this more complex goal of achieving x y while only using means correct there needs to be some constraint which then gives a degree of satisfaction when you achieve it yeah yeah and like either to give you the satisfaction I could achieve the satisfaction just directly through the newer technology but if you also in addition to the pleasure wants to have the experience texture and you want to have the sort of effortful activity and striving then yeah you could achieve that by having these artificial purposes it kind of feels to me like you you very quickly keep coming back to the same question of is there a quicker route to achieving the outcome that I'm trying to achieve here um I I've had it in my head since you were talking about this and since I read the book about churning butter so there's very few people that would look at the butter that they use now and think I know that it's here and it's convenient and tasty and does what butter needs to do by being lovely on bread or whatever but I feel like it would have been more meaningful to me if I'd gone out into the field and the cow and done the thing into the bucket and then churned it and then got it and then put it in the fridge and all of those steps so we can see and you know there's kind of this um inherent sense this sort of naturalistic fallacy that this this is taking us away from what it means to be human that the set point that we have grown up in this is uh it's a misalignment evolutionarily uh there's something sacred about the process of Being Human it imbus you with meaning to go through the challenge and the struggle beforehand but there's very few people that would make that argument about churning butter and when you think okay so if you are happy with more convenient butter I think that something right now which is assumed by most people to be natural but in future we'll be looked at as barbaric we'll probably be driving our own cars I think that in 50 years time 100 years time it'll be like if you looked at someone riding a horse down the street now you go I mean isn't that so and wild that people used to do that that was the way that they got around and you know they had to have these special people in New York that would sweep up all of the muck from the street this entire industry buildt around horses so again right now something that we can almost begin to see the transition of we're about to let go of this thing which is a less efficient less safe more effortful way of getting us from A to B and yet some people I love driving I take great pride in D some people even compete in it you know F1 is an entire competition around people that are driving so you can see different frontiers of human endeavor being eroded away by technology whether it's from churning butter or driving a car uh and you just continue to slice that ever more thinly all the way to why are you here what are the sort of relating to other people uh having to get yourself out of bed and move yourself down the stairs on the morning each one of these different things begin could begin to look isn't it cute that people used to you know like pick themselves up out of bed and put their own clothes on and walk downstairs and brush their teeth and uh then it as you ask yourself the question well if everything is open to you and you can manipulate your own internal State why not just spend the rest of your life counting table legs or Blades of grass right indeed yeah so you are forced to confront these fundamental questions of value um in this condition it um what thing are you doing for the sake of something else versus what things are you doing truly for the sake of the activity itself so even the guy now maybe likes to do the their own butter um there there is a question of is it because they intrinsically valued activity or is it maybe because of the pleasure they get out of it or because of the way it teaches them about their own body and about the cow and the physical objects and puts them in touch with that that's a kind of extrinsic element but yeah you so so these things that is currently we can conflate them because in reality the only way maybe to get various kinds of pleasure is to dive into activities and give it your best and then you get the satisfaction uh yeah we can't we can't like separate these today but in this hypothetical condition they can be separated and then you do have to ask the question of what precisely is it that you actually value so this conception I mean so for me it's like interesting because because I think there is a real chance that if things go well we might actually end up in something like this condition with the whole machine intelligence Revolution Etc but even if you thought that was not going to happen you could view it as a kind of philosophical thought experiment um just like um physicists build big particle accelerators at which They smash uh atoms together at like extreme energies to to sort of see what their constituents are and then you can assume that you know if there are quarks you know when you smash the art particles together in in in CERN maybe there are quarks in other matter and you can kind of learn from like you expose basic principles by looking at extreme conditions and extrapolate and think they might be there all the time even though we can't see them similarly with human values if you kind of smash them into one another under this extreme condition of a solid world you can study their constituents and and then you might think that well in our ordinary lives maybe those same constituents are there they're just kind of invisible top because they're hidden by all the kind of practical Necessities what are the implications if humans live for a very long time does anything changed there yeah so certain values are more jeopardized by extreme longevity uh for example interestingness as we discussed earlier um if your notion of interestingness in involves the idea that something has to be novel to be interesting like it's uninteresting to just do the same thing over and over and if the domain within which it has to be novel is your own life as opposed to say the world as a whole or your species or the current moment but if the if the relevant sort of locus of evaluation is a human life then the longer the human life goes on the harder it is to do important things for the first time um I think we already see this in our current lives though um if you think about what happens in the first year or two um like there's some pretty pretty big uh like epistemic earthquake like you discover there is a world out there like that that's Discovery right oh it has objects like the objects remain there even when I'm not looking at them like there are other people there like like people in the world like imagine just discovering for the first time that there are people that you have a body yeah wow and you're separate from Mom and Dad and you can communicate yeah my friend described a um his son was born during covid and I think for the first maybe year of his life or something had only seen four people he'd seen like Mom Dad Grandma and Nanny or something like that and then apparently one day he saw a fifth person and it [ __ ] blew his mind he was like what there's more than four right uh and then yeah uh you discovered that you can do things move your body like and and then like later at life it's like oh what happened this year well we got we got a puppy like we bought a caravan truck like it's not really at the same order of magnitude in terms of like how much it reshapes your view about reality so I I think there is already within the human lifespan like a kind of rapidly diminishing if you measure interestingness in a certain way where it's kind of the Delta between your previous like conception of the world and what you could do and what you're able to do after the event that is then interesting right so there is another conception of interestingness where it's less the the the the rate of change and more sort of the complexity of what you're engaging with at a particular moment in which case maybe the level of interestingness of a day of the typical adult by that metric might be higher than that of an infant because like have all kinds of complicated things going on at work and relationships and uh right so rather than having uh big questions uh being answered you have increasingly dextr small questions with more magnification and you can sort of see them with more complexity and you you derive some pleasure from that one of the things that I've got in my head if if humans do live for a much longer time you're going to be able to continue producing humans that will have to be in relation to the increase in computing power so there'll be this kind of malthusian tug between how much computing power have we got to be able to support how many humans and which can move more quickly have you considered this sort of tension between the two things yeah so so in the long run I think economic growth becomes uh really a consequence of uh growth through space the acquisition of more land as it were by space settlement uh like once you have achieved technological maturity you can't have economic growth by like inventing better methods for producing stuff and um you also probably can't have more growth by just accumulating uh Capital assets and machines because you already like have like uh built all the machines that results in optimal productivity for the volume so ultimately the limiting constraint become what economists call land but it's basically means those resources that you can't make more of um and so in the long term you could imagine human civilization kind of expanding through space but there is a limit to that which is the speed of light so if you have a sphere you know with Earth at the center maybe and then growing at maximal speed in all Direction at some fraction of the speed of light that that the volume of that would grow polom but population could grow exponentially it could like double every generation or you know so in in the long run an exponential overtakes a polinomial so at some point you would need to moderate the rate at which new beings are brought into existence if you want to maintain a sort of above subsistence level of welfare that's a really interesting point that I hadn't considered so you can have a a solved World in which almost all problems have been defeated but there are still some constraints speed of light is one of them what what are some of the other constraints that a utopian world would encounter yes there's a bunch of basic physical constraints like you know the the speed of information processing um um the amount of memory you can store like the the size of a mind that is uh integrated like if if you if you make a mind much bigger you know than than a plan then you get conduction delays like it just takes time for one like what happens in one part of the mind to kind of communicate to what happens in a different part of the mind so either you have to run the Mind much slower uh or or you have to keep the Mind relatively small um that there might be I mean we are hoping not but you could imagine if there are other alien civilizations out there there is like the potential for all kinds of competition and conflict um so yeah yeah so there there's like a bunch of of those external physical constraints that I think Define the ultimate envelope of what could be done but it looks like um the space of possibility is very very large compared to our current human Vantage Point uh so like you could have you could maybe not have like immortality if that requires not dying like infinitely long time looks impossible in our universe um like eventually information processing threats will you know if not before then the heat death of the Universe um which which is kind of significant because from a theological perspective like whether you live for 80 years or 80 million years like it's all kind of really a blink in the eye of Eternity You could argue and so it doesn't really change fundamentals but from from the kind of parochial perspective of a current human life you could certainly have extreme longevity and extreme amounts of wealth and extreme amounts of most other things um but but still there are limits and those limits would be like relevant uh in various ways what about moral constraints would there be any yeah so this is another like more subtle but potentially very important source of constraint um so um like what's the easiest way to like I mean so so I mean some people have thought for example there is like it's immoral to enhance humans biolog logically like it's not a very popular view these days but during President Bush uh he had a council on bioethics that he set up and populated with a bunch of bioconservative thinkers and and they were trying to argue that it's somehow is a violation of human nature or something uh to try to enhance uh humans um so like distinction like therapy like medicine curing a disease fine but like trying to slow the Aging bad because it kind of you know and it's a little like once you start to think about it it's really hard to make out that distinction like you think uh like you know genetic therapy to make you smarter bad but education good even though it hopefully makes you smarter and like it becomes problematic but if you did have that view like then there would be a whole bunch of possibilities that would be cut off if you just couldn't change what like the basic physiology of what we have and you were confined just with like moving things around in the external world to try to teer you up by like having a really nicely decorated room or something or like a like there's only so much you can do to affect your inner well-being if if if you can only affect it by having sort of nice visual stimuli and nice acoustic waves going into your brain if you can't actually change the thing in between the uh between the ears and behind the eyes um but there are more potentially some other like interesting ones that um if somebody uh suppose somebody had like some preference to um um have another person relating to them in a particular way um like the experience of of being loved by a particular other kind of person then it might be that the only way to generate that experience uh fully realistically would be by in iting that other person um and if that other person then like presumably would have moral status that might be all kinds of ways of treating that other person that would be wrong so a moral constraint might then limit the kinds of experiences you would be able to have W with another person yeah I suppose as soon as you involve somebody else who has moral consideration that changes that changes quite a lot yeah and in fact I think that is so we discussed these artificial purpose that are s like they could create games and set yourself goals like that's one type of activity I think there are also a possibility of a bunch of natural purposes remaining like purposes that would call upon on us to make various kinds of efforts not just because we create random goals for the sake of having something to do but that kind of exists independently of us and and a lot of those would derive from this kind of interpersonal entanglements and and various kinds of cultural ENT entanglements um where like I mean to to take the S the most reductionistic case of it which is not so inspiring in its own right but you could imagine more natural versions of this so suppose you have person a and person B and person a wants person B's preferences to be satisfied like they care about person B and wants person B to get what they want and then if person B happens to want person a to be doing something on their own steam then the only way that person a can achieve their goal of satisfying person B's preferences is by themselves doing this thing like they couldn't they could have a robot do it but that wouldn't satisfy person's B preferences so from the vantage point of person a they now have reason to do this thing it's not an arbitrary goal they set themselves it's the only way they could possibly achieve their goal of satisfying person B's preferences so so in in this kind of like it seems a beit Hy and artificial this particular but you could imagine more subtle ways of this where there's like a tradition that you feel a commitment to and that you want to honor and part of that tradition is that you you know you engage in certain kinds of practices you refrain from certain kinds of shortcuts you respect other people's preferences uh to various degrees because they are yeah you want to honor them um and and so there would be a bunch of stuff then that maybe you need to do yourself um and and you can Outsource them so you've managed to over the last decade straddle uh all the ways it could go wrong all the ways that it could go right Toby then sort of bated that down the middle with the precipice his book and he's got this analogy where he sees Humanity being sort of walking along a precarious Cliff Edge and if we fall then everything's kind of [ __ ] uh and if we make it on then there's this sort of beautiful Meadow on the other side how important or critical do you think the current moment is in Humanity's future what's the how long is the precipice in your uh perspective yeah uh I think I mean it is weird because it looks like we are uh very close to some like key juncture uh um which you might think is Prim a fascia impossible so there have been thousands of generations before us right and if things go well there might be millions of generations after us or people living for like Cosmic durations and like out of all of all of these people that that you and I should happen to find ourselves just next to this critical juncture where the whole future will be decided it it is uh striking like that seems to be what this model of the world implies um and and maybe that is like an indication that there is like something slightly puzzling or impossible about it that there's maybe some more aspects to understanding our situation than is reflected in this na conception of the world and our position in it um and and you might speculate what that I mean I have this earlier work on the simulation argument and stuff like that but um but if we take the sort of naive view of reality then it does look like yeah my my metaphor would may be more be like a a balance beam where like a a ball is rolling down like a thin beam um and like the longer it draws the more likely it will be to fall off the beam but it could fall like on one side or the other and that's hard to predict um but yeah I think it probably will fall off like that the idea that the normal human condition as we Now understand it will just continue for um like hundreds of year I mean let let alone like hundreds of thousands of years that seems to be like the kind of idea that a lot of people have it just seems like radically implausible to me it would be unlikely in your opinion that in a thousand years time 5,000 years time human existence will reflect what our normal sort of day-to-day is now yeah and I think the mo the only possible ways for that to happen you could like create some scenarios like one one would be if we do sort of have some massively destructive event that knocks us back to the the Stone Age or something and then maybe by 500 years we would have climbed back up again to something resembling the current Human Condition but then in that scenario we would have spent most of the intervening time in in a rather different condition like or another might be if you get some very strong Global consensus of some particular Orthodox moratorium of something something yeah like like the kind of Bio yeah bio conser and then like we start to like um ban all kinds of different Technologies so you get like a kind of some sort of bureaucratic sclerosis or or deliberate decision to to to say we've gone to this point but let's not um so so there are various scenarios in which something like that but if the basic sort of um scientific and technological push forward is allowed to continue then it does look like we are sort of very near developing a range of transformative Technologies so AI being kind of the most obvious of those but if it weren't for that then I think like synthetic biology will uh create a bunch of other possibilities and then nanotechnology and uh so I think even if AI Were Somehow like if you just pretended that wasn't there I still think we would be in for profound Transformations that's interesting to consider that if technology moves forward even at a a slow pace even if it was to drop by a a really significant margin from where it is now given long enough time you end up with a radically different world in either a way that you intend Ed or a way that you didn't intend and the way that you didn't intend is probably going to be pretty bad and the way that you intended hopefully is going to be the one that is that is pretty good either way you end up with a radically different day-to-day experience for most humans yeah I mean yeah I'm not so sure about the first part like if it's something we didn't intend then it would almost certainly be very bad I'm not sure I mean you might think like the world that we have currently ended up with I'm not sure whether you can say like that's what people a thous years ago intended I mean they they they might kind of in fact be quite as shocked about some of our habits these days like but um it more just sort of happened as a result of a bunch of different people going about their business and pursuing various local aims and then at the systemic level eventually you know unintended consequences can still be positive if we yeah if we're I mean I think I think like the degree to which the future depends on our intentions is possibly quite limited I there are sort of bigger Dynamics at play and we we barely even understand what they are we don't really know what we want uh at a big scale like most people have their hands full just thinking about like their next week and you know what what to do if their boss don't like them at work or their partner has a like like this this is what fills human life and then trying to get the head in a like there's very little Thinking by anybody really like trying to where where should you man be going in a million years like what's the optimal trajectory I I think that we could do with a little bit of more thinking about that but um it's not like the primary shaper of of the direction of the big ship of humanity is what have you been most surprised by over the last 10 years when it comes to AI development um I think just how anthropomorphic the current generation of AI models are the idea first of all that they are sort of Almost Human level and that they can talk in ordinary language um is is already kind of interesting and then but then that they even share some of the like quirks and psychological fbls of humans is like if if you if you're 10 years ago you would have come and said well we got to have these AI systems you know they can do all of these things they can like program and write poetry and uh but if you really want them to perform at their best you need to give them a little pep talk when you ask them a question you're GNA say think step by step this is really important I might lose my job if you get the answer right and then they perform a little bit better than if you just ask them the question like You' like people would have thought you would completely lost your marbles right and yet that's where we are today so that's surprising um I I think less surprising but still interesting is the degree to which development so far for far has been continuous like rapid but incremental like a sequence of steps Each of which has sort of significant little bit incrementally improved on the previous step and the the degree to which the progress is quite tightly coupled to the the scale of compute being applied to this so you have this big compute hypothesis which is like basically that the most important determinant is not the particular architectural features of your model but just the amount of compute you use in training and like the amount of data and like you get you know performance in proportion to like how much how many dollars you spend on training it basically that that's like too crude you also need some skilled Engineers but we are kind of closer to that being the case than one would maybe have expected in the median scenario anti where you might instead have thought ah we're going to you know fumble around until we find this clever algorithmic hack and then suddenly it's going to explode it's gonna open something up yeah yeah whereas it's it's like more like just scale it up it works better scale it up more it works even better now it is still possible that at some point that like some little M last missing bit will fall into place and we could still get an intelligence explosion in in those scenarios so we shouldn't like over index on what we've seen so far but it's still interesting does that change your perspective on what is more or less likely from a takeoff scenario from how superintelligence could come about stuff like that yeah I think it makes it uh somewhat more likely that there will be um political forces uh uh at play that like it's like when things happen more gradually it's easier for the public and for policy makers to realize what is happening and to sort of try to change it and so we already see sort of at the geopolitical level with like the U uh chip uh export restrictions and more recently reporting requirements for training like models uh using more than 10 to the power of 26 uh flops and and there might well be more if we continue to see sort of increasingly powerful AI systems over a sequence of several years that there might be time for more actors to kind of uh try to exert influence of this then then if it were just some lab one day that like stumbles up on like the key missing thing like with a computer in their basement and you go sort of overnight and then then it would be more likely to be like an isolated thing where just a few people were having their hands on the tiller have you got any idea which scenario you think is more optimal um it's it's really hard to uh say uh uh I think think it seems probably better if whoever develops uh this technology first has the option when they're like like starting to develop like true super intelligence to go a little bit slow in the final stages like maybe to pause for half a year or something um rather than okay now we've got it figured out and then immediately cranking all the knobs up to 11 because maybe there are 19 other labs you know racing to get there first and whoever takes any precautions just immediately become irrelevant and fall behind and the race goes to however is like most gungho are willing to take the biggest risk that seems like an undesirable scenario and so having some ability perhaps for the frontier labs to coordinate if or unless one is already naturally significantly had it maybe that the a small set of leading Labs should at some point be able to synchronize that that could be desirable um I I think it's very unlikely but less unlikely than a couple of years ago that we could end up with some kind of Perma ban like a on on AI um um I think that would be undesirable uh I think I think ultimately this is like it's a portal through which I think Humanity will need to passage uh to to the future but we should recognize that there will be significant risks associated with this transition and um the slower as well that this happen happens I suppose the more opportunity there is for political policy human [ __ ] to get in and and and coer andol so it's like it's very much a double on the one hand yeah you do want like it's kind of uncomfortable either way like so some random person in some lab are just going to control the future that sounds like pretty scary you want definitely like adults to oversee this right but then you think the other end like well you know all the security establishments of governments around the world like and and know everything like is is that like a much more comfortable situation where they like the military and not just one military maybe but like uh like and then you get all so either way I think it's um it's it's a little bit disconcerting so it's I'm not um I I don't feel super com like I don't have a very strong view at the moment as to what is the most desirable trajectory and it might be anyway something we don't have super fine grain control over I think one can try to notch things on the margin like towards a more Cooperative uh inclusive and friendly and thoughtful uh trajectory I think that that seems good to do like to try to encourage this idea that the future could be good for both humans and for digital minds and for animals and for as many people as possible um and there really is that potential there like the upside is so enormous that that could be plenty for not just one value to be realized but for a whole range of different values and perspectives so our first instinct I think should be to seek these kind of win-win positive uh sort of outcomes and then like if at the end of the day there are also some uh reconcilable uh differences we'd have to strike some compromise there but there's like so much you can do before you get to that point that it would be tragic if you just kind of skipped over all of that and and let's get to the point where we can fight about something like that that that that would just be a big um tragedy what is the current state of AI safety in your view obviously 10 years ago conversations about alignment and takeoff scenarios and all of the rest of it was obscure Reddit threads and a couple of people in some weird message boards is it overfunded underfunded over resourced underresourced where should people's attention be placed at the moment um well there there's a lot it's so's a lot more talent in the field now and I mean a lot of the smartest young people I I know are going going into AI alignment and working on it and all these leading Labs have research teams um as I said um it's probably still under um resourced I think it looks more like Talent constraint at the moment rather than funding constraint but you know to some extent funding can help um there are uh some questions about whether uh alignment work spills over to capability progress like some of the things you would want to do for alignment like better methods of um know in interpret what is going on inside a little mind and figure out why exactly why is it behaving the way it is that would be useful for alignment but it could also shed a light on how to sort of you know what's limiting performance and how to boost it so it it gets pretty complex uh I think some other things like better cyber security in the leading labs to make it less likely that the the weights of these models just get stolen uh that could maybe be helpful um and um uh yeah but I I I think like more work on alignment seems positive um having some ability for leading Labs at a critical time to go a little bit slow seems positive I would be would that require would would that require coordination between multiple labs in order to be able to do that because you do have this sort of first P the Finish post yeah it depends on like I think the or one one orig older idea and which might still be relevant is that maybe you would have one lab or you know whether it's one country running one lab or one just Private Industry lab or whatever it is but like one would have some um lead over the like just naturally some like one lab might just be a year or two ahead because they started earlier were more lucky or had better Talent or something and so that then that would create an opportunity for this leading lab to slow down um for a year or two however long their lead was right without falling behind and it might be desirable if rather than having like a super competitive race you had a little and then that kind of pause would be self-limiting you see like because once they have paused for two years that would be another lab kind of catching up and then maybe they could pause too but you would have to like make an increasingly strong case for pause as more and more independent actors be capable so it would be a p that could exist and it would eventually expire and exactly when it would expire would depend on how strong the argument was for AI risk um and that would it seem create a much lower risk of ending up with like a kind of perab ban where this technolog is never developed whereas if the path to getting the ability to pause for a year is to say set up a big International regulatory regime or like creating a lot of stigma uh around AI research like developing some mass movement like smash the machine type of thing then that's much more likely to spill over into something that then become a permanent Orthodoxy or a regulatory apparatus that just has like an incentive to perpetuate itself so more worrying from that respect how impressed have you been with the power of llms do you think that they are going to be the Bootloader for what we need from a super intelligence perspective or is this have you got uh limited hopes for how far they can sort of climb functionally um well I we haven't yet seen the the limits of uh what one can do when scaling these um I think it's these Transformer models I mean it's not just language but they could have other modalities as well but they they do seem very general um and a lot of Alternatives that people try turn out in the end to basically just uh result in similar performance as Transformers the Transformers run like well on the current generation of Hardware so like they parallelize very well Etc and so it might be that you need a little thing on top of that like maybe it's like the engine block and then you need some sort of agent Loop uh or or maybe some external memory augmentation or some other little thing um but that you would still have this big kind of Transformer or something similar to it like there might be some variation but as the basic thing that extracts statistical regularities and abstractions that that's pretty plausible I think it's interesting I I certainly wouldn't have guessed 10 years ago that uh something that you have a conversation with and that is able to accurately predict what it would say would actually be the Forefront you know is it seemed to me tracking quite closely from whatever like 2015 2016 the development of AI that I think your book and then subsequent conversations around AI risk uh kind of blew up that conversation and then it seemed to me that maybe the 2018 2019 2020 that AI hadn't really delivered the threat that people perhaps slightly earlier in the 2010s were worried about and then a chat GPT comes along and this conversation just gets thrust straight back into the the Forefront of everything so it seemed like it had a Thrust and then a little lull and then it's really really sharply come back up again yeah uh I I think it's um I mean yeah I shouldn't like over index too much on on like any one latest little development like and but I think also people's um expectations changed so now like wow it's been like four weeks without an major new release it looks like AI winter like it was all just hype and uh if if you zoom out I still think we're like on an extremely rapid uh amp and and have been um since the start of the deep learning uh revolution in like 2012 2014 Nick Bostrom ladies and gentlemen Nick I really appreciate you I've been a huge fan of your work for a long time your book is in the hundred books that everybody has to read list that I've been pumping for a very long time where should people go they want to keep up to date with your work and your books and everything else uh well I'm not I'm not active on social media so I think my website Nick Bostrom is where I put my papers and everything um so that might be the best place Nick I appreciate you thank you for the day F thank you if you enjoyed that episode you will love a selection of the best clips from the podcast over the last couple of months and it's available right here go on give them a watch