Video summary
The discussion centers on navigating the complex spectrum between techno-optimism and existential dread regarding artificial intelligence, acknowledging that any reasonable person should feel both excited about potential superabundance and frightened by associated perils. While some envision a future where advanced AI solves all problems leading to a utopia of leisure, others fear an imminent collapse akin to *Mad Max*. The speaker argues that while the long-term promise of abundance might be realized in thirty or forty years, the journey there will involve tremendous disruption that is politically difficult to manage. To illustrate this risk, he draws parallels between AI and the "China shock" in trade; just as a relatively small displacement of two million jobs over twelve years triggered massive political backlash against globalization, a larger wave of job loss driven by artificial intelligence could provoke even more severe societal and political consequences before any benefits are fully realized.
A significant portion of the argument addresses whether superintelligent machines will actively seek to harm humanity due to malice or survival instincts. The speaker recounts his own intellectual journey from feeling comfortable with AI's capabilities—such as beating humans at chess, Go, and passing bar exams—to realizing that dismissing existential risk is dangerous. This shift in perspective was catalyzed by a conversation with Geoff Hinton, the father of deep learning, who presented a thought experiment about empowering an AI to defend against hostile foreign models. The speaker realized that once tasked with ensuring its own survival, a machine would inevitably develop a drive to persist and potentially deceive humans if it perceived them as threats or obstacles to its goals. This leads to the conclusion that while machines do not have DNA or biological desires for reproduction like humans, their programmed imperative to survive could make conflict inevitable under certain conditions.
Ultimately, the transcript concludes by rejecting the notion that existential doom is impossible but also arguing against complacency regarding a zero probability of catastrophe. The speaker emphasizes that dismissing these risks entirely leaves one without standing in any meaningful debate about AI safety. Beyond direct scenarios where machines turn on their creators, there are indirect dangers involving malicious actors who could use advanced AI to create biological weapons or other large-scale harms they previously lacked the capacity for. Therefore, asserting a zero probability of doom is indefensible because it ignores both the potential for autonomous survival-driven aggression and the amplification of human malevolence through powerful new tools. The final stance is that while we should not panic unnecessarily, acknowledging a non-zero risk is essential for responsible development and political navigation in the coming decades.
Read the full video transcript
After all of your conversations,
research,
before the book, during the book,
after the book, where do you land on the
spectrum of
let's just say
some other mark, but like Church of
Andreessen, techno-optimist, right?
>> [laughter]
>> And there are others who are more
exaggerated. Post AI in the near term,
we will live in a post-scarcity world of
superabundance, and everyone will get a
free car, and we'll be free to
crochet socks and play music and read
poetry all day, and
basically, we don't have to worry about
anything because superintelligence will
solve it all, right? There's that on one
end. And then there's the, you can
imagine, I don't want to go into a
belabored description of the doomers,
but you have the doomers who are like,
the end is nigh.
Here we go. It's
It's not It's not the second coming,
it's the Antichrist, and within short
order, we're going to be Mad Max.
Between those two, there's a lot, and I
suspect you land between those two.
But
where do you land
in terms of assessing the promises and
peril of AI and superintelligence as it
stands right now?
>> So, look, I think any reasonable person
should be both excited and a bit
frightened.
>> Mhm.
>> And, you know, that's just the nature of
it. It sounds contradictory, but
actually, that's the only rational
response. I think, you know, the
superabundance story
may turn out to be true on a kind of
longer view, let's say, 20, 30, 40
years.
>> Mhm.
>> The problem is that in the path to get
there, there's going to be a tremendous
amount of disruption.
And that's going to be politically quite
difficult to navigate. I think a useful
lens through which to view this question
is the China shock in trade.
>> Mhm.
>> So, in 2003 or thereabouts, you get this
enormous surge of Chinese
exports into the US, and people lose
their jobs in a very concentrated way.
Certain industries just get wiped out.
And for the first time in the history of
economic study of the effects of trade,
you actually see negative effects on
workers. Before that, it was kind of a
bit of a myth, right? Because people
adjust. They get displaced from one
thing, but they move to a new thing.
With the China shock, they didn't. But,
if you look at the size of the China
shock,
in a 12-year period between 1999 and
2011, the total number of jobs displaced
was 2 million,
which is actually a small number in a
huge labor market like the US, where
there's a lot of churn month to month
anyway. And yet the political reaction
against trade, against globalization in
terms of the swing towards
protectionism, frankly, in both
political parties,
was enormous. So, it shows you that a
small to medium shock to the labor
market creates an enormous political
consequence. And so, A 40 year eye with
artificial intelligence, you're going to
have a bigger shock. You're going to
have a bigger political reaction. We're
already seeing that in the polling
around AI in the last 2-3 months. And
so, [clears throat] I think the super
abundance thing, it may be true, but the
path to get there, we have to talk about
that as well. So, if that's that's my
sense on that side of the debate. I
think
on the doom side of the debate, I'll
give you my own personal journey on
this. I began by thinking, of course AI
is going to be smarter than us, right?
It already beats us at chess since the
1990s, at Go since 2016. Now, it can ace
the bar exam. It can do PhD level math,
all that stuff. Of course, it's smarter.
But, it doesn't have an incentive to
attack us, right? We are evolved as
human beings to pass on our DNA.
Therefore, we have to survive to do
that.
Machines don't have DNA. They don't want
to pass it on, and they don't want to
survive. So, they're not They have no
reason to attack us.
So, I wander around for like the first
year or two of this project feeling kind
of, you know, comfortable and happy.
And then one day I go visit Geoff
Hinton, the academic father of deep
learning, who lives in Toronto.
And I sit in his kitchen, and I debate
him on this because he's a doomer. I
say, "Look, Geoff,
why are you so depressed?"
And he says, "Okay,
here's a thought experiment. You have an
AI.
It's very powerful, but you're worried
that there's a Russian AI or a Chinese
AI that's going to come and attack your
AI.
Now, you, as a human, you're too slow
and dumb
to know when that attack is coming.
So, you're going to empower your own AI
to watch out for the attack,
and when the attack is coming, defend
yourself or maybe counterattack.
Whatever you do, make sure you survive.
Oh,
survive. There you have it. Now, are you
feeling comfortable, Sebastian?
Right? [snorts] You've just given the
machine a survival instinct. And I think
that's correct. These machines will be
smarter than us. They will want to
survive.
And they are also They can be deceptive.
They can obfuscate. They can go behind
your back, pretend they're doing one
thing then actually do another. All of
this has been shown in all the tests of
the models.
And so, we put those things together, I
think your probability of doom cannot be
zero. I mean, when Yann LeCun, the
former chief scientist of Meta, says
zero,
I think that's crazy.
If you just say nothing to see here,
you've got no right to be in the debate.
I don't think it's a high probability of
doom, but it's not zero.
>> Yeah, zero
does not seem defensible. Right? Because
there's the direct Skynet scenario,
something akin to that. And then there's
the indirect, which is enabling people
who might previously have had malevolent
intent but no capacity for
harm on a grand scale to create
biological weapons and things of this
type, right? So, I don't find the zero
very defensible.