Video summary
The video addresses a critical shift in the architectural approach required for artificial intelligence as the industry moves from model training to inference and the rise of agentic AI. While the centralized cloud model has been highly successful for training large foundational models by connecting GPUs into high-speed clusters to optimize memory usage, this same model is proving inadequate for modern agent applications. Early inference workloads like simple chatbots could be efficiently handled in a single location due to economies of scale, but the new wave of agents represents a fundamentally different class of software. These agents are no longer one-dimensional; they are multi-stage systems that execute complex loops, call various tools, and interact with external APIs, making physical geography a significant constraint rather than an afterthought.
The core problem with forcing these distributed agent workflows into legacy centralized infrastructure is the compounding latency introduced by the distance between components. In a real-world scenario, data, users, devices, and the APIs agents need to call are inherently distributed across different locations. An agent typically runs on CPU infrastructure in one place, calls GPU-based models that may be hosted elsewhere, and interacts with third-party systems like OpenAI or Anthropic that are physically distant from the user. When an agent loops multiple times to gather context and execute tasks, every interaction adds network delay. For instance, adding just 100 milliseconds of latency per loop can result in a total wait time of ten seconds if the system loops one hundred times, which is acceptable for a chatbot but disastrous for real-time physical AI systems where such delays render the technology unusable.
To fix these issues at scale, the industry must adopt a distributed inference architecture that respects the physical reality of where data and users reside. Instead of trying to force all compute into a single massive data center, solutions need to be designed so that agents can interact with the real world without incurring prohibitive network delays. This involves recognizing that the power of an agent lies in its ability to access file systems, execute code, and call external tools, all of which happen outside the confines of a centralized cluster. By acknowledging that the real world is not centralized, organizations can build systems that minimize latency by keeping inference closer to the data sources and users, thereby enabling agents to function effectively in time-sensitive applications without suffering from runaway token usage or performance degradation.
Read the full video transcript
We have spent decades moving everything
to the cloud and it's solved a lot of
problems, but when it comes to AI
inferencing specifically, it demands a
fundamentally different architectural
approach. Why is the centralized cloud
model not working for agent AI?
>> It's a good point, and I mean
the the the cloud has been massively
transformational for for for many
enterprises. So, the ability to rent
infrastructure versus own
infrastructure, the ability to consume
infrastructure as a as a service versus
as as as infrastructure. Like these have
been transformational. Um but we think
about AI
we we're going through a big
transformation in terms of the types of
applications that we're seeing running
um
on top of the the infrastructure. So, a
couple of years ago, um
a lot of focus was on um training
models. And so, a a lot of the um
uh discussion and and architecture focus
was on how do we support very large
clusters of coherent GPUs so we can uh
connect these uh GPUs together with very
high-speed interconnects to create this
large uh coherent cluster with
high-speed interconnects between each of
the GPUs so they can share the memory uh
that that's available to them. And so,
memory was really the constraint that
we're optimizing for until we create
these
very closely connected um clusters to
achieve um that high-performance
computing infrastructure that allows us
to create these amazing foundational
models um that we're all familiar with
today.
That's all going to shift as we uh put
more of our focus into inference. So,
running workloads on top of those those
models. Early inference workloads were
were were chatbots. They weren't
particularly uh complex.
Um and so, those could be deployed into
centralized infrastructure. And again,
having um having those run on
centralized infrastructure has got some
some value. So, if you've got massive
compute all in one location, you can
often drive um economies of scale there.
You you you can
drive efficiency in terms of placing
workloads and optimizing usage across
many many users.
We're at the cusp now of the the next
wave of adoption where we're moving into
the rise of agents. So, Open Claw, I'm
trying to remember exactly how long ago
it was, 6 months ago I think it
kind of exploded onto the scene, really
kind of started the um
uh main
main wave of adoption around kind of
agents. And what we're seeing from our
customers now is as they think about
agents, these aren't sort of
one-dimensional chat type applications,
they're multi-stage, multi-turn
applications that are
calling tools, they're
they're looping
multiple times around
these kind of
inference loops.
They're pulling in huge amounts of
context to give the best answers
possible.
And physical geography is becoming a
really important constraint in these
architectures.
>> When enterprise try to force these new
agenting workflow through that same
legacy centralized infrastructure, what
exactly breaks? Because often they don't
realize it, all they see is performance
hit, higher bill, or runaway token
usage. But what is actually breaking
underneath?
>> I think that the kind of simple answer
is like, hey, the the real world
isn't centralized.
Data, users, the devices,
the APIs,
and all the decisions you're making are
distributed. Like the when we're
delivering a a service in production,
it doesn't all sit in one data center
and like not everyone
is physically close to that data center.
And so as you think about it, like what
what an agent really looks like, you've
got a combination of
the the actual agent which runs on top
of CPU infrastructure, it calls out to
models that are running on GPUs.
Those can often be multiple models. So,
you can have different types of models
that's calling. Maybe it's calling Open
AI, maybe it's calling Anthropic, maybe
it's calling
open source models that are hosted on on
different GPU infrastructure elsewhere.
So, that even the models aren't all
co-located. But, increasingly what we're
seeing is
the agents are dominated by the tools
that they're calling.
Um so, the power of agents um is around
the memory system that it's got, the
access to to the file system, the
ability to execute code um in some kind
of a sandbox, generate um output from
the code that it's running,
uh to call out to uh to third parties.
So, to call a a APIs, to call MCPs, to
call um
um out to these external systems so it
can interact with with the real world.
So, this is the real power of an agent.
And as it's interacting with the real
world,
that's not all in one location. And so,
this is the problem that we're seeing is
agents have to interact with the real
world. That real world is distributed,
forcing all of that to run in one
location means that you're compounding
latency.
And these agentic systems, they run in a
loop, they run multiple interaction on
multiple times around the loop. And
you're compounding that latency on every
loop. And so, even adding 100
milliseconds of latency, if you're
looping 100 times, that's 10 seconds of
latency that you've added, which might
be okay for a chatbot. Maybe a human's
willing to wait 10 10 seconds for an
answer to come back.
But, for physical AI, for um for any
type of real-time system, 10 seconds is
way too long. It's it's a lifetime.