Video summary
In the realm of traditional cloud computing, performance discussions typically revolve around CPUs, storage, networking, and memory, but the conversation shifts significantly when addressing AI, where GPU capacity and token generation speed become the primary focus. While metrics like time to first token and throughput are indeed critical for model efficiency, a deeper analysis reveals that up to 90% of an AI agent's total task execution time often occurs outside the GPU. The majority of this latency is actually generated during the tool-calling phase, where agents interact with third-party systems, invoke external APIs, and retrieve real-time data or context necessary to formulate answers. Consequently, optimizing solely for GPU performance is insufficient because even the most powerful models running on the latest hardware will fail to meet user expectations if the surrounding infrastructure lacks proper connectivity to these essential tools and data sources.
To address this challenge, Akamai emphasizes the importance of an "AI grid" that extends from centralized data centers all the way to the network edge, allowing for distributed token generation in real-world scenarios. This approach requires sophisticated orchestration to manage workloads across various physical locations effectively. Large models that demand clusters of powerful GPUs are best suited for centralized locations to ensure high utilization rates and manageable maintenance, whereas the CPU-based components responsible for running agent sandboxes and executing code should be placed closer to the end user. By separating these distinct computational needs, organizations can optimize end-to-end performance while simultaneously controlling costs, ensuring that the specific workload is located in the most appropriate environment for its requirements.
The core of this strategy lies in the ability to coordinate between CPU and GPU resources across a distributed infrastructure to minimize overall latency. An AI grid orchestrator plays a pivotal role in this ecosystem by scheduling execution effectively across different components of the application, ensuring that the CPU and GPU operate in harmony rather than as isolated bottlenecks. This separation allows developers to place the heavy lifting of model inference in centralized clusters while keeping the interactive, tool-calling aspects of the agent near the user's location. Ultimately, achieving low latency for AI agents depends not just on raw processing power, but on strategically placing different parts of the application architecture to drive down end-to-end response times and deliver a seamless experience that meets modern user expectations.
Read the full video transcript
In traditional cloud, we talk about
CPUs, storage, networking, memory, but
when it comes to AI, we mostly talk
about GPU capacity and token generation
speed.
What metrics are missing from that
conversation? The ones that actually
matter for real-world AI agent
performance.
>> Yeah, no, the the the the this is
kind of a
key optimization that we've been looking
at and again as as we've been working
with customers is that they're they're
building the these agents. Um the GPU is
really important like the end we've been
deploying a a lot of the the Blackwell
GPUs
and the speed of those GPUs so uh time
to first first token, the throughput, um
the
number of tokens they can generate per
second, those are incredibly important
metrics that that we're seeing that
customers are are optimizing for.
But for these applications in the
agentic application,
often what we're seeing is up to 90% of
the overall task execution
isn't on the GPU so isn't kind of
generating tokens.
The majority of that time is actually in
the tool calling portion of the agent.
So calling out to these third-party
systems, calling external APIs, pulling
data into pulling in context, pulling in
real-time
uh information that's needed to generate
the answers.
This is what's actually generating the
majority of the end-to-end latency for
for the task to be successful.
And so optimizing for the end-to-end
system is what's important. So GPUs are
a really important part, but often what
we're seeing is there's a a really big
portion of the end-to-end time that's
not on the GPU at all. So you can have
the most powerful
fastest model running on the on the
latest GPU,
but if your compute is in the wrong
location, um if you don't have good
connectivity out um to the tools, if you
if you're not running um
with a good connectivity to uh storage
and other data systems that you need
your system is still going to be slow
and isn't going to meet the expectations
of of the users.
>> Akamai has been talking about a compute
continuum for a while now. What does
that actually look like in practice
expanding from centralized data centers
all the way to the network edge when we
talk about AI?
>> This is a big optimization problem. So
something that we've been working on is
what we call AI grid orchestrator. So
Nvidia is talking a lot about their kind
of AI grid reference architecture. So as
we move from kind of centralized token
factories
to kind of these real world systems that
can
deploy applications for kind of real
time token generation.
Having the ability to have that in a
grid to distributed infrastructure is
important.
But that needs orchestration. So you
you've got all of these different um
locations where you can run workloads.
And what we find is there are certainly
very large models that need to be
deployed on powerful GPUs often clusters
of GPUs.
Makes sense to have that in more
centralized locations. So you don't want
to have those deployed in hundreds of
locations. It's very hard to
maintain utilization of highly
distributed
clusters of GPUs like that.
But then what we see is for the um
for the CPU where the agent is running
where we're deploying sandboxes where
we're spinning up
containers to kind of run
kind of code execution.
That's something that is
isolated in sandbox to an individual
user and individual session and it makes
sense
as we look to optimize end-to-end um
performance manage cost for that to be
close to the end user.
That's where what we're seeing is
the separation between CPU
[clears throat] and GPU is often um
um in these types of uh distributed
systems, we can place those uh different
components of the application in the
right um physical location
to drive uh end-to-end um latency down.
And then also make sure we're driving
effective utilization of that
infrastructure. Uh and then key key
requirement for the orchestrator is
to coordinate between the CPU and the
GPU and schedule the execution
effectively across those different
components of the uh of the uh
infrastructure.