Video summary
The physical location of AI inference plays a critical role in determining performance, latency, and overall cost, yet these factors are often overlooked during the initial application modeling phase. Developers frequently focus on workflow logic and expected outputs while pricing data movement at zero, ignoring that every time a request crosses a zone, region, or cloud boundary, costs effectively double. The first expense incurred is related to latency, which impacts the time required to generate the first token and process the entire response, while the second is the direct transfer fees charged for moving data between microservices or across different cloud environments.
These hidden costs can accumulate rapidly depending on architectural choices, with cross-availability zone traffic running approximately one penny per gigabyte in each direction. When models send embeddings back to storage or interact with other services deployed in different parts of an architecture, these small fees add up significantly. Furthermore, if data must be sent across the public internet, egress fees can rise to between nine and twelve cents per gigabyte. This creates a strategic tension for developers who must decide whether to keep workloads within a single cloud provider or cross boundaries, requiring a clear understanding of how their applications need to scale before incurring these substantial expenses.
The speaker argues that treating these economic constraints as immutable laws of physics is a mistake; instead, they are architectural decisions that can be optimized. Many teams lack a proper frame of reference for building distributed AI applications and fail to account for data movement costs until it is too late. To address this, organizations should look back at the lessons learned from building distributed cloud apps over the last decade and recognize that cost must be integrated into user and scalability testing phases well before launching an application.
Ultimately, the transition from prototype to production requires a deliberate re-evaluation of where data resides and how it moves, rather than assuming that distance is irrelevant. By bringing cost considerations into the early stages of development and treating them as flexible architectural levers rather than fixed physical constants, teams can build more efficient and scalable AI solutions. This approach ensures that the financial reality of data transfer is not an afterthought but a foundational element of the system's design, preventing unexpected budget overruns as applications grow in complexity and scale.
Read the full video transcript
Can you talk about what role does
physical location of AI inference matter
or play a role when it comes to
performance, latency, and most
importantly, cost? Because cost is going
to become a very big factor because as
there are reports coming out, it's
actually cheaper to hire an employee
than to use AI.
>> That is true. I mean, the first thing
that I would start with, and you're
absolutely right about the way that some
people are modeling applications.
Distance does come at a cost, and a lot
of times you're going to price that cost
at zero because when you're modeling
your application, you're thinking about
things like what is the workflow that
I'm enabling, or what is the output of a
given function or a given prompt
supposed to be, and really focusing on
that. But every time a request crosses a
zone or a region or a cloud boundary,
you're going to pay twice effectively.
The first time you pay, you're going to
pay for latency. How much is my time to
first token? What is the overall time
that I'm going to expend consuming all
of the tokens for my request and
response? The other one is the transfer
fees that basically are not part of your
token price. It's what you're charging
anytime that you need to move data
either between microservices, overall
services, or other cloud boundaries. And
if you think about that latter piece,
cross-availability zone traffic can run
a penny per gigabyte in each direction.
So if I'm going to be sending data from
a model to a set of GPUs that are
deployed in a different part of my
architecture, and then as part of that
workload, I'm going to be sending
either, you know, some of my embeddings
back to store, or if I have to go and
call other services along the way, I'm
going to pay, again, penny per gigabyte
each time that I send that data across.
Egress fees then, if I send them across
the internet, are 9 to 12 cents each. So
there's a uh
sort of tension between how do I build
my application to take full advantage of
the cloud capabilities that I I know how
to build on, or the different providers
are making available.
Does it make sense for me to keep my
workload on a given cloud or where I
have to cross cloud boundaries, am I
ready to incur that 9 to 12 cents per
gigabyte understanding how my app needs
to scale. So, things like latency and
egress, that's not physics. That's not
the speed of light. That is an
architectural choice that you're built
for as though it were
physics. The physics is what you have to
think about and again, where people are
building AI applications and afraid that
they don't have a frame of reference for
what they're building, they can look at
the last 10 to 15 years of building
distributed cloud apps and understand
what is data movement cost me and then
really start to think to your point,
when I move from a prototype to
production, how many zones do I have to
move from and to, where does my
application need to be and one of the
levers that we treat late in that cycle,
late in that consideration phase, is
cost and I would argue that you have to
really bring that into a lot of your
user and scalability testings before you
launch your app.