Video summary
When organizations transition artificial intelligence models into production environments, egress costs frequently emerge as a significant financial surprise that was not fully anticipated during initial planning. The core argument presented is that companies aiming for rapid and sustainable growth over the next few years must look beyond the specific line items highlighted by hyperscalers and instead model the entire bill of operations. This holistic approach requires understanding how various use cases, such as user acquisition, concurrency with multiple users, and active learning from global data sources, collectively impact expenses. Crucially, teams need to evaluate the cost implications of keeping data within their own environments—whether in private data centers or leased space—versus relying on public cloud infrastructure, as the decision to own all data and application code fundamentally alters the economic landscape of data movement.
A major concern driving this strategic shift is the high cost associated with leaving a cloud provider, which often acts as a barrier to maintaining data sovereignty. Industry news from the last year indicates that egress fees can reach between nine and twelve cents per gigabyte, a pricing structure specifically designed to encourage vendors to retain customer data on their platforms. While some providers offer one-time exemptions for downloading data when exiting entirely, these scenarios highlight the prohibitive nature of moving large datasets across geographic boundaries or between microservices. Consequently, there is a growing movement toward neutral architectures that allow data to be stored in a central location while federating or synchronizing it across various areas, enabling retrieval and serving where it makes the most operational sense without incurring excessive transfer fees.
The proposed solution involves adopting a converged architecture, such as the core-to-edge model utilized by Akamai, which initially appears similar to an expensive distributed setup but offers distinct financial advantages when all factors are considered. By managing data and application code within a unified framework that serves content on demand with low latency, organizations can eliminate the hidden costs of moving data between disparate services and across different regions. This approach transforms what might seem like a complex distribution challenge into a cost-benefit scenario, provided that teams accurately model their specific use cases across the entire platform. The strategy effectively decouples the necessity of centralized inference from the reality of where application activity actually occurs, allowing for a more efficient allocation of resources based on real-world usage patterns rather than theoretical cloud limitations.
Ultimately, the path to optimizing AI inference costs lies in answering three critical questions: where inference should run for a specific application, how frequently pre- and post-training or fine-tuning activities are required, and where the majority of application activity takes place. By integrating these technical considerations with business metrics like user acquisition strategies, growth trajectories, and long-term customer value, companies can determine the optimal balance between centralized and distributed components of their architecture. This comprehensive modeling ensures that organizations do not simply react to unexpected bills but proactively design systems that align their data boundaries with their operational needs, thereby avoiding the trap of being locked into expensive cloud ecosystems while still leveraging the benefits of modern AI infrastructure.
Read the full video transcript
You touched upon egress cost earlier.
When teams move AI into production,
egress cost often come as a surprise to
them. How much of that hidden cost of
centralized inference is actually just
data movement, getting results back to
users and applications? How big of a
factor is that and how can teams plan
for that as well?
>> So, I think, you know, moving into the
next
12 to to 36 months, the companies that
are going to grow the fastest and the
most successfully will have modeled the
whole bill and not just a line item
that, you know, the hyperscalers are
touting or that all of the different AI
platforms are touting. They need to
really understand all of the use cases
that will incur cost for them as they
scale. And that can be anything from
what is my user acquisition cost, what
is my concurrency cost with multiple
users, how many use cases have I enabled
on this platform, where is some of that
data that I own completely, my
intellectual property, my systems of
record, and where am I learning actively
and and developing from the world or the
industry at large and my customer base?
Because the first thing that I can do is
I can understand what would that cost be
if I said I don't want to use multiple
cloud providers or even my principal
cloud provider and I want to manage the
data in my own environment. My own
environment could be my own data center,
it could be leased space, it could mean
a lot of different things, but that's
the first question of what if you just
owned all of your data and your
application code.
Um that will have you start to
understand availability zones and data
boundaries and where distribution can
become more expensive if you use your
own versus using a hyperscaler's
infrastructure. The second thing is once
you've decided to partner with somebody,
what would that cost to leave? Because
now, if you look at all of the last 12
to 18 months of news around the cloud
space, there have been, you know, all
sorts of news stories about how much of
is cost to leave a cloud provider was a
cost for companies to take over their
own data, to export or download it
themselves? And so the reality is 9 to
12 cents a gigabyte is something that is
designed to keep your data where it is
on most hyperscaler platforms because
the cost of moving it is prohibitive.
Some vendors have said if you're leaving
their cloud all together, then they will
give you a one-time exemption where you
can download your data, but you should
be making sure that you indeed intend to
exit that cloud environment if you do
so. Um it's also given rise to something
that people are considering to be more
of a neutral architecture. So a neutral
neutral architecture is one that will
allow you to store data in one place
that might then federate or synchronize
it in a bunch of different areas, allow
you to retrieve it somewhere else, and
then serve it where it makes the most
sense. So a converged architecture like
Akamai where we have core to edge can in
some cases look like you might be
thinking about an expensive distributed
architecture on the face of it, but when
you think about all of the costs that
you don't have to pay from moving data
between microservices or other services,
moving it across geographic boundaries,
and what it would take to manage all of
your data and application code, and then
serve it as needed on demand with that
low latency, could actually turn into a
cost benefit for you if you modeled out
all of your use cases across the
platform. And I think that's the
ultimate question here is where does
inference run for your application, how
often and where do you need to do things
like pre- and post-training and
fine-tuning of your model, and then
where is the rest of the activity in
your application actually occurring?
Because if you know those things, and
you know how you're modeling user
acquisition and user growth and
long-time value, then you can put all of
those factors together and think about
what sort of an architecture am I really
focused on and which parts of my
application can be centralized versus
where I'll benefit from distribution.