Can't Track AI Inference Costs? Here's Why You're Bleeding Money | Ari Weil, Akamai
Watch on YouTubeVideo summary
The core challenge addressed in this discussion is the inability of many organizations to track unit-level economics for AI inference, which makes it difficult to determine if their AI initiatives are scaling efficiently or merely becoming more expensive. To solve this, the speaker suggests combining various observability tools with financial operations (FinOps) solutions to gain a comprehensive view of cloud spending, especially in complex multi-cloud environments where billing metrics vary significantly. By integrating active and passive testing tools, businesses can monitor real user sessions, analyze network dynamics, and replay these interactions to understand exactly how much data is transferred and which systems are involved in every user request.
Once the technical data is collected, it becomes a modeling exercise that allows companies to extrapolate growth curves and calculate the specific costs associated with enabling new use cases. This process involves using mathematical models and spreadsheets to simulate scenarios virtually or conceptually, helping leaders answer critical questions about whether a specific application feature is optimized enough to justify its cost. The goal is to move beyond simple capability enhancement and instead focus on the architectural implications of adding large-scale AI workloads, ensuring that the infrastructure supports these new features without incurring unsustainable expenses.
Ultimately, the video argues that introducing frontier large language models or fine-tuned applications does not fundamentally change how an application is built or distributed; rather, it simply adds a new capability with its own distinct size, cost structure, and workload requirements. Every business already possesses the necessary tools to measure these activities and meter them accurately, but the key lies in applying rigorous analysis to determine if the costs incurred for a specific channel or workload are commensurate with the revenue generated. By aligning these financial insights with product and business operations teams, organizations can ensure their margin profiles make sense and that they are not "bleeding money" on AI initiatives that lack proper economic justification.
Read the full video transcript
Now, let's talk about tracking and
measuring. If most organizations can't
even track unit level economics for
inference, how do they even know whether
their AI is scaling efficiently or just
getting more expensive?
>> Yeah, so I think in some cases, I I'll
use an example from Akamai. We have a
product called CloudTest. Um that
product is designed to look at the real
user monitoring of all of the users of
your application, your SaaS application
in the case of of a proxied app on
Akamai, and to show you, you know, how
many people are using it, where are they
coming from, what type of devices and
network connections are they, how many
requests and responses are part of an
average user session, and then you can
start to replay those things through
your cloud and your edge network so that
you understand exactly the dynamics.
Whether you use a Wireshark to look at
the network connections and the network
dynamics, or you use another, um you
know, active or passive testing tool to
look at those real user activities and
replay them and start to decompose them
into repeatable steps that you might use
in a testing harness, but there are
multiple ways to get whether you've
architected the whole user interaction
or you're just recording the reality of
a user interaction coming to you, all of
those individual steps, the systems that
you hit, the amount of data that you
typically transfer, and then it becomes
a modeling exercise. You can do it
virtually or you can do it maybe
conceptually in spreadsheets and
databases and actually use math just to
extrapolate out this is what my growth
curves will look like. You can use real
user examples like that CloudTest
product that I mentioned so that we can
look at the volume of users coming to
you, we can look at their recorded
sessions, and actually show you what
that user interaction looks like. And
there is a myriad of other observability
tools that can help you do something
similar. There are also FinOps tools
that you can bring to bear to help you
make sense of the cloud bills that
you're seeing, including if you have, or
maybe especially if you have, a
multi-cloud environment where the way
that things are being metered and billed
are different. So, I would take my
FinOps tool, I would take my my
observability tool, I would take my
active and passive testing tools.
And if you're an Akamai customer, we
have a number of these for you. You can
also use services if you want to to help
you with sort of this modeling and and
observing your actual user workloads.
But then it basically becomes models and
math. It's how much does it cost for me
to enable this use case? Is that use
case as optimized as I might want it to
be? How much does that cost me? And then
by channel, you can start to understand
with your revenue operations or product
or business operations teams, am I
incurring a cost that's commensurate
with what I'm making and does the margin
profile for this channel or this
workload or this application make sense?
And I think every business has the tools
to do this. It's just when we look at
AI, a lot of times we're thinking that
this special thing that is a frontier
LLM that maybe we've licensed or maybe
it's a a fine-tuned or a post-trained
model that we've created ourselves,
somehow completely changes the game. It
might change the game in the capability
that we're enabling. It doesn't
necessarily fundamentally change the way
that I'm building and distributing an
application. It's just adding a new
capability to it, but that capability
has a size and it has a cost and it has
a set of activities that workloads spawn
from it or to it and you can measure and
meter all of those things to come up
with the right architecture and the
right scalability model for your
business.