Video summary
Danielle Cook argues that the traditional distinction between AI applications and standard enterprise software has effectively collapsed, as AI now permeates every layer of infrastructure, workflow, and line of code. Previously, organizations could operate in a dedicated "AI lane" with separate teams and models, but today even legacy systems like ERPs are integrated into AI ecosystems through mechanisms such as RAG pipelines. Consequently, companies can no longer opt out of the AI ecosystem; instead, they must shift their focus from asking which applications are AI-driven to understanding how all their applications participate in a broader AI environment. This transformation means that decisions regarding performance, cost, and data movement are no longer niche concerns but fundamental architectural choices that define how modern applications are built and run.
The transition from experimenting with AI to deploying it in live production environments introduces significant new demands on infrastructure, particularly regarding latency, real-time data movement, and continuous inference. Unlike training models where organizations can focus solely on the compute power of GPUs, production AI requires a holistic approach involving storage, networking, and CPU resources to manage the constant flow of data. Architects must now design for "data in motion," ensuring that compute is placed close to users while navigating complex regulations and egress fees that can cost billions annually. The complexity of a single user request has exploded into dozens of distributed interactions across vector databases, APIs, agents, and various hardware types, making network latency and API bottlenecks just as critical as the model itself.
To address these challenges, organizations must rethink their infrastructure strategies to prioritize flexibility, portability, and cost-efficiency over simply chasing the largest possible models. Cook emphasizes that not every workload requires massive GPU clusters; often, smaller models running on CPUs can deliver excellent user experiences while saving significant money. The ideal architecture assumes distributed workloads where data, models, and compute can reside in separate locations based on regulatory requirements, latency needs, and economic constraints. By designing for data mobility from day one and utilizing serverless or managed cloud options, developers can avoid the trap of building rigid systems that require expensive refactoring when new models or regulations emerge.
Ultimately, the goal is to align technical infrastructure with genuine business objectives rather than succumbing to AI hype cycles focused merely on GPU counts. Companies should evaluate their specific maturity levels and choose between using managed services, fine-tuning existing models, or building custom infrastructure based on what best serves their revenue targets and operational needs. The industry is moving toward a future where vendors provide simple experiences that abstract away the complex plumbing of deployment, allowing developers to focus on solving real problems. Success in this new era depends on recognizing that AI is an architectural necessity for everyone, requiring careful planning around token economics, data sovereignty, and the ability to adapt quickly without redesigning entire applications.
Read the full video transcript
For most enterprise IT, AI used to live
in its own lane. There was dedicated
model, dedicated app and dedicated team.
That lane is now gone. AI is slipping
into every application, every workflow
and every layer of infrastructure. And
the moment that happens, decisions about
performance, cost and data movement stop
being niche choices and they become
architectural decisions. This shift is
felt more by the companies that never
thought of themselves as AI companies at
all. Daniel Cook, director of product
marketing at Ecomi, has been watching
this transformation from the inside. She
joins me to explore why the boundary
between AI applications and everything
else has collapsed and what it means for
how we build and run applications. Next,
Daniel, it's great to have you on the
show.
>> It's always great to be with you. A few
years ago, you could draw a clean line
between AI applications and everything
else. Today, every line of code goes
through AI in some form. So, is that
distinction is still holds true? Is it
is still meaningful? And what about
those companies that don't think of
themselves as AI companies? What does it
mean technically for those company?
>> So, I don't think you can really opt out
of the AI ecosystem anymore. Like,
you're in it. So you know where we used
to be able to say you know there are two
buckets of applications there's an AI
bucket and a nonAI like that line has
disappeared so all applications contain
AI they're either connecting to AI
through APIs they're feeding data into
AI to you know help train models or
they're supporting something that does
um so like even if you're thinking of a
legacy ERP P system for example like
that's becoming part of an AI workflow
somewhere as soon as somebody puts like
a rag pipeline pointed at it. So nothing
about that application changed but it is
in the AI ecosystem. So I think we need
to start thinking about you know instead
of asking which of my applications is an
AI application we need to start focusing
on how are my applications participating
in the whole AI ecosystem.
um because you you don't have to build
AI to become part of this this system
this ecosystem.
>> How much visibility do organizations
actually have into whether their code is
touching AI at all?
>> It depends on the maturity of the
organization. Of course, to your point,
regulated industries like they're
they're able to kind of lock things
down. But if you are having, you know,
any any code contributed any workflows,
you have to assume people are using AI.
I think we're going to hear shadow AI
more and more and more people are just
using it whether it's approved or not.
Um so it's it's going to be increasingly
important to get that visibility and
understand
um where where your entire team is and
you know I know platform teams are
struggling with like how do we monitor
this developers are generating so much
code and how do you check it like it's a
real struggle across the industry
>> when enterprises move from experimenting
with AI to running it in live
applications what actually changes is
why does production AI put different
demands on infrastructure as compared to
running traditional applications? You
know in production you tend to focus on
on or in experimentation rather you tend
to focus on the model like does it work
is my output good but in production now
you have real users real latency
requirements real data reals that are
associated with this and you know
because inference is continuous it's you
know it's not a model that you train and
walk away from um that's where you start
realizing that GP CPUs are only part of
the architecture. You need compute, you
need storage, you need networking, all
of that to actually determine how your
application performs. And you can't just
throw more GPUs at this at every step.
So, you know, your infrastructure is
changing.
You also might be looking at it going
okay so GPUs I need it for this stuff
but actually I need you know tools tool
calls APIs databases all of that can
actually largely work with my CPUs. So
choosing the right model might be part
of you know an exciting decision for you
but making everything work around it at
production scale is an infrastructure
problem that organizations are having to
solve. How much is this changing how
architects approach infrastructure? Is
it a minor adjustment or do they need to
completely rethink their infrastructure
strategy?
>> Architects should be looking at the in
the entire infrastructure, right? AI
does change everything. It it is, you
know, real time. It's interactive. It is
the the data moving all around. Complete
data in motion. So, how do you build for
that while maintaining it, making sure
it's cost-effective? How do you not, you
know, blow your entire budget, trying to
deliver an amazing experience to your
customers? And so, when you're
architecting for it, you have to be
considering things like model rooting.
You have to be considering how you're
going to, you know, use semantic caching
so that the right prompts, the right
answers, the right experience gets to
your customers. And so yes, it's an
infrastructure challenge. You need to be
putting compute close to your users, but
you also need to be thinking about the
core, you know, data movement around
inference, around AI, about the customer
experience.
>> With token cost, latency and performance
pressures rising, moving AI workloads to
the edge is becoming critical,
especially since not everyone can run
things locally. So what paradigm shifts
are you seeing in how technology
hardware and organizations like Akami
are positioning AI workloads at the edge
to cut down on cost and performance
impact.
>> So we used to move data to the compute
right and increasingly AI is forcing us
to move the compute to the data. So
we're seeing um the model, the data, the
user, those can all be in three separate
places. And so and data is often the
most or least movable part of that
equation. So whether it's because of the
size of it, um sensitivity, regulation,
right? Um or simply could be just the
cost of moving that data. So we're
seeing that workload placement is
becoming um a huge decision and it's a
decision around latency, data location,
regulations and the economics to your
point around you know tokconomics. Um
and so this absolutely belongs in the
architectural conversation
that you know because AI applications
that data again is is constantly moving.
So if your cloud economics assumes that
data is mostly staying put but that your
AI architecture um assumes it's
constantly moving like those assumptions
are going to collide. So you don't want
to be in a situation where you're
discovering your egress bill um you know
after you've designed your architecture
you want to make sure that you're
designing for it upfront especially with
these real time customer interactions
>> when someone sends a simple request to
an AI enabled app what is actually
happening across the infrastructure
where do these systems actually run and
this is a challenge for both teams
running those applications s and for
companies like Akamay who are building
that infrastructure for those and for
companies like Akamay who are building
that infrastructure for those teams. So
can you talk about what's the what's the
reality behind one simple request?
>> What looks like one request to you know
an end user whether that's internal an
external customer um it can be dozens of
distributed requests underneath. So a
user sees a prompt and a response uh
underneath that request they might hit
multiple models a vector database object
storage APIs agents GPU infrastructure
like all of those things aren't
necessarily sitting next to each other
and so that's why this networking
matters so much like a little bit of
latency repeated 50 times it that's not
a little bit of latency anymore
and so it means that the bottleneck
isn't necessarily the model or the GPU.
It could be the network. It could be an
API. It could be storage. It could be
one step in a long agent workload. Um,
and so it's again a very different
architecture from the traditional app to
database and back model. So I think the
big takeaway here is like the old
application architecture
could fit on a napkin, right? You could
draw request response, but today's uh
architecture, you know, it's messy and
it's it gets messy really quickly.
Managing that, having to be able to see
what's going on, rooting it,
understanding it, observing it are all
challenges that, you know, developers,
platform teams, they're all experiencing
>> today. Enterprise data lives across
public cloud, private infrastructure,
SAS platforms and edge locations with
regulatory requirements spanning
different jurisdictions. How should that
reality influence where AI compute runs
and how much weight factors like token
cost should have?
>> I think that really starts where you
know you need to consider where the data
lives and where it's permitted to live.
uh to your point because moving that uh
is it can be impractical, slow, there
could be a multiple restrictions that
get in the way and so geography will
impact performance, compliance,
resilience and cost. So we know that
data transfer charges can change the
economics. Um you know we have you know
egress fees you know
estimated to be around 70 to 80 billion
annually like it's expensive. Um so you
know when you are moving data
considering it across different clouds
like you need to consider like how
you're going to not necessarily be
leaving providers or if you are leaving
providers what is the cost economics of
of doing that and making sure that it's
sensible for your business. And to my
point earlier, like that's why you need
to be architect architecting for those
egress fees upfront and understanding
all of those performance restrictions,
requirements, all of that so that you're
safeguarding against that because AI is
only going to grow. If you were
designing an application platform today
for where AI is headed not where it is,
what would you build differently from
the start to avoid refactoring,
migration or the challenges
organizations are facing today?
>> I would absolutely design for data in
motion from day one. So I wouldn't
assume that the application model, data
compute, any of that is going to live in
one cloud or one region.
I'd assume applications are distributed
uh and that again that that there's
constant data in motion. Um I'd make
workload placement flexible so that
compute I can decide where um the
compute needs to service my end user
based on what their experience should
be. Um and I make portability
fundamental. So models infrastructure
all of this is changing rapidly. we see
new models coming out continuously. Um
so I'd want to make sure that I can move
to what's right um quickly and so my
goal um wouldn't be to introduce
necessarily a new model region
compliance requirement without
redesigning the application. So or to be
able to to introduce all of that without
um redesigning the application. So I
think that's really where a distributed
infrastructure becomes interesting
because if the application is
distributed by design, the
infrastructure underneath it should be
too.
>> Now let's look at teams who are
rearchitecting. Now where should they
start? You mentioned data. What else
should be on their priority list?
>> Well, I think you need to obviously look
at the compute resources you need. Who
has those? uh what's available to you.
You need to be considering your models.
What models do you want to run in what
size? What amount of the data do you
want to own? Do you want to be running
your own infrastructure, owning your own
model, or do you want to be using
another person's tools like the build
versus buy argument? Um and you know the
tokconomics of it all. So, you know, we
know that AI is expensive and um so it's
the amount of of money you want to
spend. Do you want to be spending it for
yourself? Do you want to be spending it
paying it to others? I think the other
thing that you know as we kind of move
into this world, it's whether you want
to even care about the infrastructure.
you know, do you want to consider that
there might be ways that you can
service your application, your your AI
application without ever having to
consider the infrastructure? And I think
we're going to see, you know, we we see
vendors doing that. We're going to see
more vendors coming to market with that.
>> We often get obsessed with the biggest
model possible, but the smaller models
sometimes deliver better results. Can
you clarify that? to your point you yeah
you know I want to use the biggest model
the most powerful model but that's not
always cost-effective and it's also not
always the best experience for your
customer so being able to understand
that you know I have trained a model and
now I'm fine-tuning or now I'm servicing
this customer but I actually only need
to run this on a CPU with a smalls size
model um and they're going to get a
great experience and I'm going to save
money or I'm going to do this more cost
effectively is really powerful and so
you need to understand your workloads.
You need to be able to route to the most
coste effective CPU GPU and you need to
you know not waste money on just going
bigger is always better.
>> Where does edge computing fit into the
AI infrastructure picture you are descra
you are describing. So for Aami, you
know, edge compute is obviously
something that is uh near and dear to
us. Um we want to make sure that our
customers are able to service their
customers with the compute they need.
And so that might look like um you know
our distributed GPU locations. It might
look like using containers at the edge.
Like there's so many different options
that a customer can have based on what
their customer needs to do for their
customer, what the experience they want
and also the the financial impact on
that. So for us, we're about making sure
that
compute happens where customers need it
and so we have mult multiple ways of
doing that.
>> As much as I love AI, I also feel that
we are in an AI hype cycle, especially
around AGI. But honestly, it's more
about GPU count than actual AGI. Every
company labels almost everything they do
as AI. So when an organization is
looking at production AI, what should
they actually focus on? Of course,
infrastructure has been democratized. So
real question is are you solving a real
problem using AI or you are just chasing
the next shiny object which is AI. How
what is your advice to company? how
should they look at their old AI
strategies? I think that's fair um to
and it it the way I like to think about
it is terms of there's an AI framework
that exists um not a maturity model not
you know everyone's at different stages
and so it might be right for a business
to just
use you know use an AI service and
they're good to go and they're happy to
spend the money on that and all that
they're putting on top of that is
security. It's, you know, managing bot
traffic, all of the things that they
need to do, but they're quite happy with
that. It might be that they're want
their own GPUs. They want to do some
fine-tuning of a model. And so now they
need to be considering, well, what GPUs
do I need? Am I going to use Kubernetes?
Like, what is the storage I'm going to
put behind this? And they have to be
putting that all together. You know, for
some organizations, there might be like,
we want a huge, you know, we want to
train our own data. we need thousands of
GPUs and that's where what's right for
them and then you have you know the
other end of the spectrum I don't really
care about infrastructure at all I don't
want to worry about it I don't want my
developers to worry about it I just want
to be able to access inference and then
you have serverless inference coming at
the end so it isn't necessarily I'm
going to go here and then graduate to
this stage and mature to that stage
because I think it's what's right for
the business and people have to take
remind themselves to look at what is
your business objective like does this
map to what you're trying to achieve. Is
it the right financial model, revenue
targets, all of that? Are you spending
too much on tokens or do you need to
bring inhouse? All of those things have
to be considered because it isn't a
one-sizefits-all. as AI is being
democratized through infrastructure like
Akami cloud where you can get Kubernetes
clusters, Linux cluster, GPU clusters in
seconds without having to worry about
managing them yourselves. How is this
also allowing developers to focus on
building business applications instead
of all the plumbing that waste most of
their time? It's absolutely our mission
uh at Akami Cloud to make sure we are a
simple experience for developers. So
when you
sign in to to Akami cloud, you you get
what you need there and you don't need a
degree in um in in understanding how to
deploy everything. Um as we support
Agentic workloads, AI inference, we're
doing the same thing. We're making it
really easy to come into a cloud and see
the options and just get started. So,
we'll have plenty of news coming out on
that soon. So, you'll be able to see it
see it uh for yourself.
>> Daniel, thank you so much for taking
time out and such cleareyed look at
where AI is taking application
architecture, how organizations should
prepare themselves for AI. Thank you for
all those great insights and those who
are watching please head to akami.com
and learn more about the work they are
doing in the airspace and back to
Daniel. Thank you so much uh it was a
great discussion with you and I look
forward to having you back on the show
for another great conversation. Thank
you for your time today.
>> Thank you so much.