Video summary
The core argument presented in the discussion is that artificial intelligence should be treated with the same pragmatic approach as traditional software, where proximity to the user directly correlates with better performance. Just as massive monolithic codebases are often less efficient than smaller, broken-down components, relying on enormous language models with billions of parameters is not always necessary or cost-effective. In many scenarios, smaller, fine-tuned models that are specific to a particular industry or use case offer superior efficiency and lower costs compared to frontier large models trained on vast general datasets. This shift in perspective encourages developers to prioritize the right tool for the job rather than defaulting to the largest available model simply because it is powerful.
A significant challenge in deploying AI lies in managing costs when attempting to overcome latency by replicating models across multiple geographic locations. While copying a model to various regions can reduce response times for users far from a central data center, this strategy incurs the expense of paying twice for the same inference if the model itself does not change frequently. The transcript highlights that simply adding compute power or duplicating data is not always the optimal solution; instead, organizations must carefully weigh the architectural trade-offs between maintaining low latency through replication versus accepting slightly higher latency to save on infrastructure costs. This decision-making process requires a deep understanding of whether a workload demands real-time interaction or if it can function adequately within a centralized architecture.
The necessity for distributed deployment becomes critical when dealing with interactive, real-time AI applications such as voice agents, fraud interception systems, or video-to-text interfaces that require immediate responsiveness. For these specific use cases, concentrating workloads in a single central region causes performance and cost issues as users move further away, necessitating complex orchestration to route requests to the nearest available resources. Conversely, centralized cloud models remain highly effective for batch processing, sporadic functions, or tasks that do not demand instant user feedback. Therefore, the architectural design must evolve beyond simple token and GPU pricing considerations to include sophisticated strategies for decomposing requests and managing data flow based on the specific latency requirements of the application.
Ultimately, the conversation concludes that gaining cost efficiency in AI requires a nuanced balance between model size, deployment location, and the nature of the workload. Companies must move away from the assumption that scaling is achieved merely by adding more compute or copying models everywhere, and instead focus on selecting appropriately sized models for their specific domains. By understanding the distinct cost factors such as return on investment, total cost of ownership, and customer acquisition costs in the context of AI, businesses can avoid paying unnecessary premiums for latency mitigation. The path forward involves designing architectures that intelligently place workloads where they are needed most, ensuring that real-time demands are met without incurring excessive expenses from redundant model deployments.
Read the full video transcript
Now, I always argue that we should not
look at AI any differently than we look
at software.
The closer it is to a user, the better
performance you get.
Of course, just because piece of code or
software is massive or monolith doesn't
mean it is more powerful and more
efficient. In most cases, a smaller,
that's why you break it down, a smaller
piece of code is better. Same way, you
don't need large language model with
massive billions and billions of
parameters. Sometimes smaller models
work better and they are most cost
efficient as well.
Another factor is culture. As we have
seen a lot of cultural shift, dev, off,
devsecops, sorry, all the label, all the
personas are there.
In terms of AI, how much role do you see
culture, process,
practices as we have adopted in the
software world also play in AI world
when it comes to gaining cost?
>> I mean, to me, we have to look at the
different cost factors that we use to
model our business. Many times people
will look at what is my return on
investment, what is my total cost of
ownership, how quickly am I scaling,
what is my cost of customer acquisition.
Those are three numbers, sort of when
you think about ROI, TCO, and CAC that
many people in SaaS environments are
very used to looking at all the time.
But if we think about where AI is a
little bit different in this equation,
it could be in the past using
and in the recent past, but using a
non-AI centric workload that I would
build in the cloud, that I could scale
more readily by deploying additional
compute, deploying my microservices to
additional environments that would help
me with architectural availability and
resilience, but it could also help me
with latency and scale up if I needed to
put my microservices in more places to
accommodate more users or more
concurrent users. The challenge when it
comes to AI is that if you think about
what you would need to put in multiple
locations, a lot of companies don't have
the experience, a lot of developers
haven't yet figured out. As an example,
can I afford to or does it make sense to
deploy my model in multiple locations?
Because there's a fair amount of data
and you mentioned some of these very,
very large models that people are used
to licensing or used to referencing
versus a small fine-tuned model that I
might use for another purpose. Now, the
first thing that I would say is make
sure that you understand what model
you're using for which purpose. If I am
an industry or a domain-specific
application developer and I need to make
sure that my app is going to give me
quick responses, it's not going to
hallucinate, and I've got all of the
specific information that I want coded
into the model that I need, I might be
looking at a fine-tuned or a domain or
an industry-specific model that
typically will be smaller than these
frontier large language models because
it doesn't need to be trained on all of
the same things for me to address the
use case for my domain or for this use
case. That then also has to do with the
size of the model. So, if I have a
fine-tuned model, it might encourage me
or I might be tempted to deploy that in
multiple places.
But, here's where we can run into a
challenge. You can fix a distance
problem with more compute if it's just
pure and sheer compute power. If I need
to keep my model up to date,
continuously learning, if I need to
think about the size of that model and
how I'm moving data into and out of it,
how that is going to relate to the where
I place the workload and where my GPU or
CPU infrastructure is going to be for
it, sometimes uh dealing with proximity
by having copies of my model data in
multiple places is just going to incur
me a specific uh a certain amount of
additional cost because I'm going to pay
twice for the same answer, arguably. If
I could come to one model from multiple
locations where the user is making a
request and I know that ultimately I'm
going to be serving the same answer, but
I'm copying that model to overcome a
latency barrier. Now I've incurred cost
just to overcome latency by copying my
model. And again, like that could be
okay if you think about my model isn't
really changing that much and I just
need that proximity to maintain a
latency budget. That's a consideration
that you would make at design time or at
architecture time, and then you can
think about the cost that you would
incur. But
if I think about how workloads can
become interactive and chatting, a lot
of the interfaces for the use cases that
modern AI inference are addressing are
voice-based, are are um sort of
video-to-text-based,
or speech-to-text-based, or real-time
decisions like fraud interception, for
example, or a voice agent. Those types
of things need low latency, they need
real-time responsiveness. And so if you
think about concentrating that sort of a
workload into centralized regions, it'll
work great when people are close to that
region, but your performance and your
cost will suffer the further away you
get because you will start to implement
additional capabilities to overcome that
latency, some of which could be, as we
were saying before, copying that model.
So I think the centralized cloud thesis
makes sense if you have centralized
functions that are sparsely or sparingly
used, that are not directly in the
request-response
uh timing for an end user request, or if
that request doesn't require real-time.
But if you need something like anomaly
detection or fraud detection, or to
provide somebody with a real-time query
based on something where they've already
been conditioned to anticipate a
real-time response, then you need to be
thinking about how am I going to have
the orchestration capability to
decompose a request and go to where the
right information or the right code is
being managed for my use case, and that
is an extra layer of architectural
consideration that goes beyond just how
much do my tokens and how much do my
GPUs cost.