Video summary
The discussion clarifies that not every artificial intelligence workload benefits from running inference at the edge; specifically, tasks involving model training, pre-training, or post-training do not require real-time responsiveness or federation. These processes are primarily concerned with updating weights, managing embeddings for future use, and optimizing storage efficiency rather than delivering immediate answers to users. Consequently, heavy computational tasks like fine-tuning models can remain centralized without negatively impacting user experience, as the latency associated with these background operations is irrelevant to the end-user's immediate needs.
In contrast, specific high-value use cases demand that inference occurs closer to the user to ensure speed, security, and reliability within strict timeframes, typically under thirty seconds. A prime example provided is a mortgage application powered by a large language model, where a user expects instant feedback on loan eligibility after providing personal data and undergoing real-time checks like credit or background verification. Similarly, in the real estate sector, companies must handle zero-click searches to maintain visibility against competitors and support complex workflows such as scheduling open houses or processing multi-step loan approvals. These scenarios involve significant financial risk and high stakes, requiring distributed AI architectures that can deliver secure, instantaneous decisions before a user loses interest or fears data compromise.
Beyond financial services, the necessity for edge inference extends to physical AI devices and environments where real-time interaction is critical. This includes anomaly detection in autonomous vehicles and robots, augmented reality applications on assembly lines, and medical or military facilities utilizing VR goggles. Even simpler digital interactions like real-time multiplayer online gaming rely on low-latency inference to allow avatars and game worlds to react instantly to player actions. In these contexts, the hardware and data must be positioned near the point of interaction to maintain the seamless performance required for physical engagement or immersive experiences, distinguishing them from batch processing tasks that can tolerate higher latency.
Ultimately, the industry is increasingly recognizing that the decision to distribute AI depends on whether an application requires instantaneous responses and low latency for high-risk or interactive scenarios. As companies invest in new methods for shuttling data across systems, they are deeply considering where hardware should reside, where data flows, and how the final application experience will be consumed. This strategic shift is driven by the need to support emerging sectors like online sports broadcasting, news delivery, and e-gambling, which all reward immediate feedback loops. The consensus is that while centralized training remains efficient for model development, edge deployment becomes essential whenever the user expects a real-time answer to a high-value query or interacts with a dynamic physical environment.
Read the full video transcript
Are there specific workloads or use
cases that make the strongest case for
running inference closer to users,
devices, or data sources, or do you feel
that every use case, every workload is
ideal if it is running closer?
>> Oh, no, definitely not. I mean, if I
were going to be training a model,
pre-training or post-training of my
model to do anything from, you know,
adding weights to the model or
embeddings that I might need later, if
I'm if I'm pairing down a model that I
might use in a future workload, and I
want to make it a smaller, you know,
overall model for storage reasons and
what I can hold resident in memory, we
don't need those things to be federated.
Those don't have to be real-time and
responsive in this way. If I've already
got a model, and the use case is, "Hey,
I have a an LLM-backed um mortgage
application uh or mor- mortgage um
approval application, then I can think
about when the user logs in, they're
going to provide me a certain amount of
data. I might want to access my data
about that user, and maybe I'll perform
some rag activity to do a credit check
or do a background check on that person,
but all of it needs to come together in
under, you know, 5 seconds typically,
certainly under 30 seconds, and provide
me with an answer that says, "You are or
not approved for this loan. You are
approved for this much, and here are the
next steps to actually capitalize on
this loan." I think about that in a
workload, like an end-to-end workload
sort of perspective, imagine that I am a
real estate company, and I might have
anything from my online listings. I need
to make sure that all of my listings
render completely correctly,
consistently, but certainly that quickly
piece needs to be top of mind for me. I
need to ensure that in LLMs, any
zero-click searches are actually having
my page be cited more frequently than my
competitors' page, so that people find
me in the first place. Just finding that
real estate listing already has a
significant amount of right now
architectural rework and application
rework that companies are going through
because of this phenomenon of zero click
searches and LLM scrapers. Now, that's
just again finding my listing. Next
thing is maybe I want to understand can
I set up a open house time? Can I
schedule a walk through with the
realtor? Maybe I want to find out what
it might take for me to actually get
approved for a loan on this property.
Those are different work streams that
I'm building into my application and
each of those as well needs to be real
time. You would argue that scheduling
something on the realtor's calendar is a
pretty lightweight use case and you
would be right. There's not much of a
appeal for AI in that sort of a use
case. But confirming somebody's
eligibility and what level or what size
of loan they could get is very much
something that AI can help with and is
very common in the financial services
industry specifically around insurance
and and warranties and mortgages that we
need to run quickly, securely, and get
an answer back to that user before they
lose interest and walk away or before
they worry that maybe their information
was compromised. So, now I've got a
listing that I've received, I've applied
for a loan, maybe I've set up time for
an open house. The next thing that I
might consider is the actual set of
transactions where I want to procure the
home or I want to finalize that
mortgage, I want to purchase the house,
I want to go through other signatories.
Those types of things typically as well
will require some form of multi-step
approval process that is currently being
automated by the people that are
managing it. And so those types of
things where you're sitting there
interacting in real time, where you're
expecting a response for something that
is very high value and typically has a
pretty good amount of risk associated
with it, that's a great example and a
great use case for distributed AI
because you are running inference and
you want those answers to be fast,
reliable, and secure.
If I think about some other ones that
are related, you know, anytime that I
might do anomaly detection, that could
be in an autonomous vehicle, it could be
in a robot or other physical AI device,
it could be for augmented reality if I'm
doing, you know,
work on a on a conveyor belt or an
assembly line. I might be in the the
medical field, I may be in the public
sector or even in a a military sort of a
facility, but there's real-time
inference happening through augmented
reality and virtual reality goggles
today that lend themselves to this. And
then we can think about, you know, even
simpler things like real-time
multiplayer online gaming. You're
creating worlds, you're you're having
avatars and other characters in the game
that are reacting and responding to
people in real time. These types of
things you keep hearing real time or
where you have a physical
engagement with the application, those
are typically the things that would
warrant, you know, distribution to
maintain that low latency and that
real-time performance. But I think we're
just starting to understand this as an
industry, and even things like how we'll
create online sports broadcasting, news
broadcasting, you know, gambling {slash}
you know, e-gaming, things that really
do reward latency and instantaneous
responses are where we are seeing
companies invest in this area and start
to experiment with new ways of shuttling
data across systems and starting to
think really deeply about where does the
hardware go, where does the data go,
including the models, and then where is
that application experience going to be
interacted with, and that's what's
driving the considerations.