TFiR

Move AI Inference to the Edge Before Latency Kills Your App | Ari Weil, Akamai

Duration: 2:41 467 transcript words EN summary

Video summary

Traditional approaches to AI infrastructure often rely on centralized cloud environments with massive GPU clusters, a model that struggles when AI is embedded into real-time applications. In scenarios such as retail personalization, fraud detection, or voice agents managing logistics for first responders, the system must respond instantly because users and critical processes are waiting for an immediate answer. If the processing unit is located far away, such as across a continent, the latency becomes unacceptable for these time-sensitive tasks, making proximity to the user essential rather than just a convenience. The core challenge lies in the "last mile" of reaching the device that initiated the request, which centralized models handle poorly because they require data to fan out to various resources and then fan back in quickly. While batch processing works well for infrequent tasks in a central location, real-time inference demands low single-to double-digit millisecond response times that only an edge footprint can provide. To meet these strict service level agreements, Akamai leveraged the NVIDIA hardware and enterprise software stacks to distribute AI capabilities across their high-performance network, ensuring that points of presence are situated near both the requesters and the necessary security considerations. By federating AI workloads across this distributed network, organizations can offload the heavy compute tasks required for immediate responses while leaving slower, centralized systems to handle persistence, logging, and other non-time-critical operations. This architectural shift ensures that robotics, physical AI, and human operators receive the rapid feedback they need without being hindered by distant data centers. Ultimately, moving inference to the edge before latency becomes a bottleneck allows applications to maintain the speed and reliability required for modern, real-world interactions where every millisecond counts.

Read the full video transcript

What thing more is that a lot of organizations, they are kind of used to thinking about AI infrastructure in terms of centralized big cloud and huge massive GPU clusters. Where does that model start to break down when AI gets embedded into real-time applications? >> So, if I think about things like real-time personalization, if I'm in a retail or an e-commerce sort of a workflow, my anticipation is that as I click a there's going to be a response from the application, whether that's a web application or a native mobile application, or even if I'm doing it through some sort of an agentic platform. So, real-time is real-time. If I think about things like fraud detection I've mentioned a few times now, those things need to happen incredibly quickly because of the pace and the frequency of financial transactions themselves. If I think about a voice agent or something that is doing real-time logistics, for example, to send first responders to where they need to go and manage traffic events while they're going to their first responder responder locations, these are all things where something is waiting on that answer. And that answer can't be a continent or a country away, it needs to be nearby. And so, if I think about, you know, the the common use case of like you fan out to try to go out and find the resources that you need, get all of the tools, get the data, get the context that's required, and then you fan in. The the heavy part of that equation is that last mile of reaching the device that made the request in the first place. And those are sort of the the things that a centralized model handles the worst because fanning out, fanning in, and doing so quickly is something that works well in a centralized location for batch processing or infrequent processing, but the reason that we built Akamai Inference Cloud using the NVIDIA hardware stack, using the NVIDIA enterprise software stack, was that we needed to run inference across an edge footprint that supported low single-to-double-digit millisecond response times. And the way that Akamai has found that we are able to repeatedly hit those types of SLAs is by federating things out across our high-performance network and making sure that there are points of presence to respond to those requests nearer to where the requests are being made and also nearer to where we have to apply some of the security considerations, leaving that heavier weight, slower, centralized compute to be things like persistence and logging and the sorts of things that are not going to get in the way of that real-time answer that either your robotics, your physical AI, or your humans are looking for responses to.