TFiR

Distribute Your AI Inference or Watch Latency Destroy the Experience | Dr. Robert Blumofe, Akamai

Duration: 3:14 595 transcript words EN summary

Video summary

The video explores the fundamental differences between AI training and inference, emphasizing that while both are critical, companies prioritize inference to actually deliver value to end users. The speaker argues that these two phases require distinct infrastructure architectures based on their primary affinities or proximity needs. For training, the key affinity is the massive dataset, which is typically stored in a centralized cluster; consequently, training makes sense where this data resides, utilizing dense GPU clusters in that specific location. This centralized approach aligns with the nature of computation during the training phase, which relies heavily on access to large, static datasets rather than real-time user interaction. In contrast, inference operates under a completely different set of requirements because AI agents must interact directly with distributed users across vast geographical areas. The primary affinity for inference is proximity to the people who are engaging with the system, whether they are in a single country or spread globally. Placing an agent thousands of miles away from its users in a centralized data center would introduce unacceptable latency that destroys the user experience. As AI applications evolve beyond simple text or voice interactions to include high-definition video and dynamic image updates, the need for low-latency connections becomes even more critical to support these rich modalities effectively. The transcript highlights that modern AI agents offer unique capabilities such as real-time video generation and dynamic image updating, which provide compelling value that cannot be achieved through brute-force builds of centralized infrastructure alone. These advanced features rely on high-bandwidth interactions that are only possible when the processing power is located near the user to minimize delay. The speaker suggests that as AI agents become more ubiquitous, we will see a shift toward distributed architectures that can handle these high-bandwidth forms of interaction, ensuring that the technology delivers on its promise of being helpful and responsive in real-world scenarios. Ultimately, the conclusion drawn is that while training benefits from centralization around data storage, inference demands a decentralized approach to serve a global user base efficiently. The inability to replicate the rich, multimodal experiences of AI agents through traditional centralized models underscores the necessity of distributing inference workloads closer to the edge. This strategic distinction ensures that latency does not hinder the experience, allowing users to engage naturally with AI systems that can process and deliver complex media without significant delays, thereby fulfilling the true potential of artificial intelligence in practical applications.

Read the full video transcript

Let's look at training versus inference. Most companies care far more about inference to actually deliver AI to their users. Do these two phases genuinely demand completely different infrastructure architectures? >> Yeah, that's it's a it's a great point and a great way to um sort of distinguish these use cases and and I think it's helpful to think about, you know, what is the important affinity? What does the infrastructure need in terms of proximity? And arguably in the case of training, the key affinity, the key proximity requirements is the data set, the training data set. And most training data sets are are fairly large and they're generally stored in in some large storage cluster that's going to be fairly centralized. You typically wouldn't have your training data set distributed around a large number of locations. It's going to be fairly centralized. So, it makes sense to do the training where the training data set is. Um it's also the case that if you look at the actual computation, you know, it's very GPU dense. So a dense GPU cluster centralized where the training data set is. That makes a whole lot of sense. When you move into inference, well, what's the affinity? What does it need to be near? Well, it needs to be near the things that it's interacting with. Um and and there's a lot of things that that AI um applications, AI agents have to interact with, but obviously one of the important users of course is the people, us, you know, we are going to use agents to get things done for us. We're going to engage in conversations with these agents to help specify what it is we want done, to look at results, to review results, provide feedback. It's going to be very conversational. So the affinity um of an agent I mean and we can get into this a little bit more um in a little bit because there's a lot of different affinities but one of them clearly is is to the users. Um and users are typically distributed over a fairly large swath of geography whether it's a country or a continent or or the entire world. Um so it makes no sense really for the agent that you and I are interacting with to be thousands of miles away centralized in in a single uh location. Um and relative also to what I was you mentioning earlier about the the nature of the um of the interaction changing with the web where we went to um highdefinition video and things like that. The same thing is is the the case with agents. You know you you don't want to think of an agent interaction as being simply text or even voice. Um a good agent is going to show us video is going to show us images and it's going to be dynamically updating the video and dynamically updating the images. These are capabilities that you don't get outside of of AI. And it's one of the great um benefits of using an AI agent is that you have all these modalities available, video, images, interaction that um isn't available in uh in um with other technologies. So, you know, as we see agents become more ubiquitous, I think we'll see these high bandwidth forms of interaction really take um take hold because they really deliver value and they deliver something compelling and interesting. And there's just no way to do that from uh through a brute force buildout in centralized infrastructure.