TFiR

Stop Wasting GPU Capacity: Rethink AI Inference Hardware Now | Rob Hirschfeld, RackN

Duration: 3:37 630 transcript words EN summary

Video summary

The discussion on AI inference highlights a significant shift from previous infrastructure models, moving away from a sole focus on ultra-fast interconnects and massive, tightly coupled clusters toward a more nuanced approach centered on managing context windows. While high-speed networking remains relevant for large enterprises training specialized small language models, the primary bottleneck in many inference systems is no longer just the GPU but rather how effectively RAM is utilized to load and feed context data into the model. As shared inference systems grow, the challenge of loading context between different users becomes a critical performance constraint, necessitating a balance between model size, context window capacity, and processing efficiency that often involves optimizing CPU-to-GPU ratios or utilizing specialized processors like AMD's MI300 series rather than relying exclusively on standard GPU configurations. Consequently, the traditional enterprise IT mindset of purchasing hundreds of identical machines to form a homogeneous VMware cluster is becoming less effective for AI workloads unless an organization is running the exact same model at an incredibly large scale. Instead, organizations are likely to adopt a heterogeneous mix of systems tailored to specific models and use cases, which allows them to optimize costs by matching hardware capabilities to actual workload demands. This strategy prevents the waste of capital capacity that occurs when uniform infrastructure cannot efficiently handle diverse inference tasks, acknowledging that different scenarios will require different architectural solutions ranging from standard GPUs to specialized accelerators depending on the specific processing needs. Ultimately, the video concludes that given the inherent difficulty and cost of acquiring hardware, wasting computational capacity is no longer an acceptable option for businesses looking to maximize their AI investments. The future of AI inference infrastructure lies in carefully optimizing this mix of hardware resources rather than blindly scaling up identical systems, ensuring that every component from storage arrays to CPU backends contributes effectively to the overall system performance. As the industry continues to work out the most effective configurations, the emphasis will remain on creating flexible, efficient environments that can adapt to varying model requirements without incurring unnecessary costs or performance degradation.

Read the full video transcript

Now let's talk about inferencing. Why does inference infrastructure change the conversation around multi- vendor sourcing and even previous generation hardware? >> So and this is where things get a little tricky. We we did spend a lot of time about you know two years ago very focused on what model builders needed um which is you know very fast interconnect. it it focused on you know having a a certain type of storage array having these large clusters that were very tightly inter interconnected those are still important and you know I do think that if you're a large enterprise at some point you will look at doing small language models where you where you train a smaller model based on a larger model um to become a subject matter expert on a on one topic because it makes the model smaller and and this I think is what people need to understand about these models and from an inference perspective is the way the inferencing system works is your interaction with the model. Everything you've done so far is a context window. The bottleneck in a lot of cases is how you manage and hold that context window from a storage perspective. That takes a lot of RAM. It has to be loaded and fed into the model in a certain way. And as you build up a big system, especially a shared inferencing system, you actually have to load the context in between different users as they feed into those systems. And so that is actually one of the biggest performance bottlenecks on how an inference system is built. Now the nice thing is is that you don't need to chain together tons and tons of systems with you know fiber optic networking between them to support training the model but being able to build and maintain these systems to inject the context windows into a model store the model on the GPUs or in the in memory all of those things are are all still very demanding and so there's an interesting balance between the the size of the models you use how you load the context windows, how big those context windows can be, how effective the processing can be. Sometimes you can get bottlenecked not on the GPU but on the CPU uh behind this. And so what what we're seeing is there's a lot of ways to do inferencing. We're still working out what the most effective model and system is. And in all likelihood, depending on the model and the workload that you're doing, you're going to have different systems for different models and different use cases. And that's really important when we when we look at the cost of these systems and how much RAM they have and the CPU to GPU balance or if they have special AS6 or special processors to run certain models. What you're going to end up looking to do is optimize that mix. So the idea that you're buying, you know, 60 mach 64 machines, wiring them together as a VMware cluster, and then the differentiation is in the VMs as we used to see in in traditional enterprise IT. When we start looking at at what a cluster would be for uh AI inference, unless you're super large scale and you're running the same model on the same workload at incredibly large scale, it's possible that you're going to be having a much more heterogeneous mix of systems in in your thing or you're going to end up wasting a lot of cap capacity. And frankly, hardware is hard enough to get that you you can't afford to waste capacity and or and in the I don't see that changing in the foreseeable future.