Video summary
The video presents a demonstration of the Red Hat AI Factory, showcasing how Intel Xeon processors and NVIDIA GPUs function more effectively as a unified system rather than isolated components. The core problem addressed is the significant underutilization of CPU cores when running large language models on GPUs; for instance, a Llama 3.1 model with 405 billion parameters typically only requires about 12 CPU cores, leaving the rest idle. To solve this inefficiency, the presenter proposes deploying smaller language models on the available Xeon cores to handle auxiliary tasks, thereby maximizing hardware resource usage across both the CPU and GPU simultaneously.
This multi-agent system architecture allows for a strategic division of labor where simpler tasks such as research, review, and data processing are executed by the small model running on the Xeon processor, while the more computationally intensive task of generating the actual essay is handled by the large model on the NVIDIA GPU. This approach not only optimizes hardware utilization but also leverages two specific features of the Xeon platform: Advanced Matrix Extensions (AMX) for accelerating inference in BFloat16 and INT8 formats, and Priority Core Turbo (PCT). PCT ensures that high-frequency cores can communicate efficiently with the GPU, effectively eliminating orchestration bottlenecks between the two different hardware resources.
Further enhancing this setup is the capability for dynamic routing, which intelligently directs simple or complex tasks to the most appropriate model based on specific user needs. Performance benchmarking conducted during the demonstration revealed substantial gains in efficiency, showing that combining Xeon CPUs with NVIDIA GPUs allows for approximately 44% more concurrent users while still meeting Service Level Agreements (SLAs). This significant increase in throughput demonstrates that the synergy between these technologies directly translates to better scalability and performance for AI workloads without compromising on quality or speed.
Finally, the presentation outlines a clear path toward seamless integration within the Red Hat ecosystem by planning to upstream all optimization algorithms into a CPU operator for OpenShift. By downloading and installing this specific operator, users can automatically configure their applications to utilize idle CPU cores for running small models, achieving high performance without manual intervention. This automation ensures that organizations can easily adopt these advanced co-processing strategies, turning previously wasted computational resources into valuable assets for their AI infrastructure.
Read the full video transcript
Xeon and NVIDIA GPU are better together.
Hi, I'm Alex Zin. I'm an AI software
solutions at Intel, and today I'm going
to show you a demo on Red Hat AI factory
with NVIDIA and Xeon. So, one problem
when you're running your models on your
GPU is that most of the CPU cores are
actually idle. You really only need to
use about 12 CPU cores to work with a
Llama 3.1 405 billion model running on
your GPU. So, what do we do with those
idle cores? Well, why don't we deploy a
small language model on Xeon using those
idle CPU cores?
Let me show you how it works. So, here
uh this is a multi-agent system using
the two models deployed on Xeon and
NVIDIA GPU where the simpler tasks are
running on the CPU using the small model
including researching and reviewing uh
an essay, and the more compute-heavy
task of writing the actual essay is done
with the model deployed on the GPU.
Now, this opens up to a lot of
opportunities where it better utilizes
your hardware resources, both your CPU
and the GPU.
There are two features of Xeon that's
incredibly important to make this
possible. One is AMX, Advanced Matrix
Extensions, which accelerates inference
in BFloat16 and INT8 for small language
models on Xeon.
And second is Priority Core Turbo, PCT.
This
allows you to use the higher frequency
cores on Xeon to work with the NVIDIA
GPU, hence reducing the orchestration
bottleneck between your two hardware
resources.
Another opportunity with this is you can
work with dynamic routing where you can
route simple and complex tasks to the
appropriate model based on your user
needs.
We also did some performance
benchmarking where it turns out that
when you put your Xeon CPU and the NV
GPU together to run different models,
you're we're able to get about 44% more
concurrent users while still meeting
SLA. And finally, how does this relate
to Red Hat is we plan to upstream all of
the algorithms to optimize deploying
models on Xeon into a CPU operator so
that when you run your application on
the OpenShift, you can download this
operator, install it, and it will
automatically figure out how to use
those idle CPU cores and run it with
performance without you having to do it
yourself. Thank you so much for
watching.
>> [music]