Video summary
The video introduces the concept of "computer use," a capability where AI agents can autonomously interact with software to perform tasks such as moving chess pieces, applying photo filters, or transferring data between applications without direct human hand movement. The presenter demonstrates this by asking a local AI agent to manipulate a game and take a specific photo, highlighting how these agents can view screens, click buttons, and type commands on behalf of the user. This technology effectively acts as digital hands and eyes for the computer, allowing users to delegate routine or complex software interactions to an intelligent system that understands the visual context of their desktop environment.
A significant portion of the discussion focuses on privacy concerns regarding personal data shared with third-party AI providers, leading the presenter to advocate for running large language models locally on devices like Macs, Windows PCs, or Linux machines. To make this feasible on standard hardware, the video recommends using compressed models from Red Hat AI on Hugging Face, which reduce memory requirements by up to 60% while maintaining over 99% of the original performance on coding and computer use benchmarks. Specifically, the Qwen 3.6 model is highlighted for its efficiency as a sparse mixture of experts that is natively multimodal, making it ideal for analyzing screen captures and executing tasks based on visual input.
To implement these models locally, the presenter suggests various open-source tools such as Ollama, LM Studio, or Llama C++ for different operating systems, while personally preferring the MLX project for its high throughput on Macs. These setups provide an OpenAI-compatible endpoint that can be easily deployed in enterprise environments like Kubernetes clusters using OpenShift AI, complete with custom API keys and external access routes. The video also introduces Hermez, a popular local AI agent equipped with configurable guardrails that ensure user safety by preventing forced application closures and requiring approval for sensitive actions like typing, thereby keeping the human user fully in the loop throughout the automation process.
In conclusion, the demonstration shows how these locally run agents can interpret complex visual data, such as explaining specific graphs in a research paper, by "seeing" the browser exactly as a human would. The presenter emphasizes that with open-source technology and optimized models, anyone can build their own private AI assistant capable of managing calendars, diagnosing clusters, or handling general computer tasks securely on their own devices. The video ends by inviting viewers to explore these capabilities further and suggests that this approach to local AI deployment offers a powerful balance between advanced automation and strict data privacy control.
Read the full video transcript
Okay, so I'm opening up my local AI
agent and I'm going to ask it to move
one of the pawns for me in this game of
chess. It's going to ask, "Hey, can I
use your computer?" and boom, it's moved
one of the pawns for me just like that.
I also got another example. I want to
take a photo with the alien filter on
Photo Booth. So, what it's doing is it's
saying which filters are available. It's
going to click for the specific filter I
want. Boom, I gave it permission to do
all of this and we've got the photo
snapped just like that. So, this is an
example of computer use. It's a really
cool use case with a genitive AI that
can help you to use software, like
moving data from a CRM to a spreadsheet
that I used to do as an intern, but
computer use can help you to view,
click, and type all without using your
hands. The thing is, I'm sure that you
have personal data on your own device
that you're not comfortable with sharing
to a third party or an AI provider. I
get it and that's why I want to show you
how to run your own LLM and connect it
with a local agent on Mac, Windows, or
Linux so that you can start doing
computer use, too. First off, when
you're looking for a model to run, I
highly recommend that you come to Red
Hat AI on Hugging Face, which is kind of
the GitHub for AI, and we have a lot of
models here that are compressed so that
you can cut your hardware requirements
in half or more. So, take this Qwen 3.6
model, right? The full precision
weights, if I go to the upstream
version, are about 72 GB, but if we go
back and I show you Red Hat's compressed
version and floating point four format,
it's only 25 GB of memory that we need
in order to just load the model. And
thanks to modern compression algorithms,
the quantized model performs within 99%
of the original on most of the
benchmarks for coding, computer use, and
more. So, we'll use this Qwen 3.6 model
at 35 billion parameters. It's a sparse
mixture of experts model, so only about
3 billion parameters of those 35 are
active per token and it's natively
multimodal, which is exactly why it's a
good fit for computer use since the
agent passes a capture of the screen or
the contents of the screen to the model.
Now, locally I can run it using many
tools out there like Ollama or LM
Studio, but I like to use the
open-source MLX project since I'm on Mac
and there's incredible throughput. I use
OMLX for this demo.
You can also point at Llama C++ on
Windows, Mac, or Linux. And either way,
no matter what you use, you get that
OpenAI compatible endpoint that we'll
use for the demo later on. On OpenShift
AI, I can easily deploy it from the
model catalog into my Kubernetes
environment. Plus, I can create a route
so we can access the model from outside
the cluster with a custom API key. On my
local machine, I've installed a Hermez.
It's a popular AI agent that can help
write code, manage your calendar,
diagnose your Kubernetes cluster, and
more things like that. But, to get it
hands and eyes on the desktop, Hermez
does computer use through a CUA, a
computer use agent. And it ships with
configurable and multi-layer guardrails
like Windows that can't be force closed,
typing that requires your approval, and
much more. So, you stay in the loop the
whole time.
Now, I want to try it out on this
research paper that I have on the right
here where I want to understand what's
happening in this figure two graph in
the paper. The agent is able to see the
browser just as I do as a human and be
able to understand what's happening in
that graph to explain it to me better.
So, that's how computer use works and
you can do it too using open-source
technology and models. Thanks so much
for watching and let us know what you
want to see in the next demo. See you.