TFiR

Stop Overbuilding AI: Design Agents That Actually Work | Dr. Robert Blumofe, Akamai

Duration: 4:42 811 transcript words EN summary

Video summary

The discussion on AI agents often incorrectly centers solely on GPU scarcity, overlooking the necessity for a hybrid infrastructure that combines computing resources with specialized tools. Dr. Robert Blumofe argues that an agent is not merely a powerful Large Language Model (LLM) but rather a complex system composed of many components where the LLM serves as one part responsible for managing natural language interactions and making decisions. The defining characteristic of an agent is its ability to execute actions, which requires translating text into real-world tasks through various tools. These tools handle specific functions such as reading and writing storage, retrieving data from CRMs or vector databases, searching the web, sending emails, or calculating shortest paths on a map, thereby creating a diverse ecosystem beyond just text generation. A critical design principle highlighted is to maximize functionality within non-AI tools whenever possible, reserving AI models for tasks where no other solution exists. While LLMs are indispensable for decision-making and conversational management, they are inefficient and unreliable for deterministic operations like database queries or web searches. By offloading these specific tasks to dedicated CPU-based tools, systems become significantly more efficient and robust. This approach naturally leads to a hybrid infrastructure requirement where the system must support both GPUs for running AI models and CPUs for executing algorithms, SQL queries, and managing data storage, ensuring that the agent can interact with the physical or digital world effectively. Furthermore, the choice of AI model should be tailored specifically to the job at hand rather than relying on massive, general-purpose models for every scenario. Using a multi-trillion parameter model for a specialized task, such as helping customers file insurance claims, is unnecessary and inefficient compared to employing a smaller, specialized LLM designed for that particular context. Just as one would not use a tool capable of writing code or composing poetry to simply calculate a route on a map, it is better to match the AI's capabilities to the specific requirements of the agent. This strategy of using the right tool and the right AI model for each component ensures that agents are practical, cost-effective, and truly functional in real-world applications.

Read the full video transcript

When we talk about AI agents, everyone immediately fixates on GPU scarcity. Can you explain why AI agents actually require a hybrid infrastructure rather than just a massive GPU cluster? >> Yeah, you know, this is a great point and you know, I think that the the more I think people can really wrap their heads around what an agent really is architecturally. Um the the better off better off will be cuz I think it's it's tempting to think that well, you know, an AI agent is simply a super powerful LLM with the latest and greatest LLM reasoning capabilities. That's an agent. And that's not the case. The the key insight probably is to think of an agent as being a a system with many many components. In fact, most agents do indeed have many many components. And an LLM is just one of those components. The LLM typically at an LLM will typically play a fairly central role in the agent because you need something that's going to manage the natural language interaction. You need something that's going to make decisions about how the interaction should proceed. What's the next question to ask? What's the next task to do? And so on. So, an LLM or in many cases multiple LLMs play a fairly central role. But ultimately, what makes these things agents is the ability to do things. And remember, an LLM can do nothing but but produce text. If you want to do anything, you have to translate that text to output into action. And that means tools. That's the key thing. LLM sorry, agents. Agents are systems that involve LLMs using tools. And in most good agents, there's typically quite a few tools. It could be tools to read and write storage. It could be tools to retrieve retrieve information from say a a CRM, right? To retrieve information about the customer that you're talking to. Um, it may be um, a tool that retrieves information off of the web. It may be a tool that retrieves private information from a so-called vector database. Could be a tool to send email, manage calendar, um, or could be a tool to calculate shortest paths on a map. So, most agents are a combination of AI models, multiple AI models, plus a whole variety of tools. And I would argue by the way, and I've been saying this for a while now, that a a good rule of thumb, if you're designing an agent, is to put as much of the functionality as possible into the non-AI tools. You know, in some sense, use AI only when nothing else will work. Um, you know, and and that doesn't mean don't use AI, of course, because the AI as the central component to making making decisions and managing the natural language interaction, well, nothing else will work. AI does that, and it does it so well. Um, but when it comes to other tasks, like things that I mentioned, like email, retrieving things from a database, searching the web, no. Use a use an actual tool, a non-AI tool. It's way more efficient and way more uh, reliable. So, rule of thumb should be put as much of the functionality in your agent as you can into the into the non-AI tools. Okay, the upshot of all that is that the infrastructure demands are coming from not just the LLM itself, but from the combination of multiple LLMs, multiple tools using um, data, retrieving data. So, you have a a hybrid uh, infrastructure requirement. You do need GPUs, but you also need CPU to run that shortest path algorithm, to run the SQL query, and so on. And of course, you need um, you need storage for all that data that you're going to be operating on, whether it's storing things like uh, memories or um, uh, or retrieving things from a vector database. So, you have this hybrid need. And by the way, you touched on this earlier. Uh, you know, another point I would make about sort of a good design role is for the parts that are AI, the parts that are say LLMs, use the right LLM for the job. Not everything requires a multi-trillion parameter ask-me-anything model. In many cases, if you're building a an agent for a specific use, you really can you're really going to be much better off with a with an LLM that's much smaller and specialized for that particular for that particular task. building an agent to help your customers file insurance claims, you probably don't need an agent that can write code, compose sonnets, tell jokes, and give you the cast of every Mash episode that ever was recorded. So, you know, use the right tool for the job and use the right AI for the job.