Beyond Chatbots: Creating Smarter, Personalized Experiences in Drupal with AG-UI

Okay. Welcome. Can everyone hear me okay? Excellent. So, welcome to our session on beyond chatbots with HG UI. First, a special thanks to everyone on the ImageX team that helped make the demo possible. We couldn't bring everyone, so we have uh the honor of showcasing their hard work today for you. I'm John Tran. I'm the CTO of ImageX. And I've been delivering web applications and services for over 25 years in the industry, and the last eight of those have been ImageX. Uh this is Aaron Christian, one of our architects. I'll let him introduce himself. Yeah. Hey, everybody. Uh my name is Aaron Christian. Uh I'm a front-end architect at ImageX. I've been working uh in the Drupal space for about 18 years. I've been with uh ImageX for about eight. Um and I'm super passionate about interaction design, uh animation, things like that. Uh yeah. So, this is the team that's kind of behind the demo today that we're going to show you guys. Uh we'll get into that. Can you click on that? There we go. Um so, first and foremost, uh quickly uh overview the agenda here. So, a little bit about ImageX, and then our uh AI innovation uh roadmap, a little bit about semantic search, uh some details on HG UI itself, and then we have a demo for you, and then uh hopefully some time for Q&A at the end. So, about ImageX. Uh we were founded in 2001, so we're also celebrating our 25th anniversary this year. Yeah. Uh and about 20 of those years have been specialized on Drupal, so a lot of uh history with Drupal. At Axelerant, we're about 100 people worldwide headquartered in Vancouver, Canada with hubs all across North America, Europe, and Latin America. And about 45 or half of the team are developers. Um but so that's my team, but even with our global hubs, we've got 25 hours of a day of coverage 6 days a week of development going on, so it's pretty pretty cool. Um our verticals are non-for-profits, commercial, and education. And we're a full-service agency, so we provide strategy and consulting, design, and UX. Um as well as development and integration, of course, and managed services and support for our long-term clients that have that have been with us over 10 years. So, next let's talk about Drupal and AI and Axelerant specifically our our road map for for AI. So, a little bit of background, about a year ago our CEO, Glen Hilton, who's in the back there, he and I traveled to Europe for a Drupal conference and an impromptu AI summit kind of formed involving some of the key players in the Drupal space. Dries Buytaert himself was there. Um and basically that ended up launching the Drupal AI initiative and with Glen joining the board of the Drupal Association. So, us and several other industry leaders committed at that conference to driving AI adoption within Drupal. And then fast forward a year, just 2 weeks ago version 1.3 of the Drupal AI suite of modules was released. Um and we can kind of broadly categorize the their strategy into three uh three streams here, so content generation, content governance, and content automation. Um but Adobe Project basically tried to extend that strategy uh beyond the the major experience and we're trying to enable end user uh experiences powered by AI. So, our focus is on three areas, uh internal development optimization tools that might give users to deliver these experiences. And then the two streams of experiences that we've been focused on are implementing semantic search, which we feel feel is the next generation and the natural evolution of search technology. And the second is something we call the content concierge. So, that's basically an objective digital advisor that guides users through the content consumption experience. Um so, we'll we'll show you that uh working today and it'll make more sense, but over to Aaron first to talk a little bit about semantic search. Okay. Um yeah, so semantic search um let's take a look at what it actually is. So, semantic search uh basically passes your raw query data when you type in your search um through these three lenses. So, the first one being intent, what does that user actually want? The next one being some context, so maybe it's uh uh a past interaction with the the system or their device, where they're located. And then the last one being meaning, so it really ties in um or connects intent with context to try to figure out what the the user's actually um asking about. Okay. Uh so, what do we know about search? Um this is where we kind of expose the problem. So, we We that uh majority of the users are actually bypassing your really cool design mega menus, probably that nice little hero banner that you guys spent a lot of time designing. Uh and they're going straight to that search icon. Uh the problem is that over 2/3 of those people aren't finding the results that they're actually looking for. They're getting frustrated and they're actually dropping off you guys' ecosystem entirely. So, we know that's obviously bad um and it I mean increases your your guys' bounce rates. Uh so, you guys are going to kind of uh passing those users off to your competitors. Uh so, if you can fix search, you can keep your users engaged. Uh and so, we'll take a little a little look at what that looks like and where uh legacy search fails. So, uh legacy search or um you guys might have heard of it as uh lexical search as well. Keyword searching, it's really fast. Um it doesn't require a lot of um uh CPUs, so it's low latency. Uh it's really good for deterministic outcomes. So, things like part numbers, CPC codes, error codes, program IDs. Uh anything that has like a uh actual value that you're you're looking for. So, it does have two pretty big blind spots though. Uh the first one is that it has zero conceptual understanding of any of the other um data that you pass to it. So, for example, if I'm searching for com- comfortable running shoes for bad knees, it's going to look at um comf- or it's going to look at the running shoes, but it's not going to take into context that you are actually meaning that you are looking for a comfy shoe. Maybe it's got some extra padding cuz you got bad knees. Um it also misses things like synonyms. So, if you had like a really cool designed page that was all about athletic footwear, somebody searching for running shoes, they might completely miss that. Semantic search, on the other hand, this is AI search, AI-powered search. Uh it maps words with similar meanings. Uh so, it would put pick up those uh intents that are passed through like the the medical or the cushioning intent. Uh but, it kind of has a very blind spot. So, it's not great at looking at program IDs or UPC codes. It tries to be creative and they might give you a list of, you know, similar products. Okay, so we're looking at probably a paradigm shift here, um but it's pretty important to understand how both of these uh types of search work. Uh so, the keyword search, uh it works off of an inverted index. You can think of that as like a textbook, one of those old dusty textbooks that you guys probably haven't pulled out in a while. Uh so, it doesn't search when you type in your search term, it doesn't search through every page on your site line by line. It flips to the back of the index, finds where that that word was, and then it relates back to the page. Uh so, this uses an algorithm that's pretty much uh the gold standard. It's called BM25, stands for best match 25. Um it's based off of term fre- frequency and term saturation um or rarity. Uh and so, who remembers the olden days of keyword stuffing? Yeah? You know, you hit that page and scroll to the bottom, you got all this white space, and what is that? You highlight it, and you're like, I got product product uh keyword tags in here. So, um they introduced the uh parameter called the K1 parameter. Uh It basically stops a lot of that. Creates like an async auto curve. Um essentially, so uh the first time you um uh mention, say, nursing on your page, you get a big boost. The third, fourth, fifth time that you get that that curve that just levels out. Um and then there's also another one called the B parameter, which takes into consideration the document length. So, uh sometimes, well, longer documents are obviously going to have that keyword in it a lot more than another document, but it doesn't necessarily mean that it's more important. Uh so, that's what that parameter tries to take into consideration. Uh keyword search, it does a lot of things that you guys are probably familiar with if you've ever um adjusted in your search settings in the back of the Drupal. Things like keyword stemming, does uh tokenization, uh stop word removal, removing all that the common uh uh um joining words like that. Um and then whoop. I'm not sure what that means. Uh semantic search works off of uh an embedding index. So, it tries to use natural language processing. Um and so, one of the ways that we can use an embedding index is through vector embeddings. So, a vector embedding might be text, audio, uh multimedia, any type. Uh and so, I'll try to explain this um uh for you guys or in an imaginary way. So, think about it like a digital landscape or a map. Um and so, something like beer would sit next to survey stuff. Okay? They don't have the same language, but they're mapped by meaning. Okay? And then from there, you might have a a stemming. That might be Budweiser or Modelo or something like that. So, you can kind of see where it's um it works like a daisy chain. Um you guys heard of the uh the game six degrees of Kevin Bacon? Yeah? So, it's basically uh if you haven't, it's uh you can basically link any actor to Kevin Bacon in six steps or less, uh where each step is a shared movie. So, the lower the score, the higher the rank. Uh and so, the goal here uh of semantic search, the broader goal, is uh just basically the conceptual understanding of human language. Uh so, we know that expectations are changing. Back to John. Thanks, Aaron. So, uh what exactly do we mean by smarter personalized experiences? Well, basically, uh while we've been rolling out semantic search over the last year, our version our vision was to extend that uh concept of semantic search to the entire site experience. So, uh along the way, we wanted some funding from the Canadian government to help us build out that vision. So, what we did was we partnered with the University of Waterloo to uh develop uh first iteration of this concept. So, what exactly did we build? So, in this case, we built uh an AI-powered university admissions advisor. Oops, sorry. Um so, it starts off by looking at a typical semantics looking like a typical semantic search experience, but it dynamically responds with multimodal media content um in a conversational flow. So, you kind of have to see it to to really understand how it works, but we'll show you in a in a while here. So, first a little bit about under covers, kind of how this stuff technology works. It's obviously a very Drupal-centric stack. Uh Drupal is the foundation, and we leverage everything it has to offer, including the Drupal AI modules. And then, we also contributed a new AGUI module to the Drupal AI initiative. Um And finally, we integrate with a series of enterprise uh services as needed, vector databases, uh various LLM models, ejective frameworks, third-party data. Uh so, it's a true enterprise scale uh framework. So, what exactly is AGUI? Okay, so it stands for agent user interaction protocol. So, it's a part of a suite of protocols that are the the emerging standard for ejective communications. So, I'm sure you've heard a lot about MCP um in your research and then here at the conference this week, but MCP stands for model context protocol, and in essence, it's a communications protocol between agents and regular software tools, okay? Uh so, that's straightforward. A2A is similar, but it's for agent-to-agent communication. And then, finally, AGUI is the protocol for communication between agents and the user interface. So, with this um suite of protocols, basically that defines the kind of state of the art in uh ejective communication. It's all really just just JSON um specifications on how uh these things can talk to each other. So, with our emphasis on building AI-enabled user experiences, uh AGUI becomes critical, obviously. So, that's why we have um invested a lot into it. So, what are the design principles at a high level for AGUI is basically bi-directional communication between agents and the user interface. It defines a series of standard events, 60 about 16 of them by default and you can extend that with your own events as well. It's transport agnostic so it supports streaming communication using server sent events, webhooks, web sockets, or even polling. So those of you that are deep into the Drupal stack you know PHP has some issues with streaming. It doesn't support SSE out of the box so we've also have a sister module that kind of helps with that in the Drupal stack. It's also agent agnostic AGUI. So what what that means is you can swap out any LLM models, you know, instantly with like one line of code and you can also swap out any agentic framework. So all the major ones, LangGraph, Co AI, Agno which used to be called by data, Microsoft, Google, and now Drupal all support AGUI so they basically speak the same language between the agent and the front end. So you can almost instantly with very little effort swap agentic frameworks which is really powerful. So now I'll give you over back to Aaron for a little bit more detail on how AGUI interacts with the Drupal front end. Thanks, John. Okay, yeah. So we want to stop forcing users to filter and start generating answers. And that's the reason why we got into the AI. Okay, so this is pretty much the AGUI module kind of um at a a layering level. Um, but we can get into kind of the the how, what, and why. So, why did we need a Drupal module? Well, it's um basically the bridge between Drupal and the human interacting with AI. Um, we ported in the the AGUI um uh TypeScript library into to Drupal. Um, and we chose AGUI because it is open source. Um, it's built on TypeScript like I said, has strict uh type checking. Um, and it's also built uh with Python on the back end for the HMTX side. Uh, so to touch on a little a little more about what John was talking about with some of the bottlenecks in Drupal with um PHP. Uh, so right now um when you send a uh a message through uh uh a system like a a normal chat system, goes through this HTTP request and response method. Um where we want to go is through the SSE events. Those those uh server-side events. So, think about it as like uh uh a walkie-talkie or or maybe like a voicemail type. So, you pick up the phone, you call somebody, they don't pick up. Um, you have to leave a message and then you hang up. Okay? The agent then picks up the phone, listens to the voicemail, has returned your message. Then they hang up. That process repeats. We want to go to into SSE. This is that open line uh communication. So, both you and your friend are both on the phone at the same time. You can stop, interrupt them, change the direction of the conversation, steer them in a different different way. So, that's uh SSE. Um, and to bring SSE into Drupal, we built this uh uh sister module uh that AGUI Mercur is it connects with the the Drupal Mercur module. Um and that's the one that handles the uh SSE protocol. So, um why it's important, um think about this uh in the traditional sense where um say 50 users came to your site and your server only had uh 50 PHP workers assigned to it, that 51st person that starts to interact with your chat, they start to fill up this PHP worker queue. Uh and so, if anybody knows how that works, if you get too many people in the a queue, uh things start to exhaust. You might your your server will probably crash and you're looking at a self-inflicted uh DDoS attack. So, that's not fun. Um how do we do it now with Drupal? We do the polling method, which is basically just a a fake a fake SSE um implementation. It's an AJAX polling request. Uh think about it as like the kid in the back of the car, are we there yet? Are we there yet? Um so, it's it basically it gets stuck in this like while true loop essentially. Um and so, uh it's it it works, but it's slow. Uh and it's again, you're you're running up against uh uh memory exhaustion. Uh so, Mercur is actually built on Go, um which is super fast. Um Docker, Kubernetes, they're all built on Go as well. Um and we chose it because of its concurrency. It can handle thousands of those chat threads. So, now we're not 50 PHP workers, you're looking at thousands of workers um who are able to to handle those um connections relatively inexpensively. Uh, okay. So, let's look at some of the features. We built it, um, around SDC. I I'd be remiss if I didn't, uh, mention Brian Sharp here. Uh, if you haven't seen him in like the Drupal Auto or forums or anything like that, he's, um, he's part of the Drupal AI uh, initiative and kind of heading things, um, for us. So, uh, he's, uh, he was responsible for a lot of the original porting, uh, and then the team kind of gathered around that. Uh, so, uh, one of the cool things that is, uh, AGUI supports the streaming chat integration, uh, human in the loop transactions, so that's, um, like the ability for the agent to ask you for more information and then you can provide it that extra context or or intent or whatever. Um, generative UI, so this could be things like presenting the user a commerce product when they start to search and it's in like a fancy design, a card that that fits really nicely with your your design. Uh, it could be like a program card or or a weather card or something like that. Um, it also has built-in JWT support, so I'll chase some web tokens. Um, and not just access like this digital VIP wrist band. It's, uh, basically says like, the agent says, "Who are you?" and you have to present like your passport and that verifies them and then you're able to continue the conversation or I won't provide you data back. Uh, and then it has custom tool integration, um, so anybody can create their own module, hook into the system. We have like a public, um, uh, event system inside of, uh, AGUI that you can hook into. Uh, so, when a a message starts or something like that, you could you could, um, provide an extra detail to the UI. Uh, and in the the end there's, uh, a demo page at admin/agui/demo. And that will convert Drupal's deep chat to be AGUI compatible. Okay. Well, we are an open source community, so I think it's probably a good time to talk about how to contribute back. And as ImageX, we have quite a few people who are involved in the community. Last year, we had a contribution initiative that each of our developers had to get a core credit. So, we attained that. Um we're looking for more AI stuff this year. But, you can see that the we've completed a few things from this is like the porting from AGUI into Drupal. So, these are all like Drupal specific items that have already been built into Drupal. Our road map is items that AGUI has created, we just haven't ported into Drupal yet. So, those are awesome. If anybody wants to have a look at those, there's lots of documentation around AGUI on how these things work. So, one of them would be like step tracking. That's like tells us what step the user's kind of on as they're going through the journey. Right now, we have like this thinking bubble and you kind of see it. It's essentially when you're like interacting with a cloud or something like that and you it starts to to really give you those reasoning or those those tracking um steps. So, the other one is is reasoning. There's activity events which streams live agent progress between the messages. There's the interruption life cycle which evolves human in the loop into like pause, edit, and retry. And then there's messages. So, we've all dragged an image or some sort of document into chat GPT or another system and it's enabled to read that. So, that's kind of the next level of AGUI inside of Drupal being able to drag and drop things. Okay. Thanks, Aaron. So, demo time. Let's finish this up and show you guys how it works. Okay, so this is the Waterloo digital admissions advisor. So, looks very typical for like an agentic user interface. So, we'll start off just by kind of asking it uh what it does, but why don't you log on first, Aaron? So, we've connected the system with open ID Drupal module and presented it basically a form here. Just one of the features of AGUI is it supports sessions. So, we can do uh authenticated uh interactions and then also track your history for context. So, Aaron's gone ahead and logged in. So, why don't we go ahead and ask it uh what it does. All right, yeah. So, for brevity's sake, I'm just going to copy and paste in here, but imagine I'm taking this out with lots of spelling errors. So, it's telling us what it does. It's an agentic uh admissions advisor. It's going to help us with general information and uh the walks us through the whole admissions process and answers any questions we might have. So, let's tell let's ask it uh to tell us a little bit about Waterloo here. Okay. Can you all zoom in a bit. Sure. I'm passive. Okay, so you can see it's doing a tool call right there. Um, that's basically uh AGUI talking to the agent and then the agent uh designating um a task to a tool. Yeah, at this point it's essentially a semantic search implementation, so it's uh it's reading our intent, uh feeding that in context the fact that it knows who we are and what our previous conversations might have been and then translating that to a more meaningful uh response for us. But uh next let's talk about the uh the university >> what it's known for. So we're just having a little bit of conversations with it at first, um and we'll get into some more advanced features in a second, but uh let's ask it about the campus. So again, it's just kind of remembering what we've been talking about and giving us um contextual responses and kind of building our profile around that. So it does markdown formatting and you can kind of see it starts to bold things. Um, it'll return uh this structured JSON back into a format that we can um essentially uh format into like bullet points and things like that. So now we know a little bit about uh Waterloo and the campus and uh me and Aaron are West Coast boys, so we're a little bit uh concerned about the weather. Let's ask it uh about that. Here it is pulled up there. So here's our first little um nugget here, some multi-modal response. Um, it's detected that we've talked about weather, so it makes an API call in the back end to uh tickets.weather.com. I'm not sure which API it used, but um it's getting real data live based on our prompts and creating a multi-modal response, in this case a weather widget. So, and Aaron, we're in Chicago here, so let's figure out what that is in Fahrenheit. So, it can do a little bit of math. Um and make another API call to do some conversions there. And we're actually planning to visit the campus at some point, so let's see if we can get a little bit of a layout of of the campus. You might want to zoom out. Yeah. Okay, so >> put it back to full zoom out. We're just going to zoom out so you can see a little bit more of this interesting stuff here. So, Aaron, what's going on here? Yeah, so we basically made a tool call to Google Maps API with some location data. We pulled in a canvas tool, so you can kind of see at the top there we have this new tab system, one for history, one for canvas. Canvas is like the building area, and then you can kind of toggle those to to close them or or change. So, we'll get into the history in a second, but essentially every time we do this tool call, whether it's like a um a weather card or like this product, whatever it is that you guys want to present in this generative UI, it stacks up in this history call, and then we can do things like when we come back maybe a week later and we want to you know, extend on our conversation, we can do things like tool hydration. Um so, it'll actually pull in all that data that's been saved in private kind of storage on on our side, and then it can actually build in that history so it can refresh like this exact map as well. Awesome. And so we're planning to live on campus so let's figure out where we might be able to do that. Sure. I'll just make one more note. We added some functionality around notebooks so I'll just add us and we'll show you that feature in a moment. And also we just bookmark this location for now. We'll show you why we want to do that Okay, so we just asked where we can live on campus which is probably pretty typical thing that a prospective student would come to look for when they're looking for the site. So the idea is that we don't want people out in the open site for ChatGPT and find all this information on it. Right in your guys' site and it's uh tailored for you guys' customers so it only knows about your institution or your your guys' site. So you have these guardrails in place. So this is another API call to different uh API for uh location and then we just bring that together to one experience. So once you once you find a couple of locations here for the notebook later. Yeah. So maybe I'm maybe I'm really interested in um Village One University of Waterloo. So we'll add that to our notebook. And then we're going to have to eat while we're there so let's figure out where the nearest Walmart is. So we do another API call. This one's using um directions API. And we'll see that no locations got returned, but we can maybe try it get from U W place. So, it'll give us uh another it tried to do a tool call there. Um something obviously went wrong where it didn't find any locations. So, um you can always kind of um get back into that text response um messaging. All right. Well, that's classic in live demo mode. We're working with chat or open AI, so everything's kind of unpredictable. Well, let's uh let's change tracks and see if we can get some some photos of uh campus life there. All right. So, now we've uh plugged back into Drupal and pulled out media from uh Drupal's uh media storage, and uh we've got a full image gallery right here that we can look through. Yes, that one fetched it straight from um Drupal using a view uh to output all these these um cards here. Okay, so the point of the university obviously is uh studying, so let's let's ask it about what programs are available. Okay, so you have another tool call to our program catalog. Uh it's going to pull open all the programs. Uh and so you guys can kind of set these things up like how you're going to connect this data. Um it could be from Drupal, it could be from any API really. So, we dig in to kind of structure data within Drupal, and then we can respond differently to the different content types you have, and map different tools to those content types. So, we just saw how images are mapped to the gallery, and in case, program information is mapped to a collection of filterable cards. You can do the filter. Yeah, yeah, I just popped that open. Yeah, so we have filtering and again, tagging for the notebook. And then there's another general tools we have. Maybe let's ask it about the percentage of kids that are accepted. Sometimes it takes a second. There we go. There we go. So this is kind of a I mean, it's just a little interesting module, but essentially, it's detected that it's like a highlighted statistic. So we'll have a statistic module, and then it kind of the AI actually decides what relevant icon would represent that statistic well. So just kind of add a little bit more intelligent thinking on the agent side. Next, let's look at what it's going to cost for us to go there. All right, sounds good. So just asking what it costs to get a master's degree. It's going to query to try to find out from the the vector database here what what it's going to cost. Okay, so now we've got table representation of data, which makes the most sense in this case. The agent decided to represent it as a table, and then our little front end module here allows for filtering of columns as well as uh searching go the the table as well. Okay, so yeah, there's another tool called there that would stack up in your history. Uh next, let's plan our visit here. So, we want to see what events are coming up at uh at Waterloo for orientation here. Yeah. So, again, we're pulling in some event details from the content type and we're presenting the user with uh this event cards. Yeah, so if we scroll down a little bit, maybe we find an event that we like. Uh again, this is tied back to structured content in Drupal and then that to particular tool for rendering out as kind of cards. So, let's see if we we can attend this AI uh workshop here. That looks interesting. Okay, so it's brought up a registration form for us to sign up for the event. So, uh let's uh make sure we get on the list here. Yeah, so you'll notice uh cuz I'm logged in, it's filled in my cheats, got my email address. Uh you could easily populate, you know, extra field data in in your user profile for capturing this kind of information um and then presenting it to the user, but you'll see it automatically pulls in all that data. So, this is a standard Drupal form. Um uh it's obviously rendered a little bit more interestingly and and it is at um dynamically uh selected by the uh agent uh based on our prompt. Yeah, and then you'll notice that it pulled in all the other events from our last query as well. So, if I search for, you know, events in May, I'll get different options in our swipe list here. So, Aaron will go ahead and uh register and then can you bring up the Drupal back end quickly? Yeah, sure. Let me pull it up. I'll just mention that's the Ajax form message that you get back in Drupal. So, you can actually set all that in on the web form. Okay, so Okay, so we just built like that simple page in the back end to kind of capture all this data in a a table. So, I'm just going to refresh cuz I just submitted the form there. But, you'll see yeah, we just submitted our um our form submission and went into the back end and then obviously a um email could be triggered from that. So, Hopefully, you've got the confirmation by now. Yeah, let's double check. Okay, so there you go. You can see uh it connected straight to Gmail. I have auto no responding, so I didn't uh accept it. But, um yeah, it came straight to our Gmail. Um so, you can see it's all very um uh neatly tied in with Drupal um ecosystem. Okay, so that's uh that's pretty much all the tools for now and uh you can extend the system obviously, add more tools, add uh more mappings to Drupal content, and basically build out any kind of custom experience you want. Um so, let's look quickly at some other features on the front end. We We have that history that we've been talking about. Uh so, Aaron's brought up the history here. You can see all the tool calls that have happened throughout this session. And uh you can go back and just uh revisit all the content that's uh that's stored with your user account. Um and then the reason we keep flagging stuff, we'll show you right now. There's a There's a feature called the notebook, which allows you to kind of collect content that that you found useful along your experience here. And um for those of us that uh have kids already in university, you know, whenever you talk about this stuff, you end up with with a stack of paper brochures um everywhere you go. So, uh if you We built this cool little feature in uh called the notebook, which allows you to uh download a PDF customized brochure um as part of the advising experience. So, there you see all the flagged content, uh housing, program information, whatever you want. So, we we haven't built all this stuff very very far, but essentially this will be like a full um you know, glossy brochure uh that you'll walk away with uh like a digital magazine, basically. So, that's where we're at now. Um just quickly before Q&A, we have that uh we have Stan speak quickly. Sure. So, this is another implementation of more straightforward. This is kind of like dipping your toes into the this uh digital experience, but um semantic search uh very easy to roll out. We've done this uh many times over the last year, now we have a very um um mature process for for integrating semantic search into Drupal sites. Uh this is one of our latest implementations. Um it's got some cool features. It's essentially like layering complexity on top of your search experience. So, uh Aaron's asked a question, it's answered it really nicely with uh with footnotes and references to other more traditional Drupal pages um with content. He asked follow-up, so he asked that asking about cost. Yeah, there's a little feature at the bottom there for like popular search queries. Those are they're not tied in with any system. It's just on a back end there's like a pill setting form essentially. But we we do have a feature where it will track like the most recent or so the most commonly asked kind of questions and then dynamically populate those for you. Um and if you scroll up you'll see full context history with your conversation. We also have the toggle back to traditional keyword search in case someone doesn't want to use AI search for some reason or if you run out of token budget because people are using so much of it it will toggle over back dynamically to regular search. Yeah, this is the case. This is probably as good as it gets for for Drupal. This is like a solar implementation with passive search. Um It's it's good but it's obviously got its blind spots with yeah semantic search. Yeah, and if you once you experience kind of perplexity or the Google's new AI mode kind of hard to go back to keyword search. And it's relatively easy these days to So we really think you know months or years all search will will ultimately end up being semantic. There's Just before we take a question I'll just say there's one thing. There's the hybrid approach where you get the both best of both worlds. You get keyword search as well as semantic search and it uses something called reciprocal rank fusion. And they basically takes like the the highest ranking from both comparisons and then it gives you the best um between the two. Yeah, we've done a couple of hybrid implementations as well, so we can uh we can customize this stuff as well. And the semantic search solution is actually built on the same AGUI framework that we just showed you with water wheel. So, it's a it's a nice stepping stone to that more native experience. Um and you you kind of get uh to test the the framework and and the architectural all initially that we talked about. We already have one question so I'm going to go ahead and Yeah. Okay, thanks. First of all, this is really cool. I can't wait to try it. Awesome. And then second, um one thing I'm I'm noticing every time I try something with AI, it's always using Python libraries. Mhm. Can you opine on is this a problem for the PHP community? Should we be trying to get these libraries into PHP so that we are first-class citizens? Yeah, that's a great question. Uh we we've obviously talked about it ourselves being in the Drupal shop. Um so, I'll tell you that the I've done a lot of research into this space um and the latest and greatest stuff out there is Python for the time being, but the Drupal AI initiative has made a huge step forward in the last year. Um so, our system does use uh Agno, which is a Python based framework right now, but because we implemented AGUI natively in Drupal, we can uh we can literally uh swap between uh Agno and uh Drupal AI's agent framework with literally a couple lines of code. So, uh because we started this about 6 months ago, we started building against Agno, but really uh we're actually literally working uh this week on a semantic search implementation implementation that's AGUI and 100% PHP and Drupal AI. So, it's definitely possible. We're I think we're on the forefront of it right now. But, you know, fast forward a few months, I think it's definitely possible to build this whole thing out on pure PHP. With Drupal, there is some limitations with their streaming stuff, but we also have a solution for that. It requires Sneak Peek or or some other kind of proxy solution for the streaming. But, but really that's the only kind of core fundamental thing that probably won't ever get fixed in PHP, but there is like alternatives like proxy that will solve that long term. So. Yeah. Does that answer your question? Well, yes, specifically for your project, I'm I'm concerned about the larger ecosystem. Yeah, I mean, it's uh I think they've got a head start, but like a lot of it there's a lot of momentum towards JavaScript as well. And then PHP will just feel a little bit outdated. So, I think we're we're pretty close to catching up, to be honest. There's not a lot uh left that isn't possible. Um But, yeah. Yeah, I was just going to say, I think like the the reason why um like behind the the Python is because a lot of these are built in Python. So, I don't know if you as soon as you start kind of doing something in a certain technology like PHP, you might kind of pigeonhole yourself into only being able to to use um that method. So, if you wanted to switch out your back end um that used Python or something like that, then um you might have a little bit more of a difficult time. I would add that like as uh enterprise specialists, that we can't really limit ourselves to one technology. Like there we walk into organizations and we're literally integrating with dozens of subsystems Uh some of them are on Microsoft, some of them are on Python, some of them are on Go. Uh so, the the key is to is integration. So, because everything is uh has essentially just based on APIs, uh you can build the uh the web framework on what works best for us, which I believe is Drupal. Um it's an end-to-end just integrate with the the subsystems through these APIs cuz there's no need to rewrite the the whole internet in PHP. Um just integrate with with what's in there and and let people do what they do best. That's our approach. Yeah. How do you know it's working? And I mean that from a consumer perspective. In analytics lines, how do you know it's successful? And from an audit trail perspective, how do you know it's not breaking through the the guardrails that you've set up? Yeah, great question. Um I mean, a lot of this is leading edge, so there's there's uh it's a bit of an art, but there's uh Richard, one of our guys, He's in here, too. Yeah, he's here. Okay. Well, he's an expert in uh in AVO. So, uh he could track kind of success from uh musical like from a traditional analytics perspective. And then a lot of these systems have backends that allow you to track um all the responses coming in. It's basically analytics from the AVO perspective. So, uh I'm not an expert in in analyzing that data, but the the data is there. So, our job is kind of make sure that we're tracking all that that data accurately and then um you know, AVO is pretty out. So, uh it's we're all kind of learning together at this point. And a lot of this is uh an exercise in kind of trying to be on the leading edge right now. Um you know, nobody wants to get left behind and I I was a I graduated university right when the uh dot-com boom uh hit in 98. Aging myself here, but uh this feels a lot like that, but I never thought in my entire life that we would go through a cycle like that again. Uh and we're doing that right now and it's like literally multiple times faster than the emergence of the internet, which is just insane. Like I can't I spend all my spare time researching this stuff and I can barely keep up to be honest. Like so um yeah, I still want to not get left behind. Uh yeah. So, what's the average cost to implement this? Yeah, another good question. So, uh it's complicated. So, first of the let's talk about the LLMs first, right? So, when you do semantic search or or or just more advanced um implementation, the first thing you need to do is get your Drupal data um encoded as vectors. So, what you need is an embedding model, which is a specialized model from the LLMs to basically uh parse all your traditional content, whether it's a PDF or databases or or images, and then it transforms it into this long list of numbers called vectors and it stores that in a vector database. Typically a Postgres database. So, those embedding models charge you you're probably familiar how like you're charged by the token when you use an LLM. So, the good news is embedding your content is relatively cheap. It's like orders of about 10 times cheaper than your traditional uh uh querying cost per token. And you only have to do that incrementally. So, then you do that once for your content. So, obviously if you have terabytes of data, that's going to cost more than megabytes of data, but it's still going to be like dollars per month. It's quite inexpensive, but that's a cost. And that may go up with more content. Uh so, that's the first thing you pay for. Then when your when your user is query uh uh the search feature, then you get charged by token for the LM. That's why we have the hot fill over in case you you use that like a little $100 per month for your search. Um uh for your search budget, then once you run out of tokens, you got to either uh refill the the tank or um shut it off and switch back to traditional search. So, there's that incremental search uh cost, but it's to be honest, it's not much different than using solar or Algolia. And uh we estimate like a traditional e-commerce site is still in the maybe tens of dollars per month. Worst case, it's in the hundreds of dollars. Um you do There's a lot of bots out there, so you got to be careful that people aren't like denial of service attacking you or just accidentally pounding on your search and and cranking up your token usage. Uh so, there's fail-safes for that, but um that's kind of just the cost of being on the internet. Um and then a more fancy system like this, there may be additional API calls that you need, but that's, you know, you just have to work out that architecture and and figure out the cost model, but relatively cheap. Yeah. It can be done, especially semantic search can be done for the tens of dollars of a month, which is pretty pretty well. It's almost it's cheaper than traditional search even sometimes. So, my uh one question is if we open it up for a whole external user base we currently have, but only for internal network users, but external user base, how it will differentiate between bad actors versus real users? Uh yeah, so What was the question, John? Oh, the question is how to differentiate between uh good actors and bad actors, uh specifically internally or externally? >> Externally. Yeah, so you open this up externally, uh how do you prevent uh or how do you detect bad actors? So, it's pretty much like traditional uh security there. So, uh We're not worried about this now. You can detect IP addresses, you can detect uh spikes in traffic. Uh there's more advanced technology like Cloudflare that can implement like a WAF. Um but it's getting trickier. Um we're seeing bots getting more and more intelligent about trying to break into systems, but it's not any different than traditional security for current applications. So, it's hard to deal with that question. Yeah, so with these um uh chatbots being open source, can you speak to how ImageXOR users navigate security or compliance? Like in in uh instances that you're that you're seeing. Yeah, so a lot of it um you know, by by selecting standard-based stuff that's open source um and uh well adopted, you know, we rely on the community to to provide security oversight. Um and then we rely on the people AI um modules as much as possible, you know, things that we trust and that we will uh actively um involved with. And and then we assist a lot of testing and and kind of standard best practice standards uh on the security front, making sure that uh all our code um you know, we we have a strict policy that all developers are responsible for their own code. Um like the usual saying, we don't accept any, you know, just um raw generated uh code at all. So, You wouldn't have like PII data being in like a record database. >> We do not allow that prompt even to go through. Yeah, you would set up these guardrails for sure. I mean, just say no, like don't answer anything to do with FTI. How much of this effort involved uh significant structural changes to their website and contents um versus something they already had? Was this in conjunction with a a full re-platform or Yeah, so this is For for now, this is the prototype that we're working on with them. Um they have a whole a whole bunch of initiatives that I don't want to kind of give away their internal strategy or anything, but uh this is very low um impact, so this is uh can be a standalone uh layered-on uh application. It just is it it's all Drupal. So, um it it it integrates with all their existing uh uh data within Drupal. Uh so, everything that you saw there was being pulled from Drupal. Yeah. This is a block that we created just for ease of use, but it uh basically just fills in an SDC props and con varying stuff like that. So, all these little fields that you'll you'll see here like the ant forms and all that stuff, it's just um available as an SDC. Um Yeah, but it just sits on top. It can be a modal or like a full-screen app. So, we decided to do ours as the the full-screen app. And then here's how we're kind of embedding all those tools. So, you can see there's like the events tool, the web form tool. And these are all tools that you set up in um uh your agent framework. So, we're using Agno. Um and it's it's pretty cool. There's uh in Agno, there's a team, and then the team orchestrates down to agents, and then the agents orchestrate or designate tasks to tools. Okay? So, that's kind of how that that works. It's not like a swarm. A swarm is like everybody coming in trying to figure out what's going on. It's it's a lot more structured like that. Question. I I mean I see that all the data exists within this one beautiful site. How would this work in a more distributed environment where like every faculty has its own set of events sort of thing? Yeah, so with that the databases you can pull from any number of sources. So in the demo a lot of the content is coming through other APIs. And then we have other implementations where we're we're querying PBS just in you know any kind of storage. So we can definitely pull data from multiple sources. Just curious about the the guardrails you currently have. Like what happens if how do you determine what people are asking? Let's say they say who's the best heavy metal guitarist in America? Like or something you know something completely unrelated to this. What's in place right now to handle that? Yeah, so that's built into the the framework that we're using. You can set it up at various levels. >> Okay. Yeah, so these are the guardrails guardrails. Sorry can't help it anything to do with that. That's we'll still pass you some tokens with it. I will say though that he's so Aig knows really good at caching and stuff like that. So like if someone like this uh somebody else asks the same question it's not going to cost you anything. I think we're over time if you want to get to the keynote but come by the booth and talk to us anytime. Thank you.