Your Client Wants AI. Their Lawyer Just Said No. | Paul Yuen, Jason Zinn, Kay Lima

Hi, thank you so much for coming. Um, so firstly, just I really appreciate everybody coming out today. Um, and secondly, I wanted to thank uh the folks at Modern Tribe working behind the scenes to make it possible for us to be here today. Um, as you said, my name is Paul Euan. I'm the product strategy principal at uh, Modern Tribe by Nexus. And uh today we'd like to walk you through a uh couple of scenarios uh with some case studies around um some clients who wanted AI features built for them uh and what those outcomes look like. Uh so not long ago um we pitched almost the exact same feature to two different clients. Uh the idea behind both uh was that we wanted uh to utilize AI to help guide the visitor um towards helping us understand who they were, which audience segment and what they were looking for on the site. um our client's legal team, one of them said yes and uh another one didn't want to take on the risk if AI were to spit out uh an inaccurate answer uh to a customer or to a user. Um so we built two different things. It was the uh same idea, different outcomes. And that's a story about judgment over the hype of AI. So let me set this up. um visitor lands on your site, you've got a ton of content on your site. Um and you've got one chance to point them in the right direction before they give up and walk away. And the problem is that you serve a lot of different uh audiences and they all have different goals in mind when they come to your site. Um but they all land in the same place to start with. So, what do you do? Um, we pitched AI to uh try to determine two things. Which audience segment are they? And what is it that they're trying to find in the site? So, uh, let's break it down today. First, Jason's going to walk us through the story of the client who went with AI. And then Kay is uh going to walk you through what we built for the client whose legal team gave us a yes and which is basically a no. And I'll take you through the decision framework that we used um so that if you're faced with something similar in the future uh then you can uh implement that. And then lastly, we'll give you the secret sauce so that you can try to build it yourselves. With that, I'm gonna hand it off to Jason. I learned you talk into this one, not this one. >> Oh, hey. Strange how these microphones work. Um, okay. So, I'm going to talk a little bit about the program Advisor that we built for Cleveland Community College um that uses an LLM. Probably haven't attended any talks yet talking about AI, so this might be your first um and probably hopefully your your last for a little while, I'm sure, too. Um so yeah so the program uh finder this is an existing tool on the community college website that allows users to browse the program catalog using a series of filters and keyword search. Um it works it works fine for most users but there are holes for users who don't uh have a good understanding of maybe what they're looking for. And there's also some failures with keyword search itself as I'm going to show in a in a demo in just a minute um that can not find all of the matching programs you would expect. So the new program advisor is uh trying to fill this gap by starting off the user with a natural language search that allows a tailored response to suggest personalized programs to the user along with retaining the original filterable interface that users can still use to browse the full program catalog. We also added added an additional layer of uh routing in here that allows us to catch um searches that are not related to the program catalog. Um that's that's kind of a new part as well. So I just want to take you through uh the program finder, the existing one uh and then the new one. So we'll start here by looking at the existing program finder. As I mentioned, there's a series of filters here. These are all tied to taxonomies on a program's custom post type. And then we have the keyword search at the end. And these all allow you to filter and search the the full catalog below. If we go look at the new program finder page, you're going to see it starts at the top with the program advisor, which is the natural language search. And then below that, we continue to have the filterable interface. So these are kind of two systems that can work in tandem with each other. And then of course the keyword search is gone. So I want to just show you how this works. So, uh, I'm going to start with a basic keyword search on the new system. So, I'm going to search for healthcare here. This is going to get sent over to the LLM along with program information and the program advisor is going to come back with the summary of its results. And then three, in this case, three different routes of programs is identified for the user um, and suggests. Because we've only used the keyword, the robustness of the the actual programs that it can provide is is limited. But the more the user adds nuance and detail, the more involved it can get and uh personalized. It's also gone ahead and identified the health science area of interest here and selected that for the user and then um sorted these these uh programs below. So now I'm going to go do the same kind of search on the old program finder page. So we'll do healthcare here as well and it will perform like you expect. So we do get our results. Um, but we don't really have much information about why these results were chosen. Um, we don't have a lot of nuance to it or personalization or any additional um, ranking to the results that can be done. We also, if you notice, there's only three actual programs selected, which is part of the gap uh, in the actual keyword search that the LLM can pick up on. So, now I'm going to try some more natural language type searches. So, here's a user looking for short-term healthcare program that will help me get a job quickly. So, I want you to pick up on those keywords, short-term healthare, and get a job quickly. These are the same things that the LLM is going to be good at identifying. So, here the program advisor is running and and you can see immediately in the summary, it's identified short-term healthcare certificates and diplomas that will get you working quickly. And then it's tried to group these into two different um buckets for the user. One direct patient care and then another that's more administrative related towards the medical field. And then additionally, it's applied some filtering again with health science, but also now with a duration filter on it too to really refine those results for the user. And you can see all those below. And then of course, because we retained the old interface as well, the existing uh interface, you can still reset this and have the full catalog to browse if that's not the query that you, you know, if the results above aren't matching what you want. I'm going to go show you the same query on the old program finder page as well. It's going to do exactly what you expect. It can't handle this kind of search and so it's not going to find any programs. And so I think it's a pretty obvious uh gap there that we can help fill by involving an LLM. Next, I'm going to go back. I just want to show you a couple more searches. So one's this next one is going to be a different search for the program advisor. um showing a user that's going to provide a little detail on their different interests and see if we can find some programs that match those. So here's a user say they love working on cars, understanding the science behind technology and helping other people. So the program advisor is going to put this together. And in this we send over the query along with all of the programs in a truncated form for the LLM to work with. So here's a couple different paths um it identified. So, hands-on tech and helping others and then in a dedicated automotive path as well. And again, you can go below, the user can go below and continue to work with the full catalog should they want. I want to show just one more query example here, and that's to show a little bit about the chat routing. So, this is going to be a user that would search for help with their resume. Uh, clearly not related to programs at all. So, we've built in some fallbacks in the system to handle this. You're just going to see a message here. Uh that's something that actually controlled in the admin itself. Uh and I'm going to show you a little bit about how that works. Okay. So that's the way that it works on the front end. I wanted to just talk a little bit about how we built this. So, one of the things uh I want to really highlight about building a feature on a website that involves an LLM is I think they can be on one sense quite straightforward to build and and simple, but really fine-tuning them and making them both reliable and also give value to a user is is challenging and that's where it requires a lot of iteration and a lot of tweaking things and just playing with different settings and and working through it. So we actually tried to work through three different architectures for this. We started with an agent with tools approach. Uh and that proved to be unreliable for us with whenever the LLM would get confused, it would often come back and actually narrate the tool calling pros itself rather than actual recommendation of programs. Uh and that's not acceptable. So we we backed off for the next approach and and really find uh let the LM have a narrow path and only act as an intent parser. So in this architecture, the LLM was only responsible for um picking out keywords and taxonomy filters from the query and then passing that over to a fully deterministic uh search layer. And while that was completely reliable, it just didn't add a lot of actual value beyond what was already available on the site. So the final approach was to kind of open it back up, involve the LLM and actual program recommendations, and then you continue to use a fallback deterministic layer for the reliability to remain intact. To look at that actual final architecture, the way it works is a user initiates a search. That search is going to go along with the program catalog over to the LLM, which is responsible for up to five different things. First thing is the LLM is going to classify what the user is trying to do. So that's are they actually looking for programs or is it something different like they're looking for help with their resume. If it does determine that they're looking for programs, it's then going to try to extract keywords and filters um taxonomy filters. It's also going to try to detect exclusions. So for example, if it's a a student looking for health care programs but maybe not interested in nursing, it's going to try to pick up on that. And then it's going to group the program program recommendations together and write a summary and then hand that over to the deterministic layer. Once that comes back, we're going to validate the response from the LLM. So we're going to make sure every program there is a real program. It's published. It's not been excluded or matched by any filters that should be left out. And then we're also going to do a deterministic search layer as well. And that's going to help backfill any programs that weren't found and also provide a fallback in case there's a network issue going to the LLM. We still have a fully functional search for the user. And finally, that just all gets passed down to the uh view layer, which we looked at before. I wanted to also just to take you through the admin side uh and show you kind of how we made this. So first and foremost is we use the new connectors setup from WordPress 7. And so we've set up the AI provider for Anthropic. So we can uh use the anthropic API key directly in here like so. We built this as a custom block. Um and so this actually gives us a lot of flexibility to use the program advisor across the site wherever we would like. The page that I showed you on before had the catalog below it, but it's actually not necessary, though they can work together. So, I'm actually going to remove that. You see, we have the three blocks here. I'm going to take out that third block, which is the kind of query loop. And so, we just have our program advisor itself. Right? So, I'll remove that. And you'll notice here I'm going to go through the fun exercise of adding a little bottom padding. Now that we've moved that, I'm sure none of you have, you know, had to do that with any of your blocks before as you re rework things. Um, but I want to make sure it looks good for all of you. So, let's search for healthcare. I want to call it that while this looks exactly what we would expect, I wanted you to also notice the it came back immediately. And maybe you caught that. And that's because we actually we have a built a fivem minute cache layer in there for recurring searches. So, I'm going to set this back. Go back. And I just wanted to take you through the program's custom post type a little bit, show you how those are structured. They're pretty standard. Every uh program has a title, a description, some metadata, and some taxonomies. We're not including any of the metadata for the LLM. It's not very useful, but the taxonomies and everything else is sent. The description is sent in a truncated form um to keep token usage down. And that's a standardized kind of description. Then we also built some program advisor settings. Let me just pause it right here real quick and talk about this. So some of the settings we built on here um we gave the admins the ability to control rate limiting and token stealings for users um to help with you know bot attacks or anything that might get out of control. And so we wanted to put that directly in the admin interface so that code deployments aren't necessary um if something like that does occur. And also as we start to understand what the actual token ceiling should look like for a normal use case, we also built in we have a 1.1 feature built in to start using um memory and make our searches a little bit more robust with session memory for subsequent searches. And the next thing is we built in analytics to this. The existing program finder doesn't have any analytics at the moment. So we don't know actually how users are using it that well. Um, so we're going to start collecting that and see what users have been up to the last 30 days, what the latency like is like on the search, what are the top recommended and top searched for queries, um, any queries that don't return any results. And we're also giving the admins the ability to clear out um, personal data in case something comes through that's a user doesn't intend or actually put something sensitive in. Next, I want to talk about the prompt. So, here is the system prompt, but at the very bottom is the the full preview of the prompt sent to the LLM. We're giving the admins the ability to control these top two paragraphs, which is the voice and tone. The rest of it, which is critical to the actual structure of the output, we're keeping in place. But, so I want to just kind of show you real quick how this works. Um, I've put up here, this is a, you know, an admin. They're having a bad day. So, they're not feeling so nice to the students this time. Um, and so they they just want to update the voice and tone to say, "I'm a grumpy, worn down program adviser. You've been at this for far longer than these kids have ever been alive. What do they know about what they want? You're not here to be warm and fuzzy. Give it to them real. Break them down before the world has a chance." So, we've got this delightful new personality. Um, one thing I will call out is while this is fun in games here, we don't want them to actually mess with the programs recommended themselves. And so while they can adjust the voice and tone around the the summary and how they're describing what's been presented, the actual programs are still going to be presented to the user. So they can't be that grumpy. There's there's limits to the grumpiness. So you'll see when I save this, it's going to update the preview. So you can see exactly where that voice and tone is is adjusted. There it is at the bottom. Okay, let's go take a look at our our grumpy uh advisor. Sometimes I'm always disappointed they can't get quite as grumpy as I would like, but this one did pretty good. So, we're going to search healthcare again. You notice that it's uh it's no longer cached. So, if you do override the prompt, um it gets it clears the CL cache. So, you can test that real time. Okay, let me pause it. I want to make sure we can read this. So, healthc care is a broad field kid. Not too bad. You've got a medical support roles, clinical texts, nurses. Pick what actually interests you instead of just saying healthcare and wasting everyone's time. So, so we can confirm I think that that voice and tone is uh is working. And hopefully you can also notice that's on my local environment up there, not the production site. So continuing along um the last thing I just want to show about the the admin side is the the routing. So we built in some different routes here. So ambiguous queries is you know anything that's not caught by another route. And then we have career services, financial aid, academic advising and so on. And so the user can add the admin can add a message and a call to action to help get users to the right part of the site. The next part I just want to quickly touch on is that we did include a CLI tool with this. So the example on the right is is if a developer has run program advisor search with the trace flag. This is just helpful for development and for debugging since you can't really get visibility on a lot of this information on the actual front end. So you can see on the right we've got the the full reply and in the trace itself we can see the query class. So you can see they've it's identified this as a program search along with the filters identified keyword terms and any exclusions and the full grouping of programs and at the bottom we can see the exact search WP query that's going to be run and uh the actual ranked post IDs output to the user. So we can validate everything here as well. And we also use this for a golden query kind of system that we hope to uh evolve as we have more analytics about the searches that real users are using. Okay. So we tried to build this in a way that is scalable for the future. We're leveraging WordPress new WordPress features where we can like the connectors I showed you and also some tools for prompt building. analytics built in is going to help us to start understand the users better and build this tool better and better and make it more and more useful. The scalable infrastructure is set up by starting off the chat routing and that's going to help us to expand this beyond the program finder into potentially the course catal a larger course catalog and maybe even the entire site and we also hope to start using session memory and multi-turn conversations um for future iterations of this to make the advisor as robust as possible. So in this case AI is a really solid choice. There are clear gaps I think we saw at the beginning where certain types of user searches just won't work and can't pick up any programs. Um so it made sense and on top of that we had a nice balance in the size of the data set and the type of data we have there that's very structured where we could um manually understand the results and verify them pretty quickly and pretty easily. With that, I would like to hand it over to Kay. She's going to talk about a case where we try to solve the same problem um not involving AI but in a fully deterministic way. Thank you. >> Thank you, Jason. So, what we just heard about Cleveland Community College is a success story with using AI for the future. And this is something that we're hearing more and more from our clients that they want to find ways to leverage AI. And we've been excited to think of new features that we could incorporate it with. But the truth is is that this isn't the case for everyone. And there are some legal teams who have concerns about AI potentially misrepresenting the company in results. HR groups who have very real concerns about employees being given the wrong documentation. And this was the case with Turner Industries. There's also concerns that can happen around budget. And so we still want to make sure that we're able to give the users a dynamic, personalized experience even if we're not using AI. So how do we do this? Our solution was to create a custom block that is powered by a combination of personas and goals or audiences and what they want to accomplish. And it's deterministic. So there's nothing that we're leaving up to chance. As the visitor coming to a site, what you'd experience is being asked who you are and what you need. So what content are you trying to find? What pain points do you have that we can solve? And the way we built this was through the custom block, but also through a custom post type and taxonomies. And it's the taxonomies that are matched together to then curate the content which means that we don't need to have a model in the loop. And so this deterministic content on rails approach means that every path is authored and approved ahead of time. And that's for all the stakeholders. That's the editorial workflow. It's legal. It's HR. It's anyone who needs to see this before it's public. And nothing is being generated on request. So, I'd like to show you what this looks like for Turner. This is their contact us page. As we scroll down, you can see there are the two yellow dropdowns. And this is first the audience dropdown where we have current employee, job seekers, and vendors. And as we select one, in this case we select job seeker. You'll see that both the content is updated as well as the goals. So this second dropdown does have its um predetermined goal but then they have the option to select others and you can include any type of content here. So here we see a form and each combination of audience and goals can have a different form that routes to a different place. It can have documentation or contact information. And again, it's fully tailored to each combination. And so this shows how for current employees, there are many more options that they can select from. And again, this is making sure that those current employees are getting the correct documentation and there's no HR concerns. And so this isn't the only time we've implemented this approach. We've also built this for Georgetown University in Qatar for their admissions site. And this was a completely different use case. We have different audiences. In this case, it's the prospective students. It's their parents, counselors, transfer students, and they needed different types of content than what we just saw with Turner. And they all needed to have that content displayed to them immediately without having to go through many layers of the site to get that key information. And so you can see this is a different look and feel, but it is actually the same persona by goal uh approach underneath it. Turner rendered the content as a row of blocks, whereas this is a true mosaic grid. And so here on the admissions landing page, you can see we start with the hero, some quick links, and the chart your path to GU is our mosaic block. So again, we have the two dropdowns, but in this case, it defaults to our primary audience, which is prospective students who might be applying. And you can see we've been able to include video as well as imagery. And this will do the same thing as we saw with Turner where it will update once you select a different audience type and associated goal. So here in the case of transfer students, it updates the content and it also updates the goal to default to discovering Georgetown. So there's a lot of flexibility here with how you can display the content and customize it to a particular brand or styles or layout per site. And then the last example here is we have applying to GQ where you can show the timelines right away and then immediately get them to that application form. So now here's the editor experience and you can see we have the custom post type called admissions mosaics. These are all the posts that end up being that uh curated content on the front end. In this case we identified 15 paths. So 15 posts and the audience types and goal types are those taxonomies that are used to match for that content. In the editing view for that post, we can see we have the mosaic layout. And this is a pattern that we created. We actually created a few different mosaic patterns that the client could use as a starting point. And these can be edited and changed over time, still going through that full review process before they're published. And then on the post itself, that's where you add the audience type. So that was parent. And then the goal type of discovering Georgetown. And that determines the matching. So the outcome for this type of approach is again that full editorial control. So every path can be reviewed before it ships. Because we're not using AI, there's no token cost. And so the budgeting here would be around building the custom block, styling it, and then of course the content creation. And we've seen this as a proven path to success for those sites that do not want to use AI or don't want to use it yet. It's something that can actually be extended to hook into an LLM in the future. Um, but in the meantime, this is something that they can reuse across different areas of their site, for different use cases, and it's something that we can reuse for different clients, no matter what their site looks like. So, with that, I will pass it back to Paul, who will talk you through the framework for deciding when to use AI and maybe when not to. >> Thank you. All right. So, um, we've seen a couple of different outcomes. Um, now let's talk about how you decide which one is the right fit. Um, for the examples that you just saw. Uh, it wasn't the tech stack and it wasn't the team that made the difference. It was determining the audience, um, the risk tolerance and the cost if I AI were to uh, give someone from that audience uh, an incorrect answer. Um, like Jason had showed us, uh, there's a lot you can do to, uh, ensure that AI is working with the right information, uh, to help mitigate risk, but, uh, the risk is never zero when you're working with AI. Uh, AI is great when, um, the cost of a wrong answer is relatively low. Um, it it works great when the content space you're working with uh far exceeds your capacity to manually uh write every potential path. Um, and it works great when your budget and your timeline are sufficient enough that you can uh adequately loop through uh testing and refinement uh iterations over and over until essentially good enough is good enough. Conversely, if the risk exceeds your tolerance, uh whether that be legal or uh reputational, privacy, security, um if handcurating every path is actually a requirement, um or if uh the number of paths that you have is manageable, then deterministic uh is a great approach. So, here's the secret sauce. whether or not you use AI is kind of irrelevant. Uh the first thing you want to do is figure out who the audience segment is. And the next thing you need to figure out is what is it that they want to do on your site? Why are they here today? What are they looking for? And then uh build a delivery method that dumps the right content in their lap in as few clicks as possible. Again, you want to know what it costs up front. uh if the user is presented with a wrong answer. Um you want to determine if the client is willing to accept that risk. Uh you need to measure the volume of how many permutations uh you're looking at. And you need to determine if you can support uh the number of iterations it's going to take well after launch uh for a tool that's actively running AI for every user. So please exercise judgment over the hype and excitement of AI. Um at the end of the day, the right answer was different for the same pitch twice. That's not indecision. That's judgment. Match the pattern to the client's stakes, not what's exciting to build. And AI is a tool, not a mandate. It's not always going to fit. Uh, I'll open up to questions. Um, I don't know that we have a ton of time, but uh, if we can't get to your question today or if you think of a question later, uh, feel free to come by the Nexus booth. We'll be hanging out there after this session. What questions do we have? >> Yes, please. I'm Dana. Thank you. Um, there we go. For the clients that are all about the hype, want they want the AI solution, they want all the solutions. Uh, I could briefly see it through the video and the slides, how are you presenting the compliance side and also baking that in even if the client doesn't care about it? Um, are you guys taking steps just to make it make I was just in the the legal compliance seminar? So, I'm thinking about that too. And our biggest recommendation was disclose when AI is a piece of the machine. Um, so just wondering if there are facets that you didn't display for the AI solution. >> Yeah, sure. Um, yes. So, we're trying to do what we can as far as uh we we did provide them an updated guide towards privacy policy information since we are involving the LLM um and discuss with them before going through with this of what is being sent to the LLM and if any additional action they want to take to strip out personal information. In their case, they weren't they don't view this tool as being um one that's going to be hammered with private information, right? >> But we can certainly do more always. um and then not storing anything within you know for model training that's already built into their um their system on their because this is coming from the client's account at anthropic in the end. Um so that's what we're doing and then just again like some of the security stuff around trying to protect how it gets used and how often it can get used. >> Cool. Could I have a followup too? >> Yeah, >> please. >> Let's take it a step further. Let's say a prospective student finds the the program they want. Um, if you were to then I guess you could avoid the LLM at this point where they do submit a form. Would there ever be a scenario where you would take in that that private information or are you trying to engineer solutions that are as minimal touch point to personal data just for better compliance and longevity? I I just have a client that they want the form, they want the LLM to be managing everything and it's like you're just exposing yourself to a lot more risk in that scenario. Whereas this solution kind of engineers around that where you still have the personalization, but you're not getting the details or you're diverting the details further down the road. I might have just answered my own question, but do you want to speak to that? >> Sure. Just so just to make sure I understand correctly, you're saying are we kind of submitting that information over so they can collect it and keep that. Is that correct? >> Right. You you Yeah. There's no personal information in the querying that you're submitting to the LLM. >> That's correct. >> Kind of punting that down the road. >> Yeah. Um it's it's not part of the system and yeah, I think I think any anytime you talk to the client, they're going to be interested. Hey, can we pass this on to a form too and collect it? You know, they love that stuff. Um in this case, it wasn't necessary. I thought I think we've avoided it as just as you said. Yeah. >> Um but yeah, if if a client wants it, we could go down that path and then have to handle more risk with it like you said. >> Awesome. Thank you. along those lines. Um, Turner actually did kind of want that initially. Uh, they wanted to be able to um integrate that into the search bar and kind of learn what the user's intent was and ascertain like what they're looking for and at the same time collect enough information that eventually they could push them to a contact page that would be prefilled to some degree. Um, we didn't end up doing it and partly because we we ended up getting blocked by legal before we even went down that path far enough. >> Yeah. Yeah. >> Lower barrier to entry, too, right? You don't have as many things you have to fill out. They're kind of pre-filled for you. Um,