Using Conversational AI to Support Federated Certificate Administration in Higher Education

Hi. And welcome. My name is Chaitanya Gunupudi. I'm a senior platform cloud engineer and AI researcher at the University of Maryland College Park. Uh the briefing the briefing is about something we actually built and deployed in production. A conversational AI system for managing digital certificate across our institution. But it's not about the design principle that I'm going to talk about today. And I believe it is very essential um for the artificial intelligence to the system deployed in operational constant context that is like confidence our design frameworks need to be deployed when we are deploying AI infrastructure in our cloud environment. Over the next 30 minutes, I want to share both the practical story and what we built, what we work on, what you get surprise. And the governance and the governance framework which is which we use is very safe which is is very safe to deploy. Here and today next slide. So here what we will cover. We'll start with the operation problem. Why certificate administration at research institutions have genuinely uh is genuinely hard. We'll explain the urgency need around like 47 47 date mandate that's coming for all of us. From there I will walk through our conversational AI problem, AI research problem. And this is this is this is what I want to share um in the AI design framework. In this kind of design framework, um we'll be talking more about like a confidence aware framework, like how it counts Yeah, how it counts and how it behaves and it is like more uncertain. We'll like we'll look into the architecture, the pilot results, and with some open questions and takeaways you can use at your institution. So, that is the main problem. So, because uh due to the due to this we are getting like more uh more more help desk tickets, questions, all kinds of practices which we're going to uh discuss today. And let me start with this real problem. Uh so, right now in our institution at the University of Maryland, College Park, um we have around eight we have around 8,000 SSL certificates across um across 39 departments and 150 plus domains we have. Every one of the certificates eventually expires. Every problem Every renewal involves a departmental administrator, often someone who is not a IT specialist who has to figure out the right process. What actually happens? That is like they open a help desk ticket. They ask us some questions. They ask last year, and what certificate type do they need? And how do I validate my domain? What does this error mean? Our IAM IAM team answers the same questions repeatedly repeatedly every year and uh every year. To resolve this, we designed a AI assisted automation pilot and cuts almost like 64% of the service now tickets. That's the uh That's the uh That's the number our leadership board has been uh justified. Before what's coming for all of us is like 47 47 uh 47 days certificate validate mandate in the future, like in the mostly that the target date is is March 2029. The public TLS search can only be a valid for 47 days. That means institution that renew once a year will need to renew eight times per year. So, that is very impractical if you're planning to design using manually. So, we need to have some kind of automations. I think this is the this is the similar problem which I was talking about like the scalable problem. How So, for this for this we have as as we mentioned. So, we don't have we don't right now have unlimited unlimited certificate. We need to make sure the 47 day validate period is is present validate validate date is and date is present. We need to make sure we need to have a proper proper automations. That which we're going to talk about like more more in more more in today. The next slide. The next slide is about the InCommon. It is the It is the vendor It is the vendor who who have and about the 47 day validate. Let me give some context about the federated landscape. InCommon is a national trust federation operated by Internet2 connecting more than 1,000 US research universities and federal agencies. It provides unlimited SSL EV client and code signing certificates under a single annual fees. The CA browser ballot SC00081V3 approved April 25 mandates a phased reduction in TLS certificate validity. Like today right now it's 398 days. March 2026 it's 200 days. March 2027 it's 100 days. March 2029 is 49 is 49 days. Think about Think about like what 47 days means operationally. A certificate you used it to renew every 1 year need to be renewed like eight times per year. For any institutions managing like hundreds or like usually hundreds of certificates, manual processes simply not viable. You need to have a proper automation and and almost right now 40 to 60% of the institutions doesn't have a proper They don't have a proper automation tool to renew these kinds of certificates. So we need to have a proper automation to renew these certificates and a proper pipeline is very important. So that is where So that is where like we need to immediately validate the certificates. Otherwise they will be compromised and some of the search will be decommissioned like down the line. So we need to make sure the expiry date, the automations is very mandatory. We need to have a track to set all kinds of reminder and avoid and avoid any kind of outages. To resolve these kinds of stuff, we need to have a automation that we're going to talk about like more deeper uh deeper deeper in the presentation. Uh the next slide is about the the thing which I'm going to talk about is our approach, the conversational AI which we are using in our institute. Our approach is built around like four principles that I want to name explicitly. Like because they are not what made this safe to deploy in an operational context. First, grounded. And it's not very generative. The chatbot like the chatbot retrieves we need to build a chatbot which will be talking which which will be talking in with in common or Shitty Go or any kind of federated services like uh in EM sign. So any kind of REST APIs you can able to talk you can able to talk with your conversational AI. So first, it should be very grounded. It should be very generative. The chatbot which you are planning to design should retrieve answers from a authoritative institutional documentation and in common and in common policy resources. It doesn't improve. Every answer should have site its resources uh sites its resources. That means one of the confidence AI framework. So if you don't site the resources which the data it is coming from, the confidence level of the customer or the user who is trying to use um will be very less. So only like we need to make sure it is the it is getting properly cited. The second, action warranted. Wouldn't want another FAQ use page. Users can can check expiry status by domain, initiate a renewal, open a ticket through a conversational rather than rather than going like different steps in an in common in an in common interface. So we need to make the convert like chatbot through very chat through very conversationally. And it it need to provide these kinds of operations, which is very important. So, that is one of the second framework. The third framework is the Conversational AI design. This is the principle I want to spend most of the time in this briefing, like explicitly like uh explicitly, the threshold determines when the chatbot expresses uncertainty. So, so whenever we get a data from a chatbot, we need to make sure it is properly confident. So, we need to give a confidence confidence number or like how much confident it is about. So, based upon the confidence uh the the data which we are getting uh which we are getting should be properly determined. So, if the confidence level is less, we need to make sure a human is present always or it should decline the answer. The system is designed to say, "I don't know." rather than a guess because we are dealing with the proper certificate data, which is very important research data. The domains, all these are like very important for the institution. So, the answer which we are getting it should be properly validated. The fourth, audit complete. Every interaction is logged. Every operation is tied to the requesting identity through single sign-on. And you need to make sure all the actions which is present in the conversational AI chatbot which I'm I'm mentioning for the certificate automation should be properly audited and it should be properly logged. This is These are the minimum frameworks we need to have when designing a a conversational chatbot for a certificate automation. So, now I want to talk about how the chatbot handles. in terms of scope. The chatbot handles like four four category four categories of operations. Like one is issuance guidance. Like which So, in the issuance guidance like which sub-type for your use case, domain validation requirements, and how long issuance takes, and how long the like certificate authority recommendation based upon institution portfolio. That is called issuance guidance. The second step is renewal support. So, this is where like our highest priority of like service now tickets get generated. Like we need to make sure when we are building renewal support the chatbot need to look up the existing domain. It What are the step-by-step renewal guidance for every server setup? ACME automation. So, is there any ACME automation setup? Is Right now it is possible. And how does the 47-day validity planning? So, right now we have 2 years from 2027 to 2029. Like what are the steps we are trying to follow for this? And the third we are we need to make sure the chatbot explains the process and how the revocation is appropriate. But any actual revocation action immediately escalates to a human administrator. No exemption because the reading operations you can you can make like however you want. But when you are writing or deleting the certificate, we it need to make sure it is properly notification has been sent properly to the administrator who maintains the domains. And a successful There should be a two kinds of notification. One is initiated, the another one is successfully done. So, those are the two revocation assistance. But if there is any happens, always a service now ticket need to be automatically generated and escalated to a human. The second one is policy and trust questions. As per certificate authority grounding through rag chain of trust explanations and hard explanation of federated trust decisions. So, we need to make sure the policies are explained properly. So, only like we designed rag a rag method where we deployed all of our documentation and so that so that the customer or administrator can able to review all kinds of policy documentation through our rag analysis, but using a chatbot interface so that he can able to get all kinds of policy information rather than going all over the place. The key design across all four categories, the chatbot knows how it handles and what it doesn't and when it when it's outside its scope or below its confident level, it knows how to how to like propagate through a human human interface either through creating service now tickets. Now, the [clears throat] next slide. The next slide is about the architecture. Let me Let me walk through architecture architecture briefly. Entry points are three, a web portal, a Slack chatbot, which turned out to be our most popular interface, and and an email trigger for the renewal for the renewal reminders. All entry points authenticate through Shibboleth. So, that is very important for an institution. Shibboleth. So, Shibboleth through InCommon Federated Identity. The chatbot know who who it is talking to and what department institution it is affiliated with and what certificates are associated with the Yeah, each department with each department. From there, request go to a confident confidence aware LLM framework, and then it is propagated to a plug rag engine. Before the model generates a response, it it retrieves relevant documentation through from our corpus. So, in common and Sedico, so our mean our corpus is present about in common and Sedico policy documents, our institutional procedures, and CEA specific configurations. So, all these are present in our corpus name specified directory. So, the model generates a response grounded in that retrieved content, and and it will make sure it it sites all the resources. If the confidence is very low, when the output is when the output we got from from the AI framework is very low, it either redirects to a help desk ticketing tool or help desk ticketing tool, and opens a ticket and notified to the on-call team. Everything is logged, every query, every response, every confidence level at a decision time. That's audit trail. And everything will be logged so that if something happens in the future, it will be it can be easily tracked. So, that is very key important. This is the like high-level architecture when to design to design in the conversational AI chatbot. The next slide. The next slide is the hardest part, policies through an LLM. I want to be very direct what was hardest because because I think it's a mostly general generalized lesson which which anyone who is designing the rag framework they can able to understand that. So, getting operational facts facts right is achievable like when does my certificate expire? It's a It is a retrieval problem like on like through rest APIs. So, getting policy document getting policy documents is much harder. Policy documents are not in sections. In common mostly relies on these policy documents and subscriber agreements like are written through a very legal compliance not user facing operational guidance. So, mapping that language to a practical user questions requires significant prompt engineering and document pre-processing. So, because there will be like more hallucination on very edge cases like example like when a user asks about multi-domain wildcard without no standard sign configuration the model can provide plausible sounding some kind of incorrect answers. So, due to that the rag reduces the risk does not completely eliminate these kinds of edge case scenarios. So, we need to be very make sure make sure about this. And federational context is contextual when a certificate is trusted by feder- by federation partners depending upon issuing CA the certificate profile and federation metadata configuration. These are deeply contextual decision than the chatbot which we are making generalizedly for the audience. So, only like our mitigation the confidence thresholding per query category mandatory is very important. So, some of the framework which are already mentioned is mandatory source citation on every answer and 12 and escalation strategy like what kind of escalation strategy which we have whether it's always going to a human when the answers are when the answers are not accurate and regardless of regardless of a model confidence when a confidence is below threshold the chatbot explicitly should open a service now ticket. Next slide, please. So, the early results when we deploy uh so when we deploy this chatbot to the departmental administration administrators across our school and some departmental at university. So, roughly 70% 70% of the routine renewal questions were resolved without querying without querying high escalation to the IAM team. So, that's a meaningful reduction in help desk load like when when when at least some of the at least some of the tickets has been resolved through this rag analysis through this analysis. So, first response time. How long an administrator awaited for a useful So, some of the analysis we have done is how how long an administrator has been waited to get a right answer when from So, usually like how much time a a ticket has been raised from a help desk from help desk. We just compared the help desk service now ticket creation versus the chatbot giving answer is ridiculous like the answers are very fast. We defined 12 escalation rules or categories based upon the pilot we built. The number will grow as we expand. What worked well like step-by-step renewal walk-throughs are highly valued by non-expert admins. The certificate type decision The certificate type which we mentioned the decision trees reduce the wrong certificate requests meaningfully and domain status lookup was the was the single most uh single most single most user question where every people are trying to access this access this. So, what really surprised us is this is most important fact is users initially over trusted the chatbot more than we expected like setting setting and communicating expectations through significantly is more important like we also found that the chatbot had two ti- two times the management of the web portal. So, the interactions are very high rather than comparing with the web portal. What didn't work out really is real-time what are the real-time certificate retrieval retrievals was too high for some users. The and and we considered some of the revocations need like more guardrails than we expected like some of non-English speaking staff had inconsistent experience quality. So, that is one thing which we are trying to improve in our frameworks. Next slide is about um human in the loop design framework. So, one of the most important design framework was was made being explicit like very explicit about where AI ends and human begins. The left hand the left side shows when the AI handles autonomously like explaining sub types, walking through renewals, answering policy questions, retrieving expiry status, opening informational tickets. So, those those are the perfect steps which is designed through our AI rack framework. The right side shows like what always humans what always requires a human for any query with no exceptions. All the certificate revocations, feder- federated trust configuration changes, wildcard and multi-domain SAN edge cases, anything below confidence threshold, and anything students or patient or health data or um or award management's data or federated data. So, these for these kinds of data, always human or ticket management system is established. These rules are very hardcoded. Those are not configuration options. The chatbot cannot cannot directly revoke a certificate. It If someone asks for it, it opens a ticket and notifies the IAM team. So, that is the human human like human in the loop is very mandatory. Define your escalation rules before you launch. We discovered some of our some of the things that almost went wrong in the pilot mode. So, you don't want to discover you were you want to you want to discover in your pilot state like what are the escalation strategies at what point the human need to the human is present in this chatbot interface. So, so so uh so those kinds of entities is very important. And some of the open questions which which I would like to ask is Let me Let me close this Let me close this sub Let me close this section which we haven't solved yet because I think this is where peer conversation is most most most valuable. Accuracy at scale, like we don't have a we don't have a systematic team to detect wrong answers because they cause certificate outages. We need better automated red teaming of the policy Q A pipeline. We need so rather than a human evaluation, we need to have a red team strategy like if something goes beyond our control, so a team need to be always present like how we have ongoing hours at team hours. So we need to have a red team hours. So where some of the issues can be resolved, but it's not mandatory. It's very in the initial stages. And next is auditability and accountability. If an administrator follows all the chatbot guidance that that usually is sometimes that turns out to be wrong. Who bears the responsibility? Whether the A chatbot or the or your team. So that need to be we need to make sure that the next one is the model drift. In common policy documents update regularly. So because whenever as we are in the migration phase or new development stage, so there will be there will be the the documents are updated very frequently. How do we maintain a retrieval corpus? So that is always correct and alerts when a guidance has been changed. So that is very important. And in common corpus sharing, so our grounding corpus is only the UMD specific but institution usually has a different policies and different guidelines. So we need to so every institution need to have a proper corpus of their policy documents. And some of the takeaways is six six things to check back at your institution is start with the highest volume. Lowest risk questions like renewal start when when you when you are planning to deploy these kinds of chatbots like so what are the critical questions an administrator is trying to ask with the IAM team or certificate authority team that we need to make sure. And ground your large language models to your CA certificate authority actual documentation strictly. Don't mix Don't mix it up with other kinds of policy documentation such as in common or DJ's policy documentation or any kind of federated policy documents because those are two different documents. And you need to define escalation policies before you are launched. And the next one is the building the confidence of a design framework from the day one. And which is like in this kind of framework you need to make sure what is uncertainty threshold, the visible signals, and audit trails cannot be retrofitted. They must be designed in. Set user expectations persistently in the interface. Users will always trust with the answers, so you need to make sure they need to maintain the trust using the confidence. Talk to your peers. We should not each rebuild the same in common like in common policy. Q&A from the from scratch. So, this analysis it should be should be properly used by the community to get which we can gain many things by sharing these kinds of frameworks like like overall. So, some of the So, some of the ongoing ongoing things which we are which we are working on right now is mainly in Q4, we are connecting the chatbot through a Acme Acme protocol endpoint, moving from guidance to action for automated renewals through Acme through the Acme interface as well. And in 2027 we are hoping to work with Internet 2 commodity partners on a shared shared in common policy corpus like what kind of like documents we have documents we have from the higher level so so that our corpus can be updated frequently based upon the based upon their documentation and ongoing we are trying to form a red team program with our IAM team as advisers actually finding the very edge case scenarios. So these are the main so ongoing things which we are working on. So thank you for watching and I hope this was very and I hope this was very useful whether you are deploying similar systems managing in common in common interface or thinking about confidence aware design applies to your own AI deployments. I would like to connect with anyone working with similar problems you can reach me at cgunput@umd.edu if you're exploring conversational AI over operation system start a conversation. The community has a lot of things to learn and I think this framework would be very useful like when you start building on building building on this CA authority management systems. Thank you.