Architecting Enterprise Agents Ecosystem: Standardizing APIs with MCP and Orch… | apidays India 2026

Hey, I am uh Kiran uh and uh this presentation is co-presented along with Hanitary. Uh I'm the group manager in Inude. I lead AI foundation and uh we are from Inude. So Inude basically enables uh you know tax filing and also enables uh small businesses uh in US. I think you guys have recently have filed taxes. So something similar we do across the world not in not in India right. Um today what we will be presenting is uh is a very interesting uh you know uh uh journey uh specifically with respect to uh AI right u as we are enabling uh AI agents uh how we are enabling the enabling this at an enterprise level right so all of you might have already built one or two agents you have tried it out and all that right but but the real challenge comes in how do you scale it at enterprise level how do you make sure that uh each of these agents can be communicated connecting to each other right and how it can be uh how it can connect to your existing uh infrastructure right that's one thing we will talk about we'll also talk about uh you know very interesting space uh with respect to uh how the AI quality can be governed at a very high level this is the agenda uh I I don't go very in detail about it but uh what are the key architecture decisions we made uh to uh basically achieve this right that's one thing we'll talk about we'll also talk about when you have an enterprise you have multiple multiple BUS how these multiple BUs can you know build agent at scale how they can communicate to each other right that's where the lot of challenges are coming in and many many of the enterprises are you know still figuring out figuring it out right and how these agents can be u you know published right from each of these BUS that's something we'll talk about we'll also talk about like the future looking state with respect to uh A2A uh protocol how we are exploring A2A protocol and all that and u finally the more most interesting part is uh the agent quality right uh because uh how many of you have heard of eval so right uh so it's a very interesting space uh nobody has figured out uh with respect to uh uh new agents right so that's something we'll go in deep uh about u we'll quickly we'll quickly start with the three key decisions we made right u so basically we had u multiple uh you know uh APIs multiple products which is there from multiple decade multiple decades and all of that right so how do you uh you know basically enable these APIs to agent ready right so every every team had the same challenge like you know we have the APIs how do you make it agentified right so we kind of took a middle layer where you have a middle thin layer which basically enables uh you know uh connecting this uh uh connecting to the existing services so existing services are not touched basically you build a thin layer uh build and we deploy that so I Honey is going to talk about how this is getting deployed into production. Right? That's one thing we'll talk about. uh basically other thing is like as I said right ship using plugins so plugins is heavily used uh multiple B all of them are shipping their uh you know agents using sh plugins and u also we talk about initial problems what we faced with respect to how we are building these uh how we were building these MCPS locally and how we kind of deployed it right with that I'll give for the internal deep dive architecture honey is going to present on Uh thanks a lot Kiran for a warm introduction about me and the team. So yeah uh like at enterprise some time back we started uh like as Karen mentioned we have any number of services serving millions of customers externally as well as we have internal customers as well like data engineers and data scientists for whom we used to build tools. So we thought the services are running for years and there is a paradigm shift. Everyone wants to move to agents. People now want to just write a prompt and the services needs to do your work. So uh again it's a fast-paced environment. We needed to do things rapidly. So we thought like how can we do it? uh first solution was uh changing all the existing services to agent effect services but again with a limited team having lot of day-to-day deliverables production important issues so it it was not a viable solution for us um so we thought of a different approach the approach was uh at that time MCB was getting popular and MCP is a way for users for internal users as well as for the external users to connect quickly to the uh MCP enabled services. So we thought of a way we we thought that we'll create a middleware layer that layer will be built on MCP and then later on it will be again becoming a full-fledged agent and that that will give you a route to our existing services. So from your UI interaction we will take you to the level where it will be prompts doing the task for you. So like giving your alternate route to the UI. So how we did it? Um we found a way MCP but then we started with local MCP. So local MCP has its problem like as I told you enterprise level applications running on service mesh and running on Kubernetes and on a cloud. So it has availability to number of things. Now local MCP sits on your machine and does the work for you. it it it doesn't solve your problem. Okay. So we figured out a way that local is not a working solution for us. We thought to go for remote and when we went for remote things changed. Let me tell you how. So if you can see [snorts] uh this is how on a high level our enterprise network infrastructure layer looks like. If you can see this in middle it's a single pod running on Kubernetes. Correct. Again Kubernetes is an orchestration platform. So based on load traffic the single port can scale to tens of ports based on the number of request. So what we thought here at the bottom if you can see spring boot APIs or there are again other nodejs APIs that are internal services which are doing few of the task for our products. So in between we brought this layer of fast MCP. So now I will tell you how the journey looks like. Um like we started with fast API. Fast API is again a lightweight Python framework and it runs on top of uvicon lightning server. This two together goes onto Kubernetes and inside that what we installed was Langraph Fast MCP and we made it a complete agent ecosystem. Now as you say see the title of the slide is one for one pot two phases build once exposed twice. So what it is all about the thing is this fast API micros service that we created the middleware layer it has langraph installed inside it and it also has fast MCP. So with the help of langraph langraph is again opensource toolkit it helps you to build multi- aent ecosystem. Okay, it helps you to build agent ecosystem which can be stateful which can have multiple agents involved and it can also enable your agent to talk to other agents using protocol like A2A, ACP and so on. So again this is the step where we move little ahead. We started with keeping things simple. We went with fast MCP. So fast MCP is again a lightweight framework. You can put it on top of fast API and what it helps it helps you to make your service MCP enabled. You will now write the tool with just a decorator that will make your tools as MCP enabled tools. And what's the advantage? Once you have tools inside a server which are MCP enabled, they can be accessed by your ID clients. Like if you can see these days we are all using AI powered ids. So AI powered ids follow MCP protocol and using MCP protocol they connect to MCP server. So once we had this middleware layer ready okay we were able to roll out to our internal customers as well as few of the external customers using MCP servers. Um and then they were able to connect to our ecosystem. And one thing I told at the beginning only an enterprise like us has system stable u from running from many years. So we already have certain things in place. If you can see at the bottom part we have like inherited lot of things from mesh like if you can see Kafka is our message broker that we are using for each of our services. We have vault to store the secret information or SDKs enabled service discovery observability CI/CD. So all of these things were existing in the picture just with the with the introduction of a lightweight middleware service created using fast API powered with fast MCP and langraph. We have agentified our ecosystem. This middleware now talks to the customer. customer write the prompt fast MCP server enabled MCP servers listen to those prompts and it does the work for you by routing your traffic to our existing services which are already there for years. So again choosing this approach Kiran and me we were able to get into agentic pave path I will tell within few weeks and that has been a key key milestone for an enterprise like us where like shipping any new technology shifting to any paradigm change it takes it takes huge time. So this approach actually helped us but again uh if you can see agent ecosystem needs to have some guarders in place. So we were cautious. We didn't expose everything. We expose only certain things that have non-destructivity in nature like we enabled lot of get endpoints at beginning more of the creation and deletion aspects for certain internal uh tools and services we didn't expose at the beginning and we thought like over time we'll see and and then we'll take those endpoints again to the production with proper guardrails in case. So now I will try to give a brief overview that how some of the internals of the technology works. So langraph if you can see langraph as I as I told is an it helps you to power an agentic ecosystem having multiple agents having state. So if you can see langraph has nodes and edges. So langraph isn't like kubernetes is an orchestration agent orchestration engine for deploying your containerized application to multiple servers. Langraph on the other hand is an orchestration tool that decides internally what your code is running how it will uh do certain task. So with the respect of LLM so LLM having access to multiple agents multiple tool langraph helps you to determine what to do when something you get a success response and what to do when you get a failure response for certain thing and I will give you a detailed example which will help you to gauge the power of how langraph does an orchestration of multiple agent in the coming slide but first understand the fundamentals Like it's a simple flow like we start then there is an LLM node comes into picture. LLM node has to do certain works. It calls tools. It has a description of tool available at its level. So it it's it's able to understand that which tool is for which purpose. Like if we have tens of internal services inside our company each service has its own responsibility. LLM will be able to figure out because the tools have provided a proper description. Then LLM does uh like LLM node calls a tool tool does certain action like calling or internal microservices getting the response then the assistant node having LLM will again determine this is the output whatever is needing or I need to call certain other tools also so based on that if we again have to hit another internal service they'll again use some of that tool we'll keep it will keep on iterating because it has a brain it has LLM And then once the result comes which can satisfy the end user it terminates a loop and then we send a end response to the user. Again as I told governance we maintained that helped us to release these things without critical bottlenecks at the beginning was isolated data store per MCP. We didn't allow MCP to touch our Intel microservices database. We had isolated data store no delete endpoints at the beginning and initially we started with a devance device bound identity. Now I will give you a high level overview of how MCP to an more ecosystem friendly uh bigger scale problem can be solved like MCPS can help you in solving problem for a specific domain by because MCP will give you capability to query the tools the databases the internal services to come to a certain conclusion but if you have a suppose full-fledged workflow like u one of the workflow that I think Everyone is aware aware about like suppose we have a SAS subscription okay that needs to be renewed every month. So that payment renewal workflow okay now that payment renewal workflow can involve multiple agent if you can see payment agent reporting agent and notification agent and at an enterprise level this might be obtain like this might be owned by different teams. So in this kind of system one has to shift from MCP to more agent to aagent communication uh protocol correct. So A2A was one of the uh protocol that we have started with but again there are certain other protocols like ACP and all which also one can go ahead with there are a lot of mesh protocols at this stage. So we started with A2A protocol and again it's a journey that we are still traveling. So as I told you just imagine an orchestration agent again built of Langraph and you have a problem statement like I have to uh see the payment subscription renewal workflow for my customers. Some customers payment would be charged at the end end of the month some customers payment would not be charged because of insufficient balance or some other reasons. So keeping that in mind like our in the agent will start it will figure out uh like agents related to this problem statement from a vector DV. Then once those agents are fetched out the complete workflow will be set up and that transaction related workflow will set in posgress. Again you need to learn the importance. Vector DB is for agent discovery for semantic search but posgress SQL we are using more for transactional like did the complete payment workflow notification happen successfully or not. So storing status will be part of posgress but agent related semantic search capability will go to a vector DV once that is in place using Kafka as a message broker will start communicating with different agent payments agent reporting agent notification agent okay payment agent will fetch user payment status is it successful or not at the end of the month if it is then we'll be going to a reporting agent reporting agent will again adjust internal documentation, internal account balances and also it will update a tax related information. Then finally once those things are successful, it will notify notification agent which will ultimately give a billing statement to end customer that the payment has been successful. Similar use case for a payment failure scenario. >> [snorts] >> So uh this is how the end to end a2A call and architecture look like. So now I will again hand over like to Kiran. So Kiran will be talking more about agent quality at an enterprise level. >> Hey thanks thanks Anie. Uh thank you Anie. I think that was very insightful. Um uh I think the next part which we want to really talk about is that once we identified and like you know as we started getting lot of agents into the system right uh one was like the discovery right how do you identify the right discovery that's what something you know Tani was talking about but even more problematic uh area was that um how do you you know uh gauge the quality of the agent right like previously this was your system basically you have a defined input You have a you have your application with with you know with the code base as well as the database and all of that and you have the output right where you it's all the whole flow is deterministic basically always the input is always going to be the for a given input the output is always the same. So you have unit testing, you have integration testing and all of that right which you can uh with which you can identify how your system is uh behaving. But uh with the new world with the where we have the LLMs coming in right with the agents right uh you have prompt skills and uh you know you have LLM right so all of this is not based on code right basically there is you have skill which which can degrade you have you know LLM which can which is nondeterministic for a given input you're not always assured that your output will always be the same right so it becomes uh your testing becomes non deterministic basically right you cannot say that hey assert saying that if something is not uh matching this right uh you have multiple permission combinations which you have to gauge for that is where it becomes uh extremely complex for you know to ensure that the quality of the you know how do you uh test the uh agent quality right so I think that's where uh you know we probably want to you know uh started our journey right uh specifically as I said uh there's no defined the no fixed output expected right which uh and basically failures are not always guaranteed it's going to be the failures can be uh you know you can always sometimes it might pass sometimes it might fail right and uh more importantly the silent regressions right how if your system is degrading you won't even come to know because uh uh if the user is asking for a specific uh uh input right specific uh ask but your system is degrading and you're not even getting to know right so those are like the risks which we have and traditional QA cannot uh testing cannot work right. Uh so uh the so basically how you can achieve this is having some of these uh uh you know measures right when you uh when you want to measure basically you will have to measure for correctness uh how you uh you know basically given a input uh is your agent is solving the uh problem or not right if I'm asking for a credit score eventually the user has to get the credit score right so that's what the point is and uh is your is your system is hallucinating right uh you know Is it is it going out of the context and uh tool accuracy? You want specific tools to be called in the correct order. If that is not happening, how are you you know gauging that? How are you measuring that? Right? And safety and trust and cost and consistency, right? Your system has to be more consistent. So those are some of the parameters which you have to measure, right? So this is like a very high level architecture of how uh you know we are do we are trying to do the evaluation right of the agents. So like any other systems right you basically have to run your agent uh uh you have to uh capture the observability metrics right and then you have to see how your observability outcome right is is behaving right with the with the LLM as a judge right and then you capture the information and then you uh you know look at those metrics uh how that look at those metrics in terms of dashboards and alerting right so to be precise basically when you have uh when you run an agent uh you have prompts, you have LLMs, you have tool calls, you have latency all of those are the metric all of those are the parameters which you capture and then you publish it to some observity like you know lang langfuse right uh if you have heard of langfuse basically you kind of get all the traces uh from the agent and then capture it in the observity metrics like observity store like langfuse and once you have it you kind of use judges LLM judges is another uh you know interesting area right where LLMs are kind of the the reason why you want to have a judge right is that uh you cannot this system has to be continuously monitored right you no humans can do it so initially you have to come up with some prompt based on your test data like a golden a golden uh data set with which you come up with a prompt and that prompt will measure those metrices which I talked about in the uh which I talked about earlier right and that becomes your measuring metrics as to how your system is performing right and that becomes a loop every time you define a metrics you come up with outcome and then based on that you tune your basic you you tune your prompts and lms and tool calls right um so uh that's what we do and uh the human in the loop is a very important point here because u uh what if your LLM as a judge itself is wrong right uh it is not end of the day it is a prompt which which is which is which is looking for specific measurement right So when LLM outcome out output comes in basically you the human in the loop also can come in and annotate saying that this information what the judge evaluated is incorrect right so there are various techniques where using the human in the loop annotations you can tune your judge or you can also tune your prompts right so so that is one thing you know we are also evaluating uh again as I said there are like two uh techniques within this one is offline and online. So offline is basically where you say uh where you basically calculate uh uh do everything before you go to production right uh and uh and this is more of a pre-est uh production pre you know uh not in the production right but but given these services are online so you all have to monitor them continuously how how your online system is behaving that is something uh is is called online so as as you see here right so offline evaluation is basically your you curate a golden data set curate a golden set is basically you have your production data. You take a sample out of it and then you say these are my various samples which is more or less accurate representation of my production sample. Right? So that's what a curated data is against which you kind of evaluate uh if my uh agent is ready to be shipped into production. Right? But uh in case of online basically you have to make sure that the system is uh behaving correct. Right? uh one important point here is both of them are actually connected online evalation offline evalation is both are kind of connected that's what we are talking about the fly uh flywheel loop right u once we observe curate and evaluate offline you ship it and then you monitor right when you monitor using online you come up with various data sets right which actually becomes again a curated data set for your system so that is what we call it as curated set but end of the day both are kind of connected. Uh but but you know both of them are like a important aspect of uh uh offline and online evaluation. Um just a quick recap uh is that basically we talked about uh how do we how did we identify our services and uh how did we host it, how did we distribute it and how are we measuring the quality of the performance of the agent. Right? So yeah, I think with that uh thank you so much. Uh uh we are uh uh if you have any questions, I think we'll be happy to take it. [applause]