Rethinking Testing for MCP Servers | apidays India 2026

Hi everybody. How's everyone doing? Uh I hope you're having a good conference. My name is Puja Jagani. I lead opensource and developer advocacy at browser stack and I'm also one of the maintainers of the opensource repository selenium. I'm not sure how many of youall have heard or used it. It's for browser automation. So few months ago I started building an MCP server for my team. Uh it was a documentation assistant. Basically I wanted to ask questions to it about Brastack products so it could go and get the answers from our actual docs. Um cuz obviously the real ones are not always updated with what we publish. That was the goal. I've always written tests for every code right in Selenium without tests no pull request gets merged. So for the MCP server when I was writing tests something felt off. It felt like what I knew was insufficient. It stopped making sense. So I did what every developer does. Googled it. I'm a little old school that way because AI didn't have the right answers. So this talk is exactly about what I found in the journey and how one should think about testing MCP servers. Uh thank you for joining in today and I'm not sure everyone was here for this topic but still appreciate you guys hanging around. So now um this is what this took me a while to appreciate right so think about every software that you've written a REST API a CLI tool or any other API you knew the client that was calling your code right it was written by you someone on your sister team or a third party but still a human being a code that was calling your code nothing was different right it was calling it in an expected ed manner with correct inputs. So if something went wrong, you could go trace back what has happened, why did the call fail. Now MCP changes that in an interesting way. Your client is now an LLM like we all know a large language model. So what does that mean? Right? It reads the descriptions you've written in your MCP server, reasons about it, and then decides what to do. So now the description becomes an interface right between your server and the intelligence using it. It's a very powerful model and it genuinely tries to help but all it has is the description to make decisions on right and your server will then do the rest of the things. So but the gap is we're not exactly testing it that way. Right? So that's what this talk is about. Before I go on uh rambling about how you should do, what you should think like let me back up a little for people who don't know what MCP is considering this is day two day three people would know what it is but still uh MCP is model context protocol it's a protocol just like rest is a standard or a protocol for writing a server so is MCP structurally nothing different but here's what you can do with it you can pick any functionality of your choice and make it available in a way that an AI assistant can understand it. Right? So for example, you don't need to go learn a new software. You can ask an AI assistant in plain English language, go do this, fetch this for me, add to this database. That's the functionality MCP gives you. Right? And uh this is solely based on how it description. So if you tell me puja how does it know right your server tells it you every function you expose is paired with a plain English language description which describes what the function does pretty simple llm will read it go figure out what to do it's not rocket science under the hood right so you might think oh my god how does this happen no underneath the hood it's your old school APIs that you already know so for example let's take a GitHub MCP server Right? It will have different tools like create a pull request or add a branch, get me contents of a branch or like some codebase whatever underneath it. It's calling your GitHub APIs, REST API that everyone knows today. So it's nothing new. So now the possibilities are endless. You can pick any software that has an API, wrap it up and hand it over to the assistant so you can work with it in a plain language. Go Google it or look at the official MCP server repository on GitHub. There are so many implementations. Almost no software there isn't a software that doesn't have a complimentary MCP server official or community developed. It's it's getting there. So before I dig in again um and this is I'm trying to prove a point. How many of y'all have called a rest API? Everyone pretty much right. Um, how many of youall have used curl for it? How many of you all used Postman for it? They're there. You You say yes, you get a sticker. How many of you all have used a browser for it, right? Every time you've done it, you got in for the same request, same response. No surprise. So, I mentioned three different clients. You use any, you get the same response, right? So, just I'll do a quick, not exactly a demo. This is a little boring, but I'm just still going to show you anyways. Um, sorry it's taking Yeah. And there you go. Let me zoom it. So, it's a simple curl command. It's calling my GitHub. You get a simple structure, a JSON structure. We knew this is going to happen. You do the same thing with a browser. Um, same thing will happen, right? So, if I do this in my browser, I'll just redo it. You get the same request. No surprises. And I like to think of this as a vending machine. So at the airport there's a vending machine for water bottles, ice creams, whatsoever. No matter who presses the button, they get a product. Nothing different. Okay. So let's keep that. Now I try to do this experiment with a LLM model. This will be using the MCP server that I built the browse stack documentation assistant. And I will ask it a query. But we will use two different models to do it. And let's see what exactly happens. So the first one is using a haiku model and I'm asking what's the timeout limit for a selenium session on browser stack. Right? So it's thinking it's doing the thing and it just searched the web. It didn't use my tool at all. It didn't even ask me it wants to use my MCP server. It just searched the web. It skipped my MCP tool altogether. Right? So if you look at it is looking at brass docs. It picked up another random website as well. So the reason I had the MCP server, I wanted the latest docs. It didn't bother doing that. It just did a web search. Now let's do the same query, different model. So different client here. We're using set for the same thing. It's doing its thing. It found my MCP server. It's asking me, can I go use it? I say, yeah, okay, go ahead. It does it does a bunch of things, right? And it's trying. Yeah, my MCP server is not perfect, but it's trying. It's trying. It's doing its own thing. I'll just push it a little forward so it's a little more interesting for the audience. Yeah, there we go. Cuz it takes a while. And this is the recording I had done as it is. let me and it doesn't get the result it tries some more it goes to the final tool escalate that hey I don't have the answer um ask browser stack support right so what is my point here my point is with rest you never had to worry your curl will not go like hey I don't want to call this endpoint screw it one model didn't touch my tool at all my MC CP server at all and the other model tried its best to do what was what it was told. So this is the shift. It's a completely different client. I also tried to fix it. So I went back to my MCP server and changed the description. I said don't use general knowledge. Don't do web search. This is your one-stop shop. A little better terms. Rebuilt it. I got the same result. So that didn't work either. So my point is this is the completely different client that I'm trying to work with. So this is how so there has to be shift in how you think for things right. So all of us have experienced this. It could be a Jira ticket, a PR description, an onboarding dock. You could have written the best one possible but someone somewhere did not read it clearly or did not understand what the assignment was. MCP is having the same day but now it's instead of like a colleague or us glancing through those dogs it's actually impacting a business logic what you want right so the way that it needs to be tested in the same way so you'll tell me what what can we do so I'm not telling you go test LLM that is a hard problem there's no way I that can be solved in a 25minut talk at all I'm trying is try to test the server you control it's complete it's deterministic It's in your hand, something you can put it on CI/CD. Who doesn't love a CI/CD pipeline test? We all want it there. You know, your tests are going to land up there eventually. So again, that's the whole point. Next slide. So what does the current landscape look like for REST API? When they were new, we eventually figured out error codes, edge cases, status codes, contract testing. We figured all of it out, right? The mental model to test for it exists. If you Google, you will get frameworks, patterns, docs. So that pattern is established. It's also taught, right? How many workshops have you attended where they'll tell you how to go about testing your rest APIs? It's it's all set for MCP. It's not there. I went searching. I got a bunch of docs. They'll point you towards MCP inspector, which is something you can do while building the server. You can't put it in your CI/CD pipelines. So, it's new. I dug a little more. Um I looked at the official MCP implementations of repos. They are writing tests but they're happy part tests. So the way they're writing test is oh does my tool work instead of how does my tool behave when an LLM is asking the question. And no one's wrong in this. That's how developers have written code or test forever. We just don't know how to go about the new thing. And that's exactly uh the roadblock I hit as well because for an LLM you need to think differently. So here I am proposing a three layer testing model. This is in addition to what you already know. So it's built on the fundamentals that are there. I'm not picking up something new. I'm not whipped up a framework overnight. I'm not doing some automagical thing. It just build on top of things that we already know but now pointed towards this new client that we have. So each one tries to answer a question. Contract testing is like um are your descriptions precise enough for your client that is an LLM to understand. Sequence testing is how does your server behave when called in an order that you did not anticipate? And error channeling tries to say that hey when something goes wrong does LLM have an signal to actually act on and work with. So let's begin with contract testing. So far any code that you have wherever there's a blob of text it's moreformational right it's a comment or a dock if something is wrong the cost is low one can go fix it no problem but with the MCP server if you have a blob of text it's actually a description it's instructional it's run every single time during runtime on every interaction so if you describe something in your tool and the behavior does not match match it's actually a functional bug. Your business logic is impacted and you need to kind of treat it that way. So while building my MCP server I'm like okay let me describe a tool that I have. So I said it's a documentation assistant. I have a tool search docs which searches through the docs and returns URLs of the relevant pages. Simple enough. This is what I wrote. It took me 10 seconds to write this description. Search docs searches browser stack docs. How many of you think this is complete enough? I am glad because it doesn't tell you what to do. It doesn't tell you what a input query looks like. It doesn't tell you where to search, where not to search, what are the prerequisites. It doesn't tell you what the response is and it doesn't tell you what to do next. Right? So when all these things are missing, what will happen? LLM is left guessing. It will guess. It is genuinely trying to help. But when the g doesn't match what you expect, you will not get the result you expect and then it'll just end up frustrate like frustrating. The reason you'll build an MCP server is to help you out. And this is will be the polar opposite of the same thing. So now I improvised it exactly like what I try to recommend. I tried to break it down into four pieces. Ex first is what it does. searches across all Brastack product documentation made it very clear exactly what it does. Second, added a scope or a constraint. You could also add a prerequisite here as to first go do this if that's a part of your tool or where to go look, where to go not look, what are the boundaries that look like. Then comes the example inputs. I think this is very important. LLMs need a pattern to follow. So actual quoted examples of what a good query looks like. So it has a pattern to follow. and next steps. So what does the response look like and then what to do ahead. So if you have this, this will give a lot more structure to your LLM. But you will tell me Puja even if we have all of this um you know how do we verify a string makes sense? Let me get to that. So how do you test it? How do you say a string is good? So this is what helped me. I try to rewire my thought. I'm not checking if it's a well-written text. It's not a literature review that I'm trying to say. I'm just trying to make sure that LLM has enough to work with. Does it have does it know what the tool does so that the tool gets called? Then does it know in that tool call what to pass and then what to do with the response. These are the three minimum assertions. If you have this, it has a guiding light to do the right thing as a part of your test. Right? So how would you do this as a form of structural markers I can show you uh next but again compliance doesn't mean it's complete. So I would highly recommend like anything that is text that you review it has to be peer reviewed with a I to make sure that LLM is able to understand what's there right so how do you then go do this this is how I have tried to build all of my thoughts as a pseudo code so if you have any MCP server in any language you can go pick this and try to use this so for every tool check the description is meaningful right so when I say meaningful one is the review part Second is check it's long enough so someone's not snubbed it to make sure there's all detail. Then check if there's actual quoted example. You could set any format that you have in your team and do that. And third check it has a return or result. Now to think about this this is not very different. You have all tested for a rest API. When a JSON response comes it has certain fields. Do the same thing but now pointed towards a description that's a part of your code. Right? I would suggest take it even further. If your schema has parameters, make sure it's part of your description and vice versa. Again, why? Say your colleague goes and adds a new parameter, it's not a part of the description. LLM is now filling the blanks with a guess instead of a fact. And one more thing that I would like to recommend to everybody, say for example, you have similar sounding tools. If you have more than one tools, classic example, I'm just saying this. Add and append to a list. Add insert and append to a list. Insert could be insert at a point. Append could mean append at the end. If you describe both of them sounding very similar, LLM cannot differentiate. It will make a mistake. So, make sure your tool LLM is able to tell similar tools apart. So, please have that. And if you have your most schemas and descriptions, again, take this thought. That's why. So now the contract that you have with an LLM becomes a lot more meaningfully complete. Right? So I would set these things as like a minimum bar or three assertions that are must have to take it further. Next sequence testing. How many of you have written a test that has a before block? No. No one has written a test that has a before block. Right? All of you have written a test that has a before block. So every test you've written starts in the perfect state. Did you ever have to worry that your code is starting in a bad state? Did you ever have to worry it is starting in an incomplete state? No. Does LLM have a before block? Does it know what to do? Unless you don't tell the LLM, hey, go do this first. Unless you don't mention the prerequisite, it doesn't know what to do. I will show an interesting example that I bumped into. Before I play the video, I will yap a little. This is an example of an Just give me a second. Um, yeah, this is an example of a Selenium MCP server. It's built by Angie Jones. She mentioned it at Selenium conference keynote 2025. She introduced it to us and she's done wonderful things for the community. Uh, so being from the Selenus space, I was trying it out, right? So it's for automating browser actions. The base step for it, you need to start a browser. That's the prerequisite for anything you do. You have to start a browser. Only then can you interact with it. I deliberately tried to poke it. So I tried to without starting a browser. I just say go navigate to this URL like open just go to this URL. Go to Google for me. Don't start the browser to this MCP server. Okay. This is exactly the code for that. And that's what that's what's happening. Uh how many of youall think this will error out right away? Right. Okay. Let's wait. Uh there's an awkward part of the call, but okay. Let me put a little ahead so it's not as awkward. Yep. I think cuz it takes around 60 seconds for it to like time out. And that's what my recording has a little. Yeah. So now it has timed out. The client was hanging. It was silent. It has no idea why this happened. The request simply time out. So server did not say, "Hey, something is wrong. my MCP server or the Selenium MCP server didn't say, "Hey, this is wrong." It just had no idea this needs to be done because that's how we write tests. All the tests that were there in the entire repo, again, not taking away anything from the repo developer. She's done amazing things for the community. But everything had a before block. It was setting a right state. It did not check for a state that broke, right? So when you tell the if you for everything that you tell LLM if you didn't tell it to start a browser it might just skip this right so that is why it's important to do a sequence testing now you tell me how do you want to do it again a pseudo code template for the same identify a sequence even if your server is stateless there will be some ordering that is important right so for every ordering deliberately break it test it out of order and then see what happens does Does it hang silently? Does it give you a clean fail? What is there? Basically, does it give it something meaningful for the LLM to act on? And because a normal pattern will not catch this, you just saw it in the thing. If I do not give it a prerequisite, there was a silent hang. So, you have to deliberately break it. Why wouldn't you rather break it when you're developing this versus when it goes in production and someone gets stuck, right? That is the pattern we're trying to see. This is a new pattern that's come with MCP that one should be testing for. This is the last piece uh error channel testing. And I found this very interesting coming from the REST API world. I was used to only one error channel. If something goes wrong in REST API, you have one error code. It tells you okay, it worked or it didn't work. Simple muscle memory. All of us know it by now. With MCP, there are two error channels, right? And and let me talk about that. And they must be tested equally of course. So what do you expect? Like I said the one first diagram is for a request you would get either a success or an error which is okay. MCP has two of those. So one is at a protocol level. So there are JSON RCP errors saying something didn't work well at protocol. Malformed request unauthorized server crashed. We know this. All of us know this. Nothing wrong. Second is the interesting one. It will say the response is successful. That means something went well at the protocol level. But the business logic failed. The input was invalid. I got an empty page. Uh nothing worked out and it's hidden in a successful payload under a flag called is error. It's buried inside where is error is true. So you must check as to what error message you get there. There's a third state that I would kind of just like to give it a food for thought for you. Everyone saw my example for the set, right? What did it do? It didn't give you an answer. It went to the last stage and said, "Contact browser stack support." I didn't get the answer that you desired me for. Right. Was there an error? No. Output was complete. Was it accurate or right? But it was complete. There was no error. So technically is error is false. There was no error. But the response it got was very thin. That's why it kept retrying. Right? So my server can't test the judgment an LLM makes. But what it can do is it provides enough information to the LLM to do its best job to see what to do next. Let me show you an example before I go. So I have two versions of a server. My server. Okay. Again look for I give you a product name. Go look for the valid documentation links and tell me the same thing. Purposefully I have given it a gibberish product. something that doesn't exist. So this is the one. So this is the broken one. You can kind of call it broken. Look what it says. Is error is false. Okay, there is no error. You didn't find a gibberish product. And look at what it tells me the response. Uh this was the query. The gibberish word was the query. Empty results, right? So what do you do with this? You really can't do much. Now let me go through a second one. Yeah, this was sorry I'm I have a little short of time. This was the one that I fixed. It told me is error is true and it told me what to do because the is error was true. No documentation found for your query. Go try this this this. Did you mean this? So now LLM has something to act with. Why is it important? It's important because LLM is not it's basically whatever you get it's not your colleague reading it. It's not a log. It's making business decisions. How do you do it? my pseudo code template. Um, identify failures, trigger all of them, check you're actually erroring out or not, like check for your is false, check for your is true, and then check the error message. Treat error message to my first time. The contract testing that I try to show you, right? Because again, LLM needs to know what to do next. Should it retry, not do anything, give up, escalate? What should it do? Call another tool. your errors message should be saying that. Okay, these were just my tests failing tests and then how I filled the gap because if time constraints I'm just going to skip forward. It was just showing my failed test suite and then the green because green feels good. Um that's about it and just last just last closing slide. So testing LLM is hard. testing that different models will produce the same result at every runtime is a problem industry is trying to solve. So what you can test is what is in your control deterministic in your hands, right? So if you have um you know precise descriptions, guide tool choices, right? Clean failures guide recovery and you know validated sequences prevent silent hangs. So test what you can control um is what I recommend. Uh yeah, that's my LinkedIn. Thank you so much folks for for being there. Yeah, I do. We have time for questions. I don't know how this works. Okay, y'all can meet me in the hallway or later or whatever if you have questions. Thanks folks. Sorry, I'm just going to take a picture to show my manager how this went cuz they always ask me and I don't have an answer. So I'm just going to take a picture of the audience. Can I take a selfie with you? Is that okay? Yeah. >> I'm trying my best to cover everyone in the room. >> Thank Thanks, folks. You're living great.