The New Brain Behind Copilot in Visual Studio

My name is Dalia. I am one of the PMs on the co-pilot in Visual Studio team. And I'm here to talk to you about the new brain behind Copilot and Visual Studio. I I wrote the title vague enough that it's like intriguing, but um I want to be very specific about what I going to talk about today. And um I'll talk to you about the technology that we use under the covers. Um, I'm going to talk to you about some of the lessons we learned as we're using Copilot in our own team. The Visual Studio team uses Visual Studio to build Visual Studio. So, I'll touch on that. And, uh, overall, I want to be completely real with you about the lessons we're learning. So, I think that's kind of what's interesting during this time um, is sharing lessons because we're all learning. We all don't really know what these models are fully capable of. And the best way we can all learn together is to share these lessons. Um, so I'm not gonna try to uh pitify the stuff. I'm gonna run some live demos and if it fails, then you watch me fail and you watch me recover as well because that's what happens in your day-to-day. Um, all right. So I'm going to start with the why because I always like to start with the why. So why why this talk? Um so we talked to a lot of developers and um they came in and said hey uh Dalia I use Opus on uh Visual Studio Code I use it with Visual Studio. I use it with the CLI and I'm getting different results. How come I'm using the same model? Why am I getting different results across all the different tools? And um sometimes they're getting worse results in Visual Studio and to us that's not acceptable. We want the best output for folks that are using Visual Studio. So in order to understand um why you're getting different results in different uh idees, I need to explain a concept called the harness. Anyone heard of term harness? Okay, not a lot. So this is a a a concept that not a lot of people talk about because it's kind of really under the covers, but I do want to explain it here. So when you're talking about a model and an LLM, what an LLM does is it takes in a lot of uh content and it generates content. It doesn't really do anything. In order for an LLM to do something, you have to tell it what it can and can't do. And that's what the harness is. It's the runtime around the model that it allows it to do what it needs to do. So for example, if you're running Opus or running GPT and it runs by itself, it doesn't really know is it running on a Windows machine, is it running in Visual Studio, is it running in VS Code and that's where we feed that information to it. So we tell it, hey, you can actually create files, you can search for files, you can you can find the solution here, you can do this and that. So all that is in a harness. It also tells it what it can't and can't remember. So in between sessions, you're having one session with a model. How would it know what other sessions you had, what kind of memory you have? And that also lives in that harness. Um what it sees, what files are on the file system, what things can it do and can't it do? It also controls it. It govern it governs it, which is a really big deal in enterprise applications. That's why we think really hard about what kind of things do we allow the model to do and not do because we don't want it to, you know, delete your hard drive, right? Um, and we also tell it when it's done. So, LLMs, uh, sometimes are lazy. Sometimes midway they're going to stop doing the task that they're supposed to do, and we have to tell it, "No, keep going. Keep going." We actually have, uh, prompts in there that says, "No, keep going until the task is done. Don't just stop. And sometimes you notice if you use agent mode that you have to tell it, hey, like I told you to do five things and you only did three. What's what's up with that? So the harness will tell it keep going, keep going, keep going. But so that's all what's in the harness. And why am I explaining this? Uh the reason we have a new brain is because we're switching harnesses. So here we have it doesn't look the the greatest here but you have here a box. So what happens is when you sent your prompt the harness is going to kick in and we used to have a customuilt harness in Visual Studio when you use Copilot in there. And this one was handbuilt up by our team. So the harness would tell the model what to do and then that's when you get your response. Now, what we're doing is we're replacing our old handbuilt uh harness with a new one coming from the Copilot SDK. And what is the Copilot SDK? This is um the same uh engine that powers Copilot CLI. Anyone use Copilot CLI here? Awesome. That's a good group there. Um so by doing this that means that if you switch from the CLI you go to Visual Studio to VS Code you're going to get the consistent experience across all of them and then you'll get the answer here. So uh ultimately what you should see in Visual Studio is now that when you ask questions you're going to get better answers. Um, so I want to pause here and just talk about a little bit of a like an engineering lesson. And I was debating, do I want to talk about this? Do you not? I want to talk about this because I think we don't talk about the human element of this enough. So I'm going to uh just do a sidetrack if y'all don't mind. So as engineers, we take a lot of pride in the code we built, right? We spend months perfecting the code, making it perfect. And this slide is actually pretty hard on the team to see. Why? Because they spend months perfecting this harness and all of a sudden we're coming in and telling them, "No, actually I want you to throw away that code and replace it with new code." And the reason I want to say this is because we everyone here as AI is becoming more and more powerful, you kind of are continuously having this existential crisis of like but what's what do I build? What do what does it like at what point do I actually create something custom and at what point do I let the model do it? And in our case, our question was always what's going to serve the customer most, not how can I build the perfect software? How can I build the perfect code? How can I do the perfect things and claim that everyone else don't doesn't know what they're doing, right? Because I'm going to build the best software. And this was a lesson. This is we have like a concept of growth mindset in Microsoft and I think everyone in the software engineering industry is really getting challenged and they have to have that growth mindset of thinking if someone else is doing it better I need to ask myself is it going to be good for my customer to bring in that better library that better framework that's going to do something better and then make that decision. So, the reason I say this is because you're probably asking yourself the same questions in your teams. Maybe a sister team is building something that's better and you're like, "No, I'm going to I'm I'm going to build the best thing ever." What's what is your time better spent? Building something custom or bringing in something that already knows what it's doing and then differentiating. So, that's what we chose and where we're choosing to differentiate is all the things that make Visual Studio amazing. the debugger experience, that's not something you're going to get from the model. Like all that deep debugger experience that you have in Visual Studio, all the things that we built with profiler and all the memory stuff, all the things that make Visual Studio what it is, the version control experience, the UI experience, those are the things people come to Visual Studio for. So that's what that's how we're saving time on that and going into um copilot. Thank you for uh letting me go on a side tangent. I think the human aspect is really important to talk about here. So that's why I wanted to touch on it. All right, back to the content. So I was talking to a gentle me gentleman here and um it's interesting to see the spectrum of developers I talked to, right? On one hand you have the developer that is like I like to write everything manual. Don't touch my code. Don't let agents do anything for me. I want to write everything by hand. And I think tools should work for that. But on the other hand, we're seeing more and more of the industry delegating. It's we're realizing that some of these agents are actually good at certain things. So why not delegate? And what we've noticed is over like maybe two years ago when I came to VS Live, I heard more of that first thing of like I don't want it touching it because like I don't trust it. as as software engineers realize that okay these agents are good at certain things we're seeing more and more people move towards delegation in certain cases right not everything but it we are moving more and more into delegation and that's why we when we talked about uh you know replacing the harness the first thing we started out with is agent mode anyone here use agent mode in copilot oh wow that's a big chunk okay awesome So that's what I want to start out with is I want to talk to you about the new Asian preview. So if you have insiders 18.8 or the latest the latest Visual Studio, anyone here the latest 18.8. Okay, that's a good chunk. You'll see a new option called Asian preview and that will let you use the new Copilot SDK. um under the covers, you'll get better responses. And I want to show you um what that looks like. So, I'm I'm giving you a real example here because I'm not going to do like a you know simple app. But in this case, like I said, Visual Studio uses um Copilot to do real work. In our case, this is a real bug from our Azure DevOps uh ticket and it was complaining about uh the chat and it had this ticket and you know I was investigating it and I was like I wonder how it would do how the two agents would do uh against solving this problem or helping me reproduce this problem. So in this case, I did record this demo because it took forever, but the next demos are going to be live. But so but bear with me. So here's what I did. I have Visual Studio and H two halves. On the left side, I have the uh old customuilt agent mode. And on the right side, I have the preview agent mode. And I'm gonna play here the video. So what I'm doing here is I'm gonna ask it can you please help me reproduce the problem that's in this ticket and notice that I passed it in the URL for the ticket so that the agent can go off extract the information and come back and understand the ticket a lot of times I don't waste time telling the agent what I want to happen I point it at a place where that information is already there so in this case I tell it help me repro and I pass in the ticket and in both cases the agent went off they It found a skill that the Visual Studio team actually has for investigating bugs. So, it invoked that skill and it thought over it, thought over it, thought over it. Now, notice on the left side is the old Asian mode. The old Asian mode is very verbose and it's going to give me a lot of information that I don't really need. This is a problem with LMS in general. They're very chatty. They want to tell you everything and everything and you're like, I don't have time to read all this. just give me the information that I need. So, we actually have something in the system prompt uh in the Copath SDK that says, "Please be concise." It's as simple as that. You tell it, "Hey, please be concise." So, on the right, you'll notice that it's a lot more concise. You'll get a response that gets to the point a lot faster instead of giving you a novel just to get an answer on a question. But what's super cool, when I saw this, I was like, "Oh my god, that's amazing. I didn't even think it would do that." on the right side with agent preview. When I tried this same prompt, it realized that the agent mode is running within the chat in Visual Studio. So, it was like, oh, I can repro it right here. So, it reproed the problem in Visual Studio. And in this case, the ticket is when we're asking a question in the chat and the answer is very long, we're actually truncating the answer instead of wrapping it. So that's exactly what someone was complaining about and that's exactly what it repro did on the right and this is why we were like okay we have something great here this is why we integrated this uh cop SDK here that reasoning that thinking doing the right thing right away instead of having to have a million conversation with your agent this is how we saw that improvement and how we're uh why we're replacing our harness now notice the reason why I recorded this is because the agent on the left kept going and kept going, then kept going some more and some more and some more. It actually took um almost five minutes to return a response. And notice all the chattiness. And on the right side, it took under a minute. So, the new agent is faster. You're going to get answers faster. The new agent is more concise. It's less chatty, but more importantly, it'll give you what you need. on the left side, the agent actually give me steps that I need to follow to repro. It didn't actually repro it in the chat like I wanted it to. So, better quality responses, faster responses, and less chatty. Um, so that's what you should expect with from the new agent preview. All right. Now, uh, when you're evaluating these LLMs and agents, you don't want to just take one anecdote and say, "Okay, it looks good. Let's just ship it, right?" Uh, we actually have evals. Anyone here of like Sweetbench? So, SWEBench is how different models evaluate how well a model does like you know how uh anthropic uh, you know, evaluates different models, how open AI evaluates different models. What they do is they have a bunch of real issues from different repositories and they have the agent try to resolve these issues and see if they get an actual passing test after an agent went off and resolved the issue. So that's what we bench is. So what our problem with the ones that are in the industry is that they're Python, they're running on Linux, all this stuff. It's not very relevant to Visual Studio. So we built our own CWEB bench that tests exactly the kind of things that people in the room here would be using you know like .NET framework net core Azure Aspire all this stuff and then we're evaluating how are these agents doing with these issues and are they actually resolving the issues or not. So, we ran this against the uh old agent and the new agent. And you'll notice a new agent. It costs less to run, which is like, you know, on everyone's mind right now. Cost is a big deal here. It actually cost a lot less to run uh with this run and it because it to it it used tokens to do what it needed to do. It also worked in less steps. The reason this matters is because it goes into the the the uh cost as well and how fast it goes. So we used less steps in order to resolve the issues or resolve the tickets. Um and it errored out less. Um I know we're we always try to go for like a 0% error rate but really a lot of times these model fail and you don't know why they're failing. Um so it did fail even less. Um, it's also faster. In this case, it was 10% faster, but in my case, it was like, you know, a lot a lot faster. Um, but more importantly, what I really care about as a PM on the Copilot Visual Studio team is how many tickets or how many issues did it actually resolve. If you, you know, if you have your agent do 10 things a day and it fails on eight of them, that's really annoying. That's just not helpful. But if you have it run on 10 things and it resolves five things for you and actually helps you on those five things, that's an that's a useful agent. Um, so in this case, I care a lot about that five number. How much how helpful is my agent? Um, but not only are we getting all these gains and all this great stuff, but also because we're integrating the Copiot SDK, we also get a lot of sub agents for free. Uh, anyone know what sub agents are? It kind of, you know, you kind of know it in the name. It's like if you have a main agent that you're talking to, it can spawn sub agents, peril agents that go off and do their own things. Um, and in this case, we have specialized agents that know how to do specialized tasks. And you can tell like these are common development tasks that people care about a lot. So, we have the explore agents. it it goes off in your codebase, reads all your files and comes back with answers when you ask a question about the codebase. We have code review, task, research, general purpose. My favorite is the rubber duck one. The rubber duck one acts as a second opinion and it goes off kind of kind of the concept of a rubber duck in programming where you can talk to it can have it kind of go off and um try different things in different uh models with different models. Maybe you want to get a different um evaluation of your code from different models and see who comes back with the best code review for example. So those are the kind of things that you can expect from the sub aents. All right, I'm going to go into the tricky part of the presentation which is live demos. Again, if it fails, you'll watch me fail, but if it succeeds, we can all celebrate. Um, okay. So, first demo I want to show is really go into how to use Visual Studio with this new agent mode. So, I'm going to switch here to Visual Studio and hopefully everyone can see. No, no, no one can see. All right. All right, it's coming up. Hopefully, let's try that again. Hopefully, everyone can see. Y'all good in the back? Okay, awesome. All right. So the first thing that I want to show um showcase is one of the use cases for using agents um that are around researching, exploring uh looking around uh and finding different answers. Actually I want to show you something. So um I love this survey stack overflow survey. So they asked a question asking hey part which parts of your development workflow are you currently integrating AI into? And notice that the top answer is not generating code. The top answer is searching for answers. Searching your codebase, searching docs, searching because as programmers, we're not just paid to write code. We're paid to make sure that whatever code that we're writing in whatever codebase that we're writing it in is not going to break that code base. So we're actually reading code a lot. We're understanding the code a lot. We're searching for things in the code. We're looking for examples. So on so forth. So it made sense to me a lot that a lot of people are saying searching for answers is actually the top use case for them and that's why I wanted to show that show that use case and how you can use agents um for searching for answers. All right. So here's Visual Studio and we have here the chat on the right and we have uh uh multiple pickers here. Some people don't notice the picker. So I'm just going to do a quick recap of Visual Studio and the pickers and all that good stuff. So here the way that we think about this is your prompt is like what you want to accomplish. So you want to say hey help me fix this code blah blah blah all the stuff. This is who do you want to accomplish that task. In this case I want agent preview because that's the better agent of the two. But you could also choose the debugger agent. You can use the modernize agent, the profiler agent. So when you're writing a prompt, you want to ask yourself, who would be the best agent to do whatever I'm trying to do. Next is how you want that task to be accomplished. So we have different modes here. Interactive is when the agent does something. It asks you questions. It gives it asks for confirmation. Hey, can I read this file? Can I edit this file? all those kind of things. Bypass is like a lightweight yolo mode where it's like don't ask me a lot of questions, don't ask for a lot of confirmations, but if you're about to touch something sensitive, can you ask first? This is like yolo mode, right? This is autopilot mode where I want you to just go go do the task that I asked you to do. Don't ask me a million questions. Just keep going. And the way personally I used autopilot is when I've done a task a million times and I've seen the agent do it well enough that I trust it not to do anything destructive. So like I I love this mode because it's it just lets me walk away from the computer, but there is also kind of a risk to it that you need to be thinking about as you're using the different modes. Do you trust your agent to go off and confirm, confirm, confirm? And the other thing we do in autopilot mode, which I didn't realize until yesterday, was we actually tell the agent, keep going, keep going, keep going, keep going, don't stop until the task is done. So that's the other thing that we do with autopilot mode. We tell it, hey, someone walked away from the computer, just keep going. And we also tell it, don't ask questions, just assume that the answer is yes. So use it with discretion. On the opposite side of the spectrum is plan mode. Plan mode is when you have a vague idea of what you want the agent to do, like you have a feature kind of know, but you don't quite know and you want to plan with the agent. This is going to be very good to use when you're So, okay, a lot of people when they use agent mode, what they want to do is assign it a task, have it read their mind, do the task correctly. If the task is not concrete enough, what's going to happen is you're going to assign a task, give it to the agent, and the agent's going to do the wrong thing. The reason you do plan mode is because you're having a conversation with the agent. I actually a lot of times I tell the agent, ask me questions. If something's not clear, ask me questions. Because I'm not thinking, oh, I need to clarify this. I don't realize I'm being vague. So, I go into plan mode when I'm about to do something like a new feature that it probably not know the answers to. I'm designing. I'm doing all these kinds of vague things. I have it formed the plane with me and when I'm very happy with the plane, then I switch to autopilot mode and tell it, "Okay, go do what we just agreed to." So, think about how you want to use your agent in those different modes when you're about to do a task. And then finally, we choose a model. This is the raw brain, the raw model, and it makes a difference. So, in our case, we're always bringing in new models in Visual Studio. We added Opus 5 here. You'll see um GPT56, all the different models here. You can always open this up and check off the model that you like. I've been using 48 a lot, so that's what I'm going to stay on. Um but feel free to switch your models to what you'd like. Okay, so back to the agent mode. So why how would we use the agent mode to search for answers? So let's say I am in this application. So I have the eShop application is kind of an e-commerce.net application and let's say I'm trying to add something new like a lot of times you're adding a new method or a new class but you already have examples in your codebase and a lot of times as developers we go off and read how someone else did something before we add our own thing. Well, you don't really need to do that yourself. You can have the agent do that on your behalf. So in this case let's say I have a status order and I have different statuses. I know there's different statuses and I want to add a new one called refunded. So, I'm going to start talking to my agent. Uh, by the way, I use voice. So, if I'm talking, I'm actually talking to the agent. So, I'm going to do that really quick because I I don't like to type anymore. So, I'm going to do this. Hey, I am trying to add a refunded order status. Um, I know we have a few statuses in the codebase. could you go do some deep research on the codebase and look at how each order status is implemented um and then come back to me with a summary and maybe a suggestion on how I can add that refunded status. By the way, uh this is I'm using a separate app for this, but we're actually uh we have an interim project to add voice in Visual Studio. So hopefully that will come here soon. So notice what I did here, right? I I'm about to add something new in my codebase. If I had all the time in the world, I'd be like reading all the classes, reading all the methods, making sure that I'm making sure that I'm flowing correctly with the codebase. And instead of doing that yourself, think about it from kind of a research assistant perspective. You tell tell a research assistant, hey, go off read all these files yourself and then come back with a a customized summary for you that will make sense to you and in the context that you want it. That's why I started it with what am I trying to do? I'm trying to add the refunded status so the agent kind of understands what my goals are and then I tell it go off and do what it needs to do. Now, when you used to use agent mode in the copilot um old mode, this used to uh work just fine. But now, because we're using the copilot SDK, we get advantage of those sub aents under the covers. So, it it'll probably spin off the research sub agent and it'll have it go read code line by line and come back and give me a deep analysis of what it found. Some people ask, hey, like, should I be spinning off the sub aents myself? You can do that. You can tell the main agent, hey, can you go off and uh spawn off like three or four sub agents to do this task? It is more expensive, but I found that you get good results if the task is very isolated. It's uh like for example, I want to uh Okay, this is something I do. Don't tell the VS Code team I do this. So, we learn a lot from the VS Code team. So, if I'm about to add a feature like voice in Visual Studio, I actually have my agent go off look at the VS Code implementation and I have it go off and look at the CLI implementation and then come back and tell me how it applies to the VS implementation. So, those are two separate tasks. Go off and look at the CLI implementation. go off and look at the VS code implementation are two separate tasks. So I spin off separate agents to do that and the reason I do that is because a lot of times they kind of lose track of the context and they can't keep so much things in their heads. So you have to have them go off do that investigation come back talk to each other and then summarize. So that's how I've used sub agents. Um again you don't have to do it. It is more expensive. Um, but if you know that it's going to be better to paralyze, then you can do that and ask it to do that. Um, okay. So, I sent it off. It did the exploration. It did the research. And here's what it came back with. So, I I want to just talk about a little bit of how you interact with your chat here. When you when I sent off this thing, this prompt, you'll see it do a lot of work. And this is uh this is a place where we argue a lot on the team. Do we show the thought thinking? Do we show the thoughts? Do we show the tooling? Do people care? And then we could hear sometimes they care. Sometimes you don't. Sometimes you know you want to know what the agent does and sometimes you don't. You just want it to do what it needs to do and just shut up about it, right? So the compromise is that we have thoughts but they're collapsed by default. So if you're ever wondering, hey, what did the agent actually do? Which I do that a lot. I'm like, huh, I wonder how it interpreted my question and what it's going to do with it. You can expand the thought and look at, hey, I'm diving into the ESOP codebase to understand how it order. Okay, then what did it do? What did it do? Oh, I need to dig into the core files. Okay, so it went off looked at the order enum. And this is really good for junior developers like this is what a senior developer might do actually. So you kind of learn what is it doing? How would you go about this? And if you're a senior developer, you might look at them and be like, I don't know about that. And then you can tell it no, call back and do something different. So you you can read through the thoughts, figure out what it's doing. You can look at the tools. We still tell you what tools is doing because we need to be transparent what about what it's reading so that if it's reading something you really don't think it should be reading, you can stop. And then notice that font difference. this is all the things that it did. Once that I see that font change that I know that this is the response. This is what it's telling me. Not what it's thinking and what it's doing. It's what's telling me. So it's saying, "Hey, how order statuses work today." So saying, "Hey, this is the enum um that you need to touch." It has all these different uh order statuses. It looks like we have six enums in here. This is a state. And what's nice about using Copilot and Visual Studio is that you can click and go, right? you can click and go and do kind of a tour along with your agent and go to um the different classes and read them and then move on and just kind of keep exploring as you go. This is something that's really powerful about using Copilot in Visual Studio is that you actually get the visual. You don't just have this like black box that you don't know uh what's happening in. And then finally, it gave me a suggestion for how to implement refunded. And back to the gentleman that was saying, "Hey, I don't like Asians touching my stuff." You can have it not touch your stuff. It can just do the research and come back with a proposal. It doesn't need to do it on your behalf. You can just ask it for a recommendation. So, in this case, it's giving me a recommendation. And even more importantly, it's asking me design questions. So, it saw some stuff that it wasn't really sure how to decide things on. And so it's coming back and asking me, hey, if you're going to implement refunded, here are the design questions you need to be thinking about. So that's one use case is like having the agent go off, look at your code, kind of do that research. But the another one that's interesting is what about the agent doing research for me out in the open? So, let's say for example, I want to say, "Hey, could you research what's new in the latest C versions and find some spots in my app that could use it?" So, what I'm doing here is I'm telling my agent, I want you to go off to the web and search Microsoft Learn, search all the Microsoft Docs. The reason I say Microsoft is because I actually have the tool here enabled, the Microsoft learn tool. By the way, this is where the tools live. Um, and the skills. So, the tool is going to go off and it's going to fetch the car docs that we write every time we release a new C feature. It's going to summarize it for you and then it's going to look at your code and be like, "Okay, this is where I think this can be applicable." Because you can't keep up with all the announcements, you know, you can't keep up with all the features. So, why not have the agent go off, do the research, come back, and then give you what's relevant to you. So, that in this case, that's what I'm doing here. Now, when I am writing these prompts, I actually trigger my engineering team. Um, they're like, "Dia, you're being so vague in your prompts. Like, why are you writing them like this? Why aren't you more specific?" And I tell them, "If I'm doing work dayto-day, I'm not sitting here like thinking, "Oh my god, how do I explain?" Like, I I I just kind of let things flow and then I keep having a conversation with my agent. So, I was actually vague on purpose in this case because sometimes you write a prompt or you ask the agent to do something and you're not thinking through your wording. In this case, I said, "Could you research what's new in C?" And I said, "Find some spots in my app that could use it." When I run this multiple times, I get multiple different things that it does. Sometimes it will change the code for me. And that's not what I asked it to do, but I kind of asked it to do that. I said, "Find spots. What does fine spots say mean? Do you want recommendations? Do you want me to choose the app? what do you want from me? So if you want really if you're really specific about I don't want you touching my app the instead the right prompt should have been could you research what's new in the latest C# versions and give me some recommendations that I can apply in my app so I can use it. Notice even okay as we're talking it actually changed the code. These agents are really eager to please you. they want to do the work on behalf. The the definition of an agent is someone is something that actually does work. So in this case, the agent is like it wants to please you. It wants to find spots in your app and integrate the new C# application. So that's what it did. But on my end, that's not what I wanted, right? I wanted it to actually give me recommendations and then I'll apply it one by one. So I could to get out of this mode, you can always come back to the changes window and be like, ah, that's not what I want. And then you can undo the work. So this is what this undo button is for. You undo and you just you can undo here or you can undo here. So the reason I'm showing you this is because I'm not going to, you know, you're not going to have someone sit next to you and tell you like this is how you should prompt. You should prompt the way that you should you want to prompt. If the agent does something that you don't like, come back and tell it no, that's not what I meant. And correct it. Be forceful. Be specific. It's not a they're not humans. They don't get hurt feelings. You can tell it next time I ask you for the recommendation, don't touch the app. Um okay. So, uh lastly, um I do want to just touch on like the research uh thing that uh that we did here. Um when I do research, I don't want to read it in this little small little window here. So a lot of times I tell the agent how I want to read it. So what I will say is, can you write a report on the new C# features that you just found and open it here in Visual Studio? So I'm going to just have it do that while I I tell you why I did this. The reason I do this is because I learn in different ways. Different people learn in different ways. And what's really great about agents is that they can provide you information and content in the way that you learn. So I like for example uh like tables. I like diagrams. That's how I learn. So I ask it or I like overview first and then deep dive bottom. A lot of times I tell it give me a report. I want it to be collapsible. I want the overview to be at the top and then I can read it the way that I want because people learn in different ways. So when you are doing research, you're asking the agent to do research on your behalf. Don't be afraid to tell the agent the way that you want the content to be rendered to you. Like don't just assume I'm just going to read the chat this uh a crap ton of text that I need to read carefully. You can tell it no produce a report for me and I want to read it. So in this case, oh sorry um open up. This is what happens when you're on main. [laughter] Um, in this case, it created an MD file and um, it created the tables. It created what I have in here. And in this case, I can what I do because I I'm a little bit old school is I print it off and I actually start reading it on a piece of paper. I highlight I circle. Anyone print off stuff anymore? Okay. Not me. Not just me. Sorry y'all, [laughter] but I like to print off things, read it in hand because that's it's going to help me absorb information a lot faster. So that's what I do. So you can have it or you can have it produce an empty file and look at it in the markdown uh preview in Visual Studio right here. Right, I can see the table. It's a lot prettier and it's a lot easier to read than just the chat. You can look through and it has some like before and after. It's very well formatted. it takes advantage of that whole screen on the left that I'm not using right now and I can read through the report and just take my time parsing through. So just something to keep in mind when you're doing some research or something to do with reading a lot of content from the agent. Great. So uh with that I want to switch to a different use case and this one is my favorite. Um, okay. So, let's switch to this. How many people here um have a bug or a ticket assigned to them? Like one bug or one ticket that you have to work on when you get back? All right, keep your hand raised. How many have five? 10, 15, oh, 20? I'm concerned. 30. [laughter] So I used to uh work as a software engineer and uh my whole job for three years was handling customer tickets. It was a miserable job. Uh by the end of it I was like I don't want to do this anymore and I switched roles. Um truth is that's a lot of what we do as software developers is like we handle tickets, we handle issues, we handle um customer reports and I love that use case for agents because it's like a tedious work that I don't want to be doing. I'd rather create a new feature, work on new code, uh create new like customer value, like help more customers. Um I mean I'm helping them by fixing their bug, but I'm not helping them with creating something new, right? And so I want my agent to help me with that tedium. So the first thing is um I have an example of an issue here. This is imagine this is your ticket that you have been pro procrastinating for you know a week or two. you're like, "Ah." And then your manager keeps asking, you're like, "All right, I'll get to it." Um, so imagine this is that ticket that you you do not want to get to. Um, first thing is a lot of times when people open up tickets or customer issues, they're not very well described. People are not very good at describing their problems. Um, so you'll get like a short uh tier, don't get offended because I had I had tier create this for me, but it's a I gave him the content though. I was like, I want you to be vague. I want it to be like a representation of what a ticket will be. It'll be vague. It won't have a repro. It'll be something that I squint at. In this case, it's kind of simple, but like some of the tickets we get is like, I don't know what you're talking about. So that's that ticket, right? The one that you've been not wanting to touch. So, what I do is I go in Visual Studio and we're going to create a new chat. I'm actually going to uh let's close out of all this so that it's not so that you know what to pay attention to. [laughter] Um, what I'm going to say is the first thing I do with any ticket as a as a software engineer is I want to reproduce the ticket or reproduce the issue on my machine. The reason I do that is because I'm going to have to do it anyways when I have to create a test. I also want to know if it's already been fixed and I don't have to worry about it. And like you can be like, "Oh my god, it's fixed. I don't have to deal with this anymore." So that's usually my flow. I don't know what your flow is when you're fixing bugs or tickets, but that's my flow. So that's what I'm going to do. I'm going to say, um, can you read this issue and, um, give me some steps to reproduce it? So, similar to the video I showed you, I'm going to send over the URL. So, how many people here use GitHub for their ticket uh management? Uh DevOps. Okay. Anything else that I didn't ask? Jira. Okay. Okay. Awesome. So, we have different MCP servers. The whole point from those MCP servers is to go off, read your GitHub issues, bring back the ticket information, the title, the description, all that. Highly recommend taking advantage of those. If you're finding yourself copying pasting descriptions or customer issues from tickets, ask yourself, would it be better if the agent has access to the title and the description themselves so that they can also look at the comments? They can also look at the logs. They can also look at all the attachments that you forgot to mention. So, I always send over like either a URL or a ticket number and make sure uh some of my MCPS are uh on. So in this case I asked it can you read the issue come back with some steps to reproduce it. So I read the issue it looked at it it gave me some steps to reproduce. So you can imagine you got a ticket you don't know how to trigger it. So let's take a look. So it said run the solution via the app host. So it's telling me F5 in Visual Studio which might of some of you probably do for your applications as well if it's not too complicated. And then it's telling me on the landing page note the left filters panel and uh with the type group. Okay. So usually what I do is I have two monitors and the app is running on one monitor and I have Visual Studio on the other. So I'm going to follow the steps. I'm going to click here and it brought up the app. This is Aspire. So this is all the services that are running. And then I'm going to go to this online store. And then I'm going to take a look here. And I see a brand filter. So I'm going to come back and say um says on the filter page note the left filters. And I'm like ah I'm looking at this. Okay. I'm going to do this and you're going to think I'm being lazy, but sometimes I'm not having a good day. And I look at something and I'm like I'm not seeing it. What is going on? I actually do this. I take a picture and I tell my agent, "Hey, I don't really see the type filter. Um, what's going on here?" And what I do is I paste in an image so that it can help me figure out what I'm not seeing. The reason why this is important and uh this is a silly example but the reason why I want to show this is agents can read code they can look at all the code the text all the stuff they can't see your app they don't see the UI elements like if there's an alignment issue it thinks it just produced the alignment just fine so whenever I'm criticizing it for something it should have seen I actually take a snippet of the web page a snippet of whatever element I'm trying to show and I send it to it so that it knows what I'm talking about. Does that make sense? Okay. Awesome. So, in this case, I'm being a little bit lazy or maybe I'm having an off day, so it comes back and it's just like, "Oh, good catch. They're so condescending. Good catch." It's always like really like, "Oh, you're so smart." I'm like, "Okay, yeah. Um, in your screenshot, the type is missing entirely. Let me check the data." Okay. But what's nice about this is it didn't just give me an answer. It went off and looked at the code to make sure I'm not seeing it for a legitimate reason or is it a human problem. In this case, it's a human problem. It says, "Hey, to confirm, can you scroll down in the filters column and like look at it?" Oh, okay. I actually have to filter a little bit more to see the type filter. Uh again, it's a silly example, but I'm just giving you an example of like those visuals where you can actually pass visuals to agents and have it look or see the picture. So, in this case, I'm going to go back to the instructions. I'm going to click on this and yes, the filter is not working. So, I just reproduced that ticket that I was like dreading working on, which is I don't know about you, but I got so excited whenever I can repro something because I don't have to look at the customer logs. I can look at my own logs. I can debug in my own environment. I can set break points. This is why a repro is so important to me. So there it is. It's local. I don't have to even look at that ticket anymore. I can reproduce it. So I can keep going with my agent. So what what I noticed was in the in the ticket I'm thinking, dude, this is this has worked before. I don't know what's going on here. So I tell it, hey, I think this was working before. Could you search through the commit history and see if any commit could have uh uh caused this regression? This is really powerful. Why? Because a a lot of times you're in the code and you know something has worked before and if you had all the time in the world, you'd go through each commit and be like, who broke this? Right? Who do I blame for this? Well, you don't have to do that anymore. You can have your agent do it for you. So, that's what the Asian is doing. It's going through every commit in the history and looking through it and figuring out, hey, did any code change could have because you're you're kind of filtering down to the things that could have uh caused the issue because it knows the files that could be influencing this. So, it kind of has a trail but not quite. So it'll go off and look through all the commits and like a lot of things in life um sometimes you discover that you're the problem and in this case I was the problem. [laughter] So I committed something thinking I want to encode my URL. I was thinking really like oh I want to do this safely but I encoded the entire URL instead of encoding part of it and that's what introduced the problem. So that's another way that you can have your agent go through find regressions, search through commit history. I basically whenever I'm searching through things manually, I'm like why do I have an agent do this for me? That's how I trigger it in my brain. Okay. So finally I'm like um okay. So it says the clean the cleanest fix is to go back to what it was before. So either kind of revert the commit and um and it's asking me hey do you want to implement this? So in this case I can say yes and have it actually fix it. Now you might be sitting there wondering do why did you have all this conversation just to finally tell your agent to fix it. The reason I show this example and not the example of like hey can you attach a Asian to this ticket and have it fix it and fix it and yay. I don't love those demos to be honest with you because those are not the kind of tickets I got when I worked as a software engineer. They were very like intricate. You have to be very careful what you're changing in the code. It spawns different repos. There's no way the agent would have figured it out. There were easy tickets, right? There were tickets where you look at it and you're like, yeah, that'll probably take me 30 minutes to figure out. This one is kind of one of them, but there's tickets that were like, I have no idea why it's doing that. I need to go through different code bases. I need to make sure if I change the behavior here, it's not going to impact all the people that are calling me. Those are the kind of tickets that I worked on. And I don't know about you, but that's probably a lot of the tickets you work on, too. So, that's why I go through all this is because as you're working one stat at a time, you can work with your agent alongside you and stay as your like you're the pilot. You're the one that tells it what to do and have it be that co-pilot that goes off and acts as your assistant. Oh. Oh, I'll go off and do the research. I'll go off and look at the commits, but you're still in charge. So, here it went off and uh did the fix. So, I'm going to uh like always always write your test. Hey, can you go off and write my tests for me? Because no one likes to write their tests. So, that's what I'm going to do. I'm going to tell my agent this writing tests are one of the best use cases for agents, by the way, because it already knows what's expected. It's not vague. It needs to be something that is like, well, it depends on what kind of app you have. Some tests are easier than others. Unit tests are easier than others. But the a test is an expectation or a spec of what you're expecting to happen and the agent can work with that. So, writing tests, if you're finding yourself writing tests, highly recommend reaching to your agent and asking it to write the tests for you. Um, okay. In this case, it's going to go off write some tests. And every time I've had this demo, I was like, what am I going to talk to folks about while we're waiting for this? So, I'm going to cheat a little bit and I'm going to go back to one of my previous um So, it's asking me questions. All right, I'm going to go back to one of my previous um things where I actually wrote write a test for this. I swear this is legitimately working. So, I'm going to go back to an old um uh conversation here where it actually came back and told me that I it wrote the tests for me. Oh, did you notice this? So, I you're able to uh rename your conversations now. I think that's super nice because a lot of times I'm like, what was this agent doing again? Hey, it was fixing bugs. Bug fix test. uh type issue. If you're ever having like three or four agents running at the same time, a lot of times you lose track of what who's doing what. Highly recommend renaming your threads. Oh, go ahead. Great suggestion. I'll I'll I'll tell my agent to do that for us in the repo. No, you can't right now, but that's a great question. Yes. Yes. I'll bring it back to my team. Okay. So, it it wrote the test. We're happy with it. What I do when I have had a really good session with my agent because I'm about to start a new ticket and I don't want to go through this flow again is I say, "Hey, this was a really good session. Can you turn this workflow into a skill so I can use it later?" I have my agent write my skills for me. Uh, anyone know what skills are? Okay, good chunk here. So, the way that I look at skills is basically you're pro like you're programming in fuzzy words what an agent should do. So, agent can't read like unlike uh structured languages. You can actually tell agents to do things in natural language. So, it's going to write in natural language some instructions for what it's going to go off and do. In this case, I told it, I just had a conversation with you. I asked you to go repro a ticket. I asked you to go fix it. I asked you to go write tests, all the stuff. Next time I asked you to fix a ticket, I want you to do the same thing again. And I don't want to I don't like repeating myself. So that's what I do. I ask the agent, can you please write a skill? Because when you write a skill, you're codifying that workflow. write the skill for me and then I can use it next time that I am uh fixing the ticket. So, here's what it did. What what's going to do here? Um oh, there's already a skill. Uh maybe I didn't clean it up from last time. So, last time I we did the skill. It was going to go off. It's going to look at the investigating bug stills. Let's see if it's uh writing the skill up. And the reason I want to show you the skill is it's pretty much like just a natural language structure of what you're instructing the your agent to do. So I'm going to look here and see if we can open this up. Um actually let's do this. Hey, can you open the skill here in Visual Studio again? I don't want to go searching for the file and like opening it up again. I'm uh I'm a little bit lazy. Uh some days I am lazier than others, but some days I actually it's not laziness, it's efficiency, right? I I want to spend my time not searching for files. I want to spend it coming up with the best um outcome here. Let's see what happened. Oh, did I not open it again? Sorry. I don't You don't have to apologize to your agent. Yes. Um, let's take a look. If it's not, it should be. Uh, let's see. I don't know why it's rendering. So, this is a problem in Maine, I think. Um, so anyways, um, let's take a look if it's run. Can you click on it? It is. Ah, good call. Yeah, I can. Let's see. Let's look at We're going to look here. Yes. So, it went there. Um, okay. So, the reason I want to show you this skill is so that I demystify what skills are. So, here's what skills are. It went off. It gave a skill a name and a description. And the reason this is important is when the model goes through and it's looking through all your files, it's like, oh, I already have a skill. Then I should probably call the skill when the description is true. So, if you're fixing a bug, it knows that you have a investigate bug skill. So, it'll call this. If you're creating a new feature, it'll look and it'll find the investigate bug skill and it'll be like that's not relevant. It'll move on with its life. So that's why it has a name and a description at the very top. But in the skill itself, it's asked it has the guiding principles when to use the skill. Again, very verbose, but that's okay. I'm not reading it. The agent is, so that's okay. But what's important here is notice it said, hey, first thing is I want the agent to read the issue. Yes, that's what I want. I wanted to locate the real code path. Yes, that's what I want. I wanted to try to reproduce. So that's something specific to me. If if you don't like reproducing things first, or maybe you like TDD more, you like writing tests first, you can tell the agent, hey, write me a test that fails and then fix it and then uh make sure that it passes. So this is going to be specific to you. So you you're in control here. Confirm that the regression. So if it's a regression, go through the commit history, search through it, and come back with what commit was the problem and so on so forth. So it wrote its own skill. Now, a lot of times the first skill I write, I'm not very happy with. And the next time I investigate a bug, it does something that's like, oh man, I forgot to tell it to do that. You can tell it, hey, go back to that skill and fix it. So this skill is a living document for me. It's always getting updated and it gets better as I make I I'm training my agent to work for me. So, it's really important when you're writing skills to figure out how can you help your agent help you. Awesome. So, that's uh all about skills. I want to go back here because we have 15 minutes left and I'd love to show you more things here. Okay. Um, so again, I like real examples. I don't like this. Uh, I I I actually think Eshop is a little bit too simple. So, I went through the Visual Studio repo and I looked at our own investigate bug skill and I read it and I was fascinated by some of the stuff that's in there because it it's like it tells you all the things that someone else some engineer was sitting there just yelling at the agent and then it was like ah I need to program this into the skill. So, these are some of the things that we have in our own investigate bugs skill in for the Visual Studio repo. Our repo is huge. Has a lot of files. It's like very real. This is like real things here. So first thing is this one was very interesting. Read the bug and the data first and then people theorists uh last. Why did why did it do that? When agents read something, they treat everything with the same priority. So when it reads a title and it reads a description, it's like, "Oh yeah, that's the issue." and then it starts reading comments and it's like oh yeah that's issues two those are two different things a title or a description is coming from a customer this is their lived experience what's breaking the comments are sometimes an engineer saying oh I think that's what the issue is and the problem is if the agent treats all the information the same then it's going to think that someone's engineer's perspective on what the issue is is actually fact and thus not true. So in our own skills we say hey I want you to read the title the description and we specifically say don't read comments yet. So what happens is it goes off reads the title description looks at the logs looks at the files looks at the dumps looks at what the looks at everything and then it comes back to the comments and be like hey is this person's theory match what I already searched for is you know think about it from your perspective when you're when someone tells you something a customer's perspective is probably right because their stuff is breaking but if an if someone on your team says, "Oh, I think this this is this thing." You're like, "Sure." And then you go verify it. So that's what we're asking the agent to do. The other thing is stay skeptical. Agents really want to just appease you, appease you, appease you. So sometimes you have to tell it like, "Hey, treat any existing analysis as a guess." It really ties into that. Uh um make sure that you verify things first. The other really important thing is very very important is you tell it your local knowledge. So Visual Studio repo has very specific things about it that doesn't apply to other repos and your own apps have very specific knowledge that lives in your head. You've done this for years and there's a lot of conventions and a lot of things in your head that no one has taken the time to write it down. and now is the time to write it down because those agents if they know what's in your head, they're going to operate a lot better. So in this case, we say, hey, our logs uh the VS logs actually don't just live in, you know, if you're a simple app, you'll have the cute little logs, but for us, we have like a lot of log files, including dumps and logs and copilot logs and all these logs. If you have a specific place where you use specific information to diagnose your app, tell it where to find that information is attached to DevOps. What what's what are the files? The description of the files. Something that if you had an enter next to you, you'd be explaining that to them, right? So that's what you want to say is here's where you find stuff. The other thing is a lot of people uh apps don't just live in one repo. A lot of times it lives across repos. So I tell it, hey, like don't just look at the solution and what you have open in Visual Studio. The rest of the code lives in this folder where I have a bunch of my code cloned. Look at all these repos. Hey, this like have your have your agent see what else is there that they don't know exists. And also the way that you name things. 18 like VS 18.8 is only applicable to us. I don't know how your product does versioning, but everyone does versioning differently and you're going to have to explain that to your agent. And I love this one is don't let it cut corners. There's a bottom portion of the skill that was like a lot of like don't don't like oh someone got hurt. [laughter] Don't guess what the code does. Uh sometimes the agents are being lazy and instead of actually reading the code, it guesses what the code does based on the method name and it's like no no I want you to actually read the code, read through it and don't just redescribe something. Right? The other thing is um don't be overeager in this case leave what you don't understand. What means this is sometimes the agents go off touching code they shouldn't be touching because they're trying to be helpful. So we tell it, no, I want the minimum amount of fix to fix the issue I told you to fix. Don't go refactoring. Don't go doing stuff that I didn't ask you to do. Again, this is something that's in our own skills. Um, so I wanted to just give you a little bit of a glimpse of the types of things that we do with agents for like a real enterprise application because I think again I think that's really interesting and that's how you figure out what you want to do for your own application. Um, our most popular skill which was a surprise to me is managing PRs. is such a long tedious skill. But if you think about it, the way that you have git configured for your app is very specific. People probably have very specific ways that you want their branches, very specific ways they want their PR descriptions, very specific ways that we don't like in Visual Studio, we don't push to main, but the agent thinks everyone pushes a main. No, that's not what we do. So, we have a whole skill that's all about get management and managing PRs. So that's one thing that you can do for your um uh environments. Other thing is investigating tests. I just did something and committed it. My CICD uh pipeline is now broken and my tests are broken. Hey, can you go off look at the logs, look at my uh issue and tell me did my issue break this or someone else's. So those are very good examples because it's a concrete ask. Perf investigations. So uh a lot of these skills are written by uh one of our my favorite engineers. His name is David Keen. Anyone here of David Keen? He's a performance expert and he writes a lot of blogs on Visual Studio performance. And he basically um saw this as I don't want people to keep pinging me all the time with their performance issues. I want to codify all my knowledge into a skill. And that's what he did. If you're an expert in a certain area and you want to codify your knowledge so that you don't have to be talking to your juniors all the time and you're and they can go and ask that skill questions and figure out how you do what you do. That's a really good way to solidify your experience and instead of him spending a bunch of time on performance issues, he's helping us upskill the entire repo and helping us make Visual Studio perform better. You know, some people worry about like job security, but in my case, what I want David to actually focus on is make Visual Studio perform better. I want it not to be slow. I want it not to hang. I don't want it to deadlock. So, why not turn this into a skill? Because a lot of times the agent will fix like simpler issues, but it won't fix those hard ones that I want Davis time to be spent on. Uh, someone was telling me, "I'm here because I want a promotion." That's another way to get a promotion, by the way, because you're using AI to make your team more efficient. So, just an idea there. Um, and then managing builds. So, um, similar to failing tests, you can also help have it help you uh with failing builds if you're introducing a problem and have it pull in the uh failed build and the logs and tell you if you're the problem, right? Um overall what I want to leave you with is uh for the skills is don't treat it as I wrote this code this is perfect I don't want to touch it this is it needs to actually be touched it needs to be improved so whenever I'm doing something over and over if I'm constantly fixing tests think of all the things that are repetitive that you have to do a lot every day I always turn those into skills now um I have to do these weekly status status report. My manager just left so I can I hate doing weekly status reports. It's such such a pain. So I created a skill for it. And what I had the skill do is I want you to go through all the code and all the commits and tell me what the what the engineers did, describe it in the way that my manager likes, give it the format that the manager likes, and do it for me. And then I go off and edit it and click on things and make sure that it's correct. But then I don't have to spend an hour on status updates anymore. I have the agent do the things that I don't like doing. And that's where that skills come in. And I I keep yelling at it every time I work with it because it's like, ah, this is not what I wanted. Keep yelling at your agent. No, be nice to your agent, but just tell it, I didn't like this. Improve the skill. Why do you keep doing this? I want you to do this instead. Keep working with your agent. It doesn't have feelings, so it's okay to be a little bit more direct. Um, and it only gets better from here. So everything that I showed you is just a point in time. Right now we're we have better agent in Visual Studio. And what we're going to do here is as we pull in Copilot SDK updates, we get all that goodness for free. The nice thing about going off and using someone else's code is that if they improve it, you get it those improvements for free. And that's what's going to happen in Visual Studio. We have a dedicated data science team that is always improving these models and doing evaluations and all that good stuff. And we're going to you're going to be seeing that in Visual Studio a lot sooner than you did before because we get those improvements for free. So you're going to um take advantage of those improvements. The other thing is you're going to have a consistent experience. I know a lot of people switch between VS Code, Visual Studio, the CLI. you'll have a more consistent experience when you're switching back and forth between your different tools because we're using the same harness under the covers. Lastly, and that's the one that I care about most as the PM on Visual Studio Copilot is we can really focus effort on the things that make Visual Studio stand out. Um, I noticed that with the CLI are becoming more and more popular. I feel like we're going back in time because I'm like, dude, like this is so primitive. Why do I have to read all this all this text and invoke a bunch of commands? The whole point we have idees is because humans need actual visuals when they're working. They can't just read text. So, I want to spend more of the team's time figuring out how do we make Visual Studio work for you? How do we make it work for you when you're working with agents? Um, I'd love to hear more from the folks in this like after the talk if you have ideas on like this is not working for me. I have the agent here and I have like I always tell my team like why do we have this small little chat like why aren't we bringing it to the middle like all those kinds of things how how do can you visualize things how do you not have a bunch of context in your head how do you work with this within the context of Visual Studio um more integration with like the tooling debugging testing those are the things people come to Visual Studio for version control I think that's super important agents changing a lot of files. Can you make it easier for me to review all the files and like understand the changes it did? How can we make that shine and really specific net and speed C++ uh you know u integrations. So that's what you should be expecting moving forward. Um I've left some resources here um so that in case you want to read more about it. I have the first one being the agent mode. Uh, if you want to read more about Asian mode, um, the copilot SDK, read more about that. Um, and then the built-in agent that I mentioned, like the rubber duck agent, the explore agent, the ones that you get for free if you use agent mode preview. And the last one is my favorite read, and I sent it to everyone. So, one of our engineers on the net team, his DM is Steven Tob went off and he was curious like, are these agents as good as we think they are? and he ran a bunch of copilot uh agents with our own real repo, the .NET repo, and figured out what is it good for, what is it not good for. And in this case, um he found like it does really well with refactoring tasks. It doesn't do so well with these kind of tasks. So if you're wondering what are these agents for, how can I use it for my own uh application, my own environment, it's a really good read. It's very long, very detailed, very technical, but it's a very good read. Highly recommend that one. Uh lastly, I just want to invite you to uh come to So, we have a session that's coming up at 3 today uh where the team is going to have more of an interactive uh session where we ask you questions and you can give us input on Visual Studio, how to shape Visual Studio. That's what we need, right? We need input from you. You do the real development. come come and tell us what do you need to make copilot work for you better in Visual Studio. I'm going to be at the assert come find me and the Visual Studio booth booth tomorrow as well. Um and I'd love love love if you can try the agent preview in Visual Studio and give us feedback. Um we look through every ticket and we read through them and we try to uh make sure that we are making it work for you the best we can.