How the Visual Studio Team Uses AI Internally

Thank you for being here today. To to have some discussions about Visual Studio and how the Visual Studio team uses AI internally to build Visual Studio. Raise of hands, I'm assuming most of you all use Visual Studio in your day-to-day. That's good to hear. My name is Sebastian. I've been a software engineer within Visual Studio for 3 and 1/2 years. So not that long. But as many of you know, things have changed a lot in the last 3 and 1/2 years. I interned at Microsoft three times and when I started back in 2023 I don't think like ChatGPT and agents were in anyone's mind when working in Visual Studio. And heck, Visual Studio has changed a lot in the last 6 months within itself. And so when the opportunity presented itself to chat about how the engineers in Visual Studio are utilizing AI to build VS, I thought this conversation would be a walk in the park. But I was wrong. Mostly because as I started talking to the different engineers in VS I, as someone who just mostly builds features and fixes bugs on the version control space and the code review space as I was interviewing different engineers in VS, I understood the difficulty, the complexity of not just building Visual Studio but shipping Visual Studio to each and every one of y'alls machine. And not only VS 2026, but 2022, 2019, all of the servicing, the security, the compliance, the accessibility that we work on, right? So as I was interviewing engineers in VS to understand how they utilized AI and I expected that to happen. Um I kind of saw that everyone had different workflows, different tasks depending on what they worked on, which is totally normal in a software development company, right? Especially when you have engineers working on the back end for how to ship Visual Studio, engineers who work on compliance, security, right? So, the workflows can be different. And so, as I was interviewing these engineers, I had to find commonalities between each of them to understand, okay, how are engineers in Microsoft and VS utilizing their own tools, leveraging them to build something like Visual Studio, to make sure that we're delivering a quality, secure, reliable, compliant, accessible product to our professional, enterprise customers. And so, as I was looking for the common denominators between engineers, I saw a workflow that kind of matched between everyone. And that's what we're going to talk about today. We're going to talk about the three phases of development that, no matter the task, I saw engineers doing. So, phase number one will be the planning and the implementation phase. Phase number two, we'll talk about the validation phase. And the last phase, which I bet a lot of us are spending a lot of time on, is the code review phase. So, I have you all for a little bit of time today. So, we're going to try to build a feature on top of something that already exists within Visual Studio using this workflow today. So, let's go ahead and jump into VS. Can you all see my Visual Studio, okay? Perfect. So, one of the things that we're going to build on top of today is the ability to see pull requests within the IDE. So, as some of you may or may not know, we have the ability My screen's frozen. >> Okay. My pointer extension's not working properly, so I'll use my mouse today. We have the ability to see pull requests in the IDE today in the Git repository window. So, if you go to Git you go to your hosting provider like GitHub or Azure DevOps and you go to view pull request, the Git repository window right here uh which before used to only have branches and tags, will also have the list of active pull requests. Active being a keyword there for now because right now we only show the pull request in the hosting provider that are active. We don't show completed, we don't show abandoned. That's soon to come. But the thing that we want to build today is that on this filter box right here if for some reason the pull request has been completed or abandoned and you want to search based on an ID, let's say like 1 2 3 4 we want to go to GitHub we want to go to Azure DevOps and we want to pull that pull request and show it on the list itself. So, that's the goal today using those three development workflows. Do we understand what we're building today? Thumbs up, perfect. Okay. Now, before we jump into those phases, I do want to point out something that's been discussed in a few sessions, but I think it's good to point out since it's being utilized now more than ever. Uh and that's the ability to create work trees from the IDE. Something that I saw a lot of the engineers I interviewed was just that they had multiple work trees to work on different work in parallel. So, within the IDE in the Git repository window, we support that now. So, one of the things that I love to do whenever I get a new task, a new bug, a new feature is that right from the IDE, I'll typically go to main and go right click and create a new work tree from here. So, the context menu right at the bottom. And when I click on that you'll be able to create a new branch, put the branch name, um and if you'd like to work uh open the work tree in a current window, you can do that as well. So, I wanted to point that out because there's going to be a few things that engineers constantly mentioned when I was talking to them about how they utilized AI to build VS, and just having worktrees in general was one of the things that constantly came up. Okay. So, the difference is that the worktree will allow you to spin off a different branch, right? And you'll be able to work in parallel. So, you can have your own stage changes, the things that might not be committed or committed, right? And be working on the same repository at the same time, right? But with different um worktrees pointing at different branches, so that you can use them in parallel. Good question. Okay. So, I'm going to go ahead and jump into a different instance of Visual Studio to start going over what we want to build today. And so, we're going to start with the planning and the implementation phase. So, you can see that I have the GitHub Copilot Chat open right now. I want to point out a few things that we're constantly utilizing. Uh there's the different modes that we have, right? We have agent, agent preview, and I'll go over these in more detail in a bit. Uh we have the debugger agent, profiler, but one of the things that came up a lot was that most engineers like to plan. They like to understand that, hey, Copilot is in the same mentality, understanding the same context as the engineers. So, planning mode's important here. Right now, because I don't want to interact with the agent, I'm just going to bypass some of those conversations. And because I want a little more reasoning power for what we want to build today, I'm going to use Cloud Opus 4.6. But, as many of you know, you have the capability to switch models, right? Depending on the task, which will be important because sometimes you don't want something like Opus 4.6. Um if you're just describing maybe a work item or something like that, you could use a lesser more powerful model. Um I also want to call out the ability for tools. Uh tools are extremely important and especially in our internal workflow. Especially like MCPs, right? Uh a lot of those work in Azure DevOps or in GitHub. Azure DevOps has a great MCP to get pull requests, get work items, understand work items, uh and and a lot more. You can get comments from pull requests if you'd like to analyze them within the Copilot chat. Those are the things that are available. And also something very important is the skills. Um right now I'm working on a repository that's an extension of Visual Studio, and we insert a lot of our code into VS. And so I'm you I'm working on the version control code right now. And one of the things that sometimes we need to do is update dependencies. I I know a lot of us need to do that constantly. Before it used to be a very manual process. Uh you know, if Visual Studio has a new API that we want to use, or if they've helped us integrate some feature, we need to update our dependencies. Typically, we'd have an engineer do that very manually on the side, put up a pull request. Now we have a skill, and then of course we'll have an engineer still running through that process, but they can spin the skill with the agent, and the agent will take care of that. So I wanted to call those few things out that can be incorporated in that planning flow. I'm going to go ahead and go to the conversation that I started right before this demo. Um I will be frank with you all. I had a lot of prompts saved for this uh conversation today that we're going to reutilize to build the feature that we're building. I do want to hone in on the planning session though, which is um planning I I know right now AI generates a lot of code for us, but planning is extremely important because the anecdotally what I heard from engineers was that if you have the design discussions, if you have the technical discussions, if you have conversations on Teams, work items, issues, pull requests, right? If you understand the constraints of what you're trying to build, and how you're trying to build it, typically you'll have a better result with these models and with Copilot and the agents, right? Because if you don't, like I've tried it multiple times and I try it every now and then where I get a work item, I get a bug, and I'll give it to Copilot. I'll say, "Do it, right?" But typically it doesn't know how to one-shot that solution, right? Because it doesn't fully understand the breadth and the complexity of some of these large systems that we work on today and that many of you all work on, right? Um as you all know, Visual Studio is 20 plus years old. Um so there's a lot of code there that sometimes we need to analyze and understand to build the things that we want to build today. And so I still think it's a very important that the human factor and aspect is still there through conversations. Because if you're able to gather that context beforehand, understand what you want to build, understand how you want to build it, then you give it to Copilot or the agents, the models, then you'll have typically a better result than if you were just to try to one-shot the solution. Okay, so I've given uh Copilot uh the prompt of of what we want to build today. Um you can see that it's thinking through a few things. Now, you might say, "Hey Sebastian, you can do this in the CLI. You can do this in VS Code." Right, there's plenty of tools to do this, which I agree. I'm not saying there's not. And depending on the task, you know, one tool might be better than the other. But I developing features within Visual Studio, right? With the UI and the code navigation and the diffs that I need to inspect, I like being in the IDE to easily navigate things. So one of the things that you'll be able to notice is that Copilot is gathering the context here. So I can easily navigate to a specific interface, right? Right from the IDE to inspect it to see if it's actually gathering the context that I expect it to gather. So what the those are one of the things that I like from being here in the Copilot chat. Another thing is that um it's not just a full wall of text, right? I can inspect it from the Copilot chat. It's a a easier to read for me. Um and then I like that it breaks things down into steps. You could probably modify this through instructions or skills, but that's one of the things that I like. And then as you can see as when creating any plan, you'll see that it'll create a dot MD file. The copilot channel would typically open that. Go ahead. >> Yeah, but using the plan, how are you using the skills? How are you organizing the skills that are relevant or >> I think it depends on the user story or where you're trying >> Good question. So everybody can hear. >> Oh, yes. Okay. Thank you, Wendy. How are we modifying the skills or utilizing skills when creating the plan? Was the question, correct? Okay. >> And then also I guess just to continue on but are you creating Are you creating like plan or feature release specific skills or these different problems that you're working on? >> Yeah. We're working on a lot of problems at the same time and because all of this is new right now, I'll be frank. It depends on the situation as a whole. So one of the things that we're trying to inspect is like what are we doing repetitively, right? And so as I mentioned, we need to update Visual Studio dependencies. Before I did, you know, when I first started at Microsoft, it used to not be so frequent that we were updating these dependencies. It would happen every 6 months, you know, because we didn't necessarily need all of these new APIs, but now because of the faster development and working on newer features that we want to deliver quick and with quality, we're having to do things faster and faster, right? So one of the things that we noticed was okay, we're having to update Visual Studio dependencies a lot and it it seems burdensome. Nobody loved to do it. It it was a very manual process, right? And so, that's when we decided, okay, a skill could be a good use for that, right? Uh, when it comes to larger complex features, one of the things that we're trying to do is whether it be through skills, instructions, or documents, right? Cuz documentation is still very important, is understanding the architectural decisions into the what, the why, and the how. That's kind of what I'm I'm going to There's going to be a few parts where I'm going to let Copilot do its thing, and we'll have some more of those conversations uh between the audience, because I am interested to understand how you all are when building features, right? Like are we are we still documenting the things that we need to? And the key decisions, the key constraints of systems, so that Copilot can also use that as a reference, right? Because I think the how and the why is very important, and it's one of the things that we're trying to work on, like how can the agents understand that, right? When these conversations are happening at every work work item, pull requests, you know, team discussions, design discussions, and things like that. So, good question. >> With skills, zero bit or dangerous ideas are dangerous really. >> The question was, with skills, how do you know if they're high risk? Um, or like in like maybe like vulnerable? >> Uh, not security-wise, but just like whether they'll work consistently or not. It's a brittle process. Like, how how do you know the skill complexity looking at a skill or while you're building it or while we are using it? >> Good question. So, I know that uh there are best practices for some skills that maybe somehow I can share uh after this. Um, but one of the things that we're navigating, I don't know if we're like I can fully answer that myself since I haven't worked fully in the skill space. Um, but there are some skills that perform better than others. As to why that is is something that we're looking into. >> I can answer a little bit of that. .NET skills repo, um we have uh added a whole a evaluation process where we are determining whether they are providing value and reducing cost over um just using doing the same activity with Copilot themselves. And you can it that is an open source repo, so you can see how we're doing those evaluations and you can implement something very similar for yourself. >> Any more questions? I should probably not step away too much as my machine keeps uh going down. Let's see. Um Get that over here. Cool. Okay, so we see that Copilot created the plan. I gave it uh this prompt right before the session to create um to be able to go to the hosting provider, Azure DevOps or GitHub, and get a pull request. I can read through the plan, I can understand it, um and I'm going to go ahead and just implement the plan. Now, one of notice uh point out a few things at the bottom left corner. Well, we were in plan mode beforehand. And so, I had to a little a bit of context of what we're trying to do. And I just told it, "Hey, go implement the plan that we came up with." And switched over to agent mode or I like to think of it as like auto mode. And so, that's starting to work. Uh one of the things I want to point out here on why I like being in the IDE for something like this is because um sometimes, as you all know, Copilot will generate a lot of code. Uh these models will generate a lot, and sometimes it can be hard to parse. So, being within the IDE itself, um we have the good capability that easily give you the before and after, right? So, I can easily understand step-by-step what changed uh from the diffs that the Copilot chat will give me. So, that's one of the things that I like. Uh we'll let Copilot do its thing. How many are using plan mode within VS or within the CLI or VS code things like that? Okay, that's good. What about agent {slash} autopilot mode? Okay. Do Does anyone have any feedback or think that there are things that are missing in VS for these modes that you'd like for it to be there uh compared to maybe the CLI, VS code? >> I was going to say shortcut keys between Copilot and VS Studio. Also like you uh >> So So the various keywords. So you do a pound sign to pick up a file at VS code and Copilot it's the at sign, I think. >> Yes. >> And we use the at for the agent in VS. I just like those all to be the same. It'd be awesome. >> I agree. Yes, I plus one to that one. We'll relay that feedback cuz I'm constantly switching from the CLI to the V to VS, right? And as you mentioned, I'm clicking pound sign and and at when I shouldn't be. Any more feedback or thoughts? Okay, so it seems like uh Copilot is building. That's one of the things that I like being right from the IDE too. I can kind of see what's going on. It started building uh the solution uh that we're building today. Let's see whether it actually succeeds. We'll let that build a little longer. Do you folks feel like they're switching a lot from cuz you mentioned the the the pound and the hashtag. So is it common for you all to switch from the CLI to VS or VS code to VS and back and forth? Uh what What do you think inspires that? Like how come you're not just in VS? >> Uh the new SQL Server projects uh SSDT projects, they don't work in Visual Studio. >> Okay, that's good. Good feedback. SSDT SQL projects. >> Sadly, my company is way behind on our VS version. They won't let us install past 2022. So, I do most of my work in VS code and then go to VS when I can, basically. >> Visualization better. >> Yeah, in >> I like the visualization, so I really like using the CLI to to do large features and planning. It something feels better about it. I'm excited for the CLI the SDK that they're all going to share. I'm excited for the preview one. But, there's something that feels better about the CLI, but then I still like viewing the code and the differences in Visual Studio. So, I'm excited about the the launch and the switchover capability they talked about earlier. >> Yeah, that's one of the things that I heard as I was interviewing engineers in VS and I think talking to PMs and designers, right? Is something in the CLI, whether it be like through planning or starting to have those initial conversation feels different. Um, I have some thoughts, so maybe after this we can kind of brainstorm a bit to try to give more concrete feedback to our designers and our PMs to understand if we can either make that experience very similar or is it that VS needs to be faster in certain ways? Like, that'd be good to know. >> To me, uh, the old custom Visual Studio harness felt really inferior to VS code harness. I'm hoping 18.8 is going to fix that. >> Thank you for that. >> Uh, Visual Studio is not cross-platform. I've used VS code on Linux and Mac. >> Yes. Anything else? These are great points, uh, and we definitely want to hear this feedback. Okay, uh it said it succeeded. So, let's go ahead and transition over to the second phase, which is a validation phase. So, we planned, we implemented. What I'm going to do now is I'm going to do control F5. And this is going to spin out a different instance of Visual Studio, what we like to call an experimental instance. This is pretty much the binaries that we have from our extensions applied on top of another Visual Studio. So, let's see if Copilot did what we intended it to do. Okay, so I see an instance of VS coming up. That is good to hear. Um Okay, so you can see here um the theme looks a little different. I'm using like a lighter mode um in this VS experimental instance. And so if I launch the folder um of of the GitHub project that I have to test out this feature, there's two ways that you can go into the pull request experience. You can do go directly to the Git repository window. Um I like going through Git, GitHub, and then new pull request. If you had Azure DevOps, it would be the same. Um the the capability here is that it'll just automatically kind of collapse the branches and tags, and it'll expand that pull request section. And so in this validation phase, with so much code being generated by AI, we're spending a lot of time making sure that the layers, right, and that in the back end side of things to make sure that the Visual Studio that we're shipping, right, is of quality. With more integration tests, more unit tests, but also one of the things that we have to do now is so many so much code being generated is manually validating and making sure that the things that we expect to be sound are sound. But of course, the thing as we know with manual validation is that you're not going to catch every edge case. And so, that's why I believe there's more support into making sure that these integration tests, these unit tests uh are more in place uh to make sure that the product that we're delivering is still of the utmost quality. So, as we can see, pull request show, this is good. Nothing's broken in this end. What happens if I type a pull request ID of a pull request that did not exist before. So, if I put 22, you see something's happening, and we see that there's a new section, the completed section. And so, from the initial looks, it seems like if I click on this pull request, there's a few things I would like to improve here uh that Copilot didn't take into account. Um but we can see that the feature's working for the most part. I put in an ID that wasn't there before, and then it appeared in a section of its own. And you can see that this pull request is merged. So, we look through completed and abandoned pull requests all the time in case it caused issues or regressions. Uh it's good to have the context. Now, what happens if I clear that? Okay. So, there's a bug there. You can see that I cleared this pull request ID. Nothing's happening. The old list did not get restored. And so, this is part of the validation phase and the back and forth that we have with Copilot. Sometimes things work, sometimes things don't. And so, how can we leverage the power of Visual Studio to understand, debug, and see what the root cause of the issues are. And so, I'm going to go over to our GitHub Copilot chat over here, and I'm going to create a new session. And during this validation phase, I heard two things that every engineer kept repeating. That was the debugger agent and the profiler agent. Right? The debugger agent will help you understand the state of the world that is currently going on when you're debugging the application to try to understand what's happening. So, the beauty of the copilot agent, and let me switch over to it. And what I'm going to do really quick is I'm going to attach to a process. So, I'm showing you a little bit of the workflow that I have and people have in their day-to-day, which is when you're developing Visual Studio, but you're also debugging and understanding Visual Studio can seem a little bit meta. What we have is we attach to the other Visual Studio, as I have done right here. And so, I will be able to then set breakpoints, understand the call stack, the threading, look at the local information of the properties that are being set, right? Sometimes with race conditions, crashes, that can be difficult, right? Especially if there's going like five threads working at the same time or more, right? And so, the debugger agent helps you diagnose everything together. And so, one of the things I heard over and over from the engineers is that debugger has always been king in Visual Studio, right? And so, having the debugger agent help you start ramping up and hopefully making it faster to understand the root cause of the issue is one of the great capabilities of being in VS and utilizing the debugger agent. So, let me go ahead and grab one of the prompts that I constructed for the debugger agent to see if we can understand what's going on. In the prompt, you can also tell that I'm telling it to set some breakpoints. And so, it'll try to understand the code that I wrote and see what breakpoints are the most relevant that could be the the cause of the issue that's happening. The issue being again that when you clear the filter, the old list is not being restored, right? And we want that to happen. So, raise of hands, do people use or know of the debugger agent? Okay, not many, so this is good that we're exposing this. Um how do you debug right now? Uh is there capabilities in the CLI or VS code, or do you just rely on the old Visual Studio debugger? Would love to hear your thoughts. The old way, okay. So, in my opinion, this is almost like a cheat code uh because it I mean, sometimes it's like if you already know or have a good assumption of what's going on, being in the old like just being in that debugger state is great. But, sometimes you'll get an exception that you don't understand. You're like, "Why did I get this exception, right?" And so, I think even in the exceptions when you get one, there might be like a little Copilot icon that you can click. It'll take you to the debugger agent so that the debugger agent understands, "Okay, what's the exception that's occurring? What's currently this call stack? What's happening in the threads, right?" So that it can pick up context a bit quicker uh for that debugger agent. Okay, um it said that I can leave launch the debugger. I'm already here. It said I've set five breakpoints, which is great and great to see. Um and so, what I'm going to do really quick um before I hit those breakpoints that it set, I'm just going to see if it can solve the solution. Uh can you apply fix. Let's see if it can do that. Do folks feel that with generating AI code nowadays that sometimes it's harder to understand why Copilot did things? And so, when there's a bug, you're kind of like it doesn't seem at all related to to what we were working on. I'm getting a few yeses. Okay, that's good. Well, actually not good because you want to understand what's going on. Um that is one of the things that we see every now and then. Um it says this fix at a check when the filter is cleared. And it calls refresh check access. Um, I I know already what refresh check access actually does in the background, and I already know that this is incorrect. It'll solve the issue, right? But it's actually performing a lot of unnecessary work. So, one of the things that I maybe would have liked to see is um actually caching the list for a bit and keeping it, right? But refresh check access will actually hit the server, uh which we probably don't need to do in this instance. But I'll I'll kind of chat a little bit of in a bit about how we could improve that. But let's go ahead and validate as part of that second phase again, right? So, I'm jumping to the experimental instance of Visual Studio. I'm just going to start kind of like a clean session. Oh, actually, before I do that, um as something that I know is loved in Visual Studio is hot reload. So, hot reload in our workflow applies those new changes, the new capabilities of the binaries that we just modified onto that experimental instance of Visual Studio. So, if I click hot reload, no issues occurred. That's good. Um, so I'll jump over to the experimental instance. I'll go to Git, GitHub, new pull request. You can see there that the debugger agent actually set those breakpoints. So, that's good to see. Uh what I'm going to do really quick is I'm just going to um undo those for now. And I'm going to continue. It's retrieving the pull request. That's great. Um, these are all active. If I put in 22, we can see that that appears. So, that already worked beforehand. What happens if I clear it now? Boom. You You see that there's a loading bar and then it fixed the issue. Again, didn't fix in the best way because that loading bar means we're we're going to Azure DevOps or GitHub and making another web request when we could have easily just catch cached the list that we had. Again, this is part of the validation phase. Copilot and these models are not going to be perfect in the first time. Now, one of the other agents that I constantly heard in conversation was the profiler agent. And so, if I switch over to profiler agent, the profiler agent is able to analyze the performance and try to optimize the code that you've written or that's already been written, right? So, it already has some context of things that can be optimized. So, right now I'm in a solution and I'm going to tell it, "Hey, can you improve this or can you see if there's things that could be better in terms of performance?" So, let me go ahead and grab that prompt. Okay. Put that right in. Okay, with the raise of hands, who has heard of the profiler agent? Okay, so wow, a lot more people have heard of the profiler agent than the debug agent. Um well, that's good. That's good that you guys have heard of it. I'm constantly using it, right? To optimize code to see if there's things that could be made better. Uh one of the things that of course improved a lot in Visual Studio 2026 is the performance of it. So, hopefully whenever folks are able to switch to 2026, you'll see that uh things are a lot quicker uh in Visual Studio. And so, our profiler agent is thinking. And it'll go ahead and analyze the code. Yes. Uh that is the whole goal of the So, the the question was uh if you want to perform it improve performance or the scalability with a profile agent help with that. And the answer is yes. It'll help you create benchmarks as well, or if you already have benchmarks, it'll analyze those. Uh and it'll also try to just optimize the code to be more performant and scale as well. Any other questions? >> You're sharing a lot of the technical steps that you guys use on your team to go through this. >> Yeah. >> Can you also share some of the process changes that your team uses? For instance, where do you require human in the middle? Um yeah, how much do you trust is at what point you're having models review model code written by models, stuff like that? How did How did you guys process-wise adapt? >> Uh we are adapting. Uh I think that is an ongoing process. Uh we're we're having constant conversations now more than ever about like like you said, um how do we make sure that the human still part of every part of the process, right? Because when you're building products that are large for enterprise customers and professionals, you don't want to deliver what they call now nowadays AI slop, right? Um we have millions of users that are depending their job and their every day on us, right? And so, it's still crucial. And you kind of beating me a little bit into the last phase of development, which is the code review phase. Which we're spending more time than ever because there's so much code being generated. Um whether that all that new code being generated is is um is fully perfect to the standards that we want it to, that's the things that we're trying to analyze. How can we make sure that it is? Um and how can we make sure that people can view reviews flow kind of improve so that we can make this process easier. Because in a way we're spending more time reviewing now than we were actually writing code in our day-to-day. Um so it is important to still have So I think it is it is an ongoing conversation that people are constantly having. Um you know you can see companies around the world that that things go out or they have issues, right? Because an agent did something or code did something, right? And we want to make sure we prevent all those things. Um okay, so I think the profiler agent did some work. I don't want to stop debugging right now. Um Okay. So yeah, it it found a lot of things that could be improved that I've been seeing. So the first one is uh there's a method that's still doing a filter evaluation uh twice. And maybe we don't need to do that. Um I'm going to say don't stop debugging at the moment. Um the search is O of N with nested loops. We could definitely improve that. I wanted to see if Yeah, it didn't call out um at the moment that um that refresh was unnecessary. Um I've seen it sometimes it does call it out. Um so that's something that one could improve through instructions or giving it more content or or comment in the code of what that method actually does. So I'm going to stop debugging now. Uh we've gone over that that validation phase, right? We've gone over how to use the debug agent, the profiler agent to be able to analyze, validate the code. Something that's still very important though is unit test, is integration test, right? To make sure that there are a lot of guardrails so that nothing can occur. Uh go ahead. >> Do you need to have a hundred holes? >> I was what? >> Outrage and and in your development when it focuses continue at the aspect >> Yeah, we do I think it depends on the repository that you work at. Let me repeat the question. Do we need 100% coverage? Thank you all for calling that out. Do we need 100% coverage to all the code that that is writing? I think it depends a lot on where you like what part of the feature that you're working on uh since we already had a lot of code coverage beforehand. Um personal experience anecdotally sometimes I've seen Copilot generate very useless unit tests. And so I might write 10 unit tests. But maybe two or three of those were actually really useful. And so I think that's one of the things one of the cause that we've been making talking about is like, "Hey, do we need all these unit tests? Or do we actually want the ones that are very important and things that we want to clear cuz sometimes it'll say like um you know, if we we will have a guard for saying the argument cannot be null, right? We have a helper method that checks for sure and it'll throw an exception. Some people like writing unit tests for that case, some don't. Um sometimes in depending on the team and all of that could be you know, developer dependent. Sometimes it just really does generate useless unit test. Um that's something that we're trying to fix as well. Um but yeah, hopefully that answered your question. >> Uh what is the How do you guys use the back and forth between like the work item process of creation of work items and things that come out of this. So for example uh in in our team like there's other humans who like write work items and things like that right like specs, designs. And then some things come out of this also. So is there a way to during the planning phase just like automatically generate the plans from some of the work items and are Uh and also the other way around like from these things say you don't want to you don't want to do all these things right now. You just want to like merge it in right now and do these things later. So just create work items for later from the results of this. So back and forth integration. >> Anything concrete? No. Uh not that I can think of. Uh the PMs here who work at Microsoft, please let me know if I'm incorrect. Um I was just was funny enough I was thinking about this this morning in Ramona's and Leah's talk for some reason where um I was thinking about one of the things I do a lot more now than before is constantly prototyping, right? But prototyping now is easy and you can spin up something that half works pretty quick, but then actually getting from the 80 to 100% is still very difficult in in certain areas. And so I feel like I'm get I'm getting a lot of these prototypes and ideas that I've been building, but like you said, there might not be a work item. I've I've now had all this planned. I've I've you know evaluated different solutions for it, but I'm not ready to actually dive deep into it, have discussions with the team. But how do I save that context? And how do I save all of this? Um great question. It's good feedback to have. >> So we do have integration with the the MCP servers with ADO or GitHub depending on where your repos are. You can have Copilot create those issues with all of the context in the plan that you had created. And so that's there for you to go and work from in the future. I also if we have the meetings and all that stuff is in place before, I use Work IQ um MCP server as well as ADO or GitHub to gather all the contacts from all the meeting transcripts and everything that we've done to talk about to help create the plans from the conversations that we had so I don't have to remember everything that we talked about. Work IQ brings in all of that context from those recorded Teams meetings that we had and and helps to create those plans and then it can create issues from those if we don't already have them and all of that. It's the the power of these MCP servers are pretty impressive. >> Yeah. Oh. >> So, yeah, if you have M365 Copilot, um there's a Work IQ um is an MCP server that connects to all of your Microsoft intelligence. So, your your exchange, your um uh your your Teams, all of all of that is is whatever you make available is is available for it to to search through and gather. >> I do want to note that the I've been looking at it this morning, the Azure DevOps MCP server is not available for on-premise. >> Good feedback. We definitely Yeah, need to relay that. Um Yes. Uh thank you, team, for being here. Um that's something that, you know, with so many tools and and so many new things, it's hard to like keep everything in the wraps with everything happening and changing so quickly. Um but yes, one of the things that I will be trying out through these little tools is that we do have the ADO MCP server there and so you can do things such as search work items, create work items, right? Um and I would assume that get the uh the Copilot chat can grab the context from the conversations and put that in the work item as well if you'd like to. Okay. So, we've done planning, we've done validation. We're ready for code review phase. Now, as I mentioned earlier, we're spending a lot of time reviewing code. And one of the things that has been discussed in previous sessions as well has been this concept now that there's two parts of the review. And there's that local review before you actually push to GitHub or Azure DevOps and you get your team involved. Right, so that's a local review before you actually commit and push everything. And there's that more like remote review or the code review that happens with your team on site as well. So, one of the things that you can leverage Visual Studio to do this is through a few ways. You can in the Copilot chat, we have the Git agent capability here. And you can already see on the right-hand side that it has the ability to review the current changes. And so, if I go ahead and kick that off, you'll see that it not only will leave comments in the chat about what can be improved or not, but it'll actually spin up spin up kind of like an editor integration where the comments will be in the code itself that you wrote with Copilot. And so, it'll give you suggestions on what could have been improved before you actually go ahead and push that to the hosting provider. Now, I do want to call out that this functionality also exists within the Git Changes window. So, in the Git Changes window, you'll see this button right over here that you can review your changes with Copilot. So, if you're not in the chat, you don't just only have it capability in the chat, but you can do this in Git Changes as well. And you'll see that they kind of interact together once it finishes reviewing. Uh with a show of hands, do people know about the Git agent? Oh, wow. I'm not I don't know if I'm seeing a single one. >> Why did you use the name Git agent instead of QA agent? >> Or yeah, it almost feels more like a code review agent than a Git agent. >> Good question. Um because um one of the things that Lea demoed earlier today is like merge conflicts, right? And so, this Git agent will slowly start getting more and more integrated with a lot of those version control capabilities as well. Um and in a way version control is very tied to to the current code that you have in your work directory. And so, being able to review the current code that you've been working on, uh Git has a lot of capabilities and understanding of things that might be stashed, unstashed, right? A a plethora of things. Um and so, what you can see here is that Copilot has left comments within the IDE. Um there's things that can definitely be improved with cancellation tokens, uh guards, things like that. And so, if you wanted to, you could go ahead and also generate a suggestion from Copilot, or you can just review that within yourself. Um we're running a little bit of out of time. Uh thank you for all for being very engaging. So, there's one thing that sometimes takes a little more time that I want to demonstrate, but I won't fully uh get Copilot to do just yet. Uh I do want to show the um agent preview, right? Which was mentioned a little bit earlier. And that was about um utilizing the GitHub Copilot uh CLI SDK. It's a mouthful. Um and one of the things that I I do in the CLI, but also do in VS a lot, is being able to order the commits properly. Like breaking things down into easy to easy to tell and parse commits. Because as we spoke earlier, there's a lot of code being generated nowadays, and Copilot loves to generate code. And so, one of the important things is breaking that down into a meaningful story. And to understanding the what, the how, and the why. And so, one of the prompts that I like to give Copilot is, "Hey, can you break this down into several commits?" Something that makes sense so that not only is my job easier to understand and parse that history, but um so that my teammates can understand, "Hey, if they're reviewing a larger pull request, and it feels as of recently in pull requests are getting larger and larger, Being able to review commit by commit will be a little easier. And so I'll share a little prompt that I've just created for today. Which is, "Hey, with the current changes in my Git directory, can you break this down, right, into a way that makes sense?" And then um one of the things that I would like to see improved, honestly, maybe within the Git agent, is maybe making this like uh experience a little uh better because uh with the Copilot CLI SDK, I see that it'll do a lot of commands that I feel like are unnecessary on the Git side. Um but it'll still do it properly based on on what you tell it. And so one of the things that I'll show, once that's complete, uh that one can do, is that if you go and I'm switching over to another repository really quick, um if you go over, let's say, to your Git history, imagine that you had those commits ready to get pushed and you forgot to review them. Or imagine that you're trying to review a commit that maybe created a bug and you think might have been the culprit of something that occurred. One of the things that you can do within this uh repository window and within the history view is that you can also review that commit. And so I just right-clicked on the commit itself. I review the commit. And then you'll see that we're reviewing changes. This is kind of like the same architecture, the same system that's happening with your local changes, right? But now it's happening for the commits themselves. And so if you wanted that functionality there to try to understand commits a little better, you can do that as part of your workflow. Um again, this is all about that local review um flow that we have. Now yeah, you can review commits that have already been pushed. Uh it seems like Copilot didn't generate any comments here. Uh but typically it'll generate comments with suggestions or things that could be improved or even just explaining a lot of the code. So, that's the local review flow. So, within the IDE, we've also added some capabilities to review pull requests within the IDE. So, imagine you've gone ahead and created a pull request, which we do have the functionality for, so I'll show that really quick. Um if you go to Azure DevOps and you go to new pull request, um if I were to have committed these changes, you'll see that um you can enter the title, you could generate it a a Copilot AI description if you want to hear. Um I haven't committed anything, so there's no diff. Uh but, that's also available there. And then, when you automatically create a pull request, this works on both Azure DevOps and GitHub, you'll be taken to the review experience within the IDE. So, let me switch over to my TypeTrip repo. And as we mentioned and we worked on earlier, you have the list of pull requests. Right now, we have the active PRs there that you could see. Uh you can open a pull request here. Now, I do want to uh say that this view might look a little different for you. We're currently experimenting with that capability of making reviews easier. Uh as we know, there's more pull requests nowadays, there's more code being generated, and because we're spending so much time actually reviewing code, we're trying to see if there's ways we can make that better with the integration of AI. So, this view might look a little different for you, but the functionality is already there. You can approve, you can complete, you can merge things, you can understand the status, the comments. Um you can navigate files, all within the IDE. So, let me go ahead and I've already popped out a pull request, so I'll show you how to do that. Um over here, you have a few commands. You can open this pull request in the browser if you would like. Um or you can put it into a larger screen. Um and so, from here, you can look at your changes. Uh you have the multi-file summary diff capabilities. And then or or if you just want a regular diff, you can do that as well. Uh, but one of the things that we're experimenting with uh, is not available to the public yet, but we're dogfooding internally is the ability to generate groups. What does this mean? Well, it means that once you press that, uh, Copilot will try to understand like the files, the diffs, the hunks that are related, right? Um, imagine that things weren't broken down into several commits. Imagine that everything was just one large commit, 70 files modified, and you're like, "How do I even begin to review this?" Uh, definitely happens quite a bit nowadays. And so, you'll be see you'll see in the left-hand side that Copilot try to broke it down into groups that it thinks are related. And so, then you can see, you know, if a change might be of of high risk, of low risk, and it'll give you a bit of a description on what those changes are currently doing. Again, currently internal, but something that we're experimenting with to try to understand um, if we can make developers' life easier within the IDE to review certain things. Okay, we've talked about a lot today. Uh, we talked about the current workflow that a lot of the engineers in Visual Studio are having, which are planning and implementation with Copilot Chat, the validation with the debugger, the profiler agent, and then the code review, right? The local code review functionalities to understand what your code is doing, if it can be improved, and then the ability to review pull request all from within the IDE. Now, if you needed a recap of everything we talked about, I know it's a little small cuz I needed to fit all in one side, but these are a bit of the things that we've discussed um, in in terms of like the workflow that the engineers have had today. Now, there's There's few things that I want to point out. This workflow doesn't have to be linear, right? You don't have to plan and then validate and then do a code review. You can hop hop into any of these features in an independent fashion, in a dynamic way, because you don't have to follow this 100% since every task might be a little different. And then one of the things that I also want to point out is that if I learned anything through interviewing a handful of engineers in Visual Studio that worked on a handful of different things is that human knowledge is still very key. Uh there, you know, these systems can be very complex. The architectures can be very complex in themselves. And so making sure that discussions are still being had, right? With designers, with PMs, with engineers to understand the constraints of the system, right? To gain knowledge and expertise on the things that we need to build, we still see as a very important part of the workflow. Because at the end of the day, we don't just want to spend all day just talking to Copilot, not talking to a single human being, and then shipping something, because that might not be of quality. And so we still need to put our customers first into what we're building, and we want to make sure that again we're delivering reliable, maintainable, secure, quality, compliant, accessible, so many things go into building enterprise software. Um anyways, we have about 7 minutes if folks have any questions. Thank you for being and interacting today. See the little one right over there? >> Yeah. Um do you have any data on like how much you think AI has sped up the Visual Studio team, your local team, yourself? Any data like that would be helpful. >> As a software engineer myself, I do not uh know or have any metrics to have under can understand whether it's been me uh more productive. So, I couldn't tell you. I don't have that data. Um what I can tell you though is that there are certain parts of my workflow that are definitely more productive. And I'll give you some examples. Like I said, prototyping. Before, you know, being here 3 and 1/2 years, prototyping me would typically take me a week or two. All right, understanding the system, understanding what I need to do, uh just spinning up something that you know, sort of works. Um that's just data that I had and calculated manually myself. It's one of the conversations that sometimes we have with our managers. It's like, how long do you think there's a prototype? How long will it take you to actually get it into production? Um now the prototyping I think it might depend on engineer, but for me, prototyping can take me about a full day or two. Uh because Copilot will generate a lot of that code. Again, that's a prototype, right? It go it got from zero to 80 very quickly, but that 80 to 20 is still very important. And it's not just that 80 to 20 about the code and the complexity. It's like did the prototype I built actually help customers? Right? Uh does it have good impact? Uh does it maintain the architectures and and respect to the ideologies that we have within our repositories? Those are then conversations that at least someone who's been here 3 and 1/2 years I still need to have with senior engineers, with principal engineers who have a lot more experience. Um I still see those as very crucial. Um and so, but I think again, it might depend on on what you're working on, how fast something might need to be delivered, things like that. Um so, unfortunately, I can't give you a concrete answer at the moment. Any other questions, thoughts, comments? What's the ratio between the coding and the reviewing. Um At this stage right now, and I I and I say this because we recently had some some new hires coming in, right? Uh you know, not long ago I was actually like the the latest new hire within certain features that I work on. Uh I'm spending a lot of more time reviewing code. Uh the exact ratio I couldn't tell you. I would say probably like a 60/40 70/30 where a large majority is reviewing. And not only just reviewing other people's code, but also reviewing Copilot's code. Um I would say now I spend more time actually sitting down front cuz I think it's still very important to to understand what you're trying to build and the code that Copilot wrote. Um so I still spend a lot of time just sitting down and what I'll typically do is I'll try to break down the code that Copilot wrote. So that could be into commits. Uh that could be into different files. And then I'll just go through the diff. I'll say what what's happening before, what's happening after. Is this being overcomplicated? Is there a better solution? Are there better suggestions? Um And so I feel like I'm reviewing the code that I've also collaborated with with AI um more than like actually handwriting a lot of code. Um I still especially if something's a lot more complex, I I still like to use uh paper and pencil a lot for like when I'm in meetings and stuff. And so when something's very complex and I'm working on it with Copilot, I call me old school, but I sometimes like to type it out it myself because maybe that'll help me understand a little bit of, hey, what what was Copilot actually trying to do? Why did it do it this way? Uh those are things that, again, still figuring out myself if that makes sense. Yeah. Any other thoughts, comments, concerns? Cool. Thank you for being here today. Uh hopefully you were able to gain something and I'll stay around for a little bit if you want to have a chat about anything else related to Visual Studio. Thank you all.