The practical guide to reverse engineering XXL codebases with agentic AI

All right, so you almost gave away the problem that I'm going to discuss. So I got like 2-3 minutes extra. >> [gasps] >> So I'm going to share a story. Like in March of this year, we actually got our hands on a legacy code base. We reverse engineered it using Claude and I'll be sharing some of those learnings with you guys here. All right, so let's get into the problem. I don't know it's not working. All right, so let's let's get into the problem. So you want to reverse engineer a large legacy application. Typically it contains like a 10,000 plus files. Uh it's around a decade or more than a decade old. Generations of developers have worked on it, right? So you don't have a really good documentation that is current stored somewhere that you can refer to. And probably you have been asked to maybe refactor it, you know, or kind of rewrite it, estimate how much time it will take to rewrite it, or you you are being just asked like can you try using can you try moving faster through this using AI? Use Claude code or whatever your favorite AI assistant is, right? And that's the whole problem. We want to build understanding about this code. Cool. So with AI it's pretty easy. We can just write a prompt. Something like this, okay? I wanted I kind of compressed it to fit on this slide, but you don't need to read everything on it. Just read the highlighted part. So, simple. We are trying to focus on the domain model. We want to extract out the architecture. We want to look at the integration map. We want to look at the execution path, domain decomposition, service boundaries. We want to know everything about it, right? We want to create uh a good knowledge base that we can use in future. A knowledge base that we can use to maybe estimate if how much time it will take to rewrite, what part we should focus on while refactoring, and so on. Right? So, what do you think? If I run this prompt, what will happen on that 10,000 plus files repository? Well, we gave a try. We created the prompt. We ran it. And we found that Claude started that agent started reading all of the code line by line, file by file. And suddenly, compaction occurred. Then the In the output side, whatever it produced, we started feeling that it's inventing new classes. These classes are not there in the code base at all. Right? Because that's what happens when compaction occurs. Like some of your uh tool called details are missed, right? They are removed. And that's where But Claude doesn't stop there. It starts inventing new things there. And with very high confidence, but zero accuracy. That's where you basically lose the ground, right? That Cool. So, how to fix it? If the compaction is happening because of our content is flowing out of the context window, let's just fix it by adding a larger context window, maybe. Will it work? Think about it. No, you will hit the wall. So, it's So, it's not like a problem where that you can solve just by adding more and more like a larger context in those. It's similar to like saying that I will increase the RAM of my hardware if my data set is larger than the main memory. No, you will definitely hit a limit on that one. So, we need to think differently [clears throat] here. You know, we have to look at this problem and first of all, we have to like while solving, we have to start acknowledging the absolute limits of our processing container. The context is not infinite. It's a finite container. Right? The LLM is a completely stateless. Your agent maintains the entire conversation at context, adds all the request response to it, and that's how basically your LLM gives you the response, right? It reasons based on the content of the context. And when you overfill it, it will start forgetting and you will start seeing all the hallucinations. All right. So, the problem here is like our context is getting overfilled. So, what could be the reason? Obviously, it's code, right? We are putting million lines of code into it. So, what if we say that if code is causing this whole problem, what if we don't allow agent to read the code at all? You must be thinking like this guy is crazy. Yeah, this sounds very crazy. But, uh think about it, why your agent is reading all that code? Uh because we want architecture, right? We want integration maps. But, do you think you require to read the code? To reason about the architecture, you want to know the structure of the code, right? You don't want to read all the code line by line because you're not just reading the code. You're also reading like you're reading all of the import statements. You're reading even the copyright comments and all of the dead code and whatnot, right? While doing this. So, the most important takeaway here is like your LLM is a thinking machine, right? To take out the structure from a code is not a thinking problem. Extracting the structure is not thinking problem. It's a mechanical problem. Your compilers have solved it decades ago. You need a parser. Need us some way to query that that structure. Query that class name, symbol index, whatever from your code, right? And that is LSP. So, we can basically use some of the deterministic tool to get better grounded response, right? So, here the important thing is we want to partition based on the type of tasks that we have. So, we started thinking about it. We created two buckets. One is a cognitive bucket that requires thinking. For example, reasoning about the architecture, enriching the thing, doing the boundary analysis, synthesizing the final outcome. Then you have deterministic task, very mechanical one. I want to parse up the code, get the AST out of it. Maybe I want to create an index of all of the symbols that are present in my code, right? I want to find out the dependency between the thing. For all of these tools are available. We can use different tools. Right? The key insight over here that we want to apply is we need to exactly identify which tool should be used for which task. If you are using LLM for the mechanical task, that means we are using most expensive and a fragile resource that we got for very mechanical and the cheapest task in our list. Right? So, once this bifurcation is done, we can start the extraction job. So, here again, we want to ensure that none of the agent overfills his context. The context is always relevant. So, we are processing it like a pipeline in multiple stages, right? Each stage takes output of the previous stage and enriches it. Right? So, let's look at each of the stage one by one. All right. So, very first stage, layer zero. Okay? We We call that agent a scout. So, the job of the scout is to explore the code. Okay? Explore the code, identify the high-level organization, your module packages. The code base that we were working was a spring code base, not Spring Boot, by the way. Spring, okay? Like before Spring Boot. And it was containing all the Maven uh pom.xml files. So, this scout is supposed to read only build files, nothing else. It's not allowed to read any of the Java files. Okay? So, what it produces? It produces a module index. So, it will read the outcome of some of the build build tools. It will look at the build files. It will build a module registry, a module index, then it will also identify the dependencies between the module. It can capture some of the important uh important statistics of the module. Like what type of module it is. Might be a preliminary guess, okay? It might tell you like how many files are there, what type of files are there, and so on. So, it only gives you a very high-level summary around the organization of your code. Right? Once that summary is present, without reading a single line of code from your 10,000 10,000 files, we can start mapping the things, right? Creating a symbol index. All right. So, the next agent, actually a swarm of agents, like you can have multiple cartographers which are running on your entire code base, right? And they will give you uh a complete index of all the symbols. For example, classes, methods. For classes, you can find out data like the annotations they contain, in the file path. You can have like on the method side, you can have uh different annotations which are on them, the the incoming call, outgoing call, and all of that without reading the code. Here, we made you use of LSP. So, LSP tool calls actually gave us all of the information. So, we can extract out all of this from your typical Eclipse project or whatever. We use JDTLS for this particular thing. LSP is a language server protocol. So, in Visual Studio Code actually brought that concept in, where if the IDE wants to learn a language, wants to interact through a language, that protocol will allow it. For example, go to definition in uh you want to Whatever you you right click on particular thing, you go to a definition of the symbol. Call hierarchy, everything. So, that is a service. That's a deterministic style service that is available within your ID. And Cloud can latch to it through an MCP. Right? So, because of that, without actually reading the code, we are able to find out like most of the structure of this code. So, we can identify static relationship between the files and modules and everything. So, it basically gives you a really good knowledge graph. So, we used structured output for these like JSON and CSV files for creating the index and some of the markdown output for humans to read. All right. So, we have now mapped out our entire code base. Cool. Once the mapping is done, now we can perform higher level operations on top of it. Now, the next stage contains multiple agents. And these are like specialist agents. They have very specific goal, and they process the different info like the information that is captured, plus they can they are allowed to read only some part of the code. Because here the thing is different. They are not sweeping through the code, right? They know what part of the code they want to read. For example, you have a domain archaeologist. So, the job of the archaeologist is to excavate the domain model that is like buried inside your code. Right? So, it will try to look for all the classes maybe which are annotated with entity. Right? It will look at How it will find that classes? You already have the symbol index. From there, it doesn't need to sweep the classes. It knows the class. It can directly go there. Same, you have taxonomist. It's doing the similar thing. It's trying to identify your services, entities, gateways, and all other different concepts or the different layers of your application. We also added a special agent because this was a spring application, but not Spring Boot. So, it's a beanstalk Jack, which was going to use the spring configuration files for identifying the beans and how they are dependent on each other so that entire graph was was extracted using the beanstalk Jack. So, all of these specialists, right? They use the symbol index that we created in the previous layer, right? And they are allowed to read only the like the surgical they are allowed to read the code surgically. They exactly know what they want to read. They never read like entire entire modules and all, right? So, like that, what you get is now entity relation. You get the entity table mapping. You get your complete domain model documentation, right? You get your object taxonomy, different layers, right? And even the spring beans and all. And here also like since we could not find a way to actually do like the even that that bean job could have been done in a deterministic way if this was a Spring Boot. I could dump the entire application context, fire it up, and get the all of the beans and everything. But here we chose to use LLM for it. Maybe there could have been a better strategy. Awesome. So, what we got is like your complete domain model, right? Once that is there, it's time to survey it further and draw boundaries around it, right? So, the next set of agents we unleashed were like was a port mapper. So, the job of the port mapper is to identify inputs and outputs of your code. Basically, it's giving me the integration map. All the APIs, all the external calls, it's able to find out. How? Because it knows like it's going to look for the rest controllers, right? It's going to look for the uh specific classes like a rest template or your outgoing HTTP clients. We had to like configure some of these things, but it's able to find out all of these entries like Kafka template and everything. Then we had the surveyor. Now, surveyor is able to look at the concept taxonomy. It's able to look at the symbol index, a domain model, and based on that, it's identifying the service boundaries. The code base that we inherited, like it did not uh contain clear separation between the different uh service boundaries. For example, customer and cart, all of these or the product was not really like separated out. But this so this surveyor actually helped us to come up with the functional domains which were present inside that application. So we could maybe in future decompose it into microservices, right? Then came in the choreographer. Now, this is tracking all the transactional boundaries. It's looking for the uh workflows that are being executed through this, right? So like that, if you again look at this one, this these are able to read some of the code, but they are mostly relied upon the output that previous stage produced, right? So step by step, we are getting richer and richer output. Once we have like done all of the surveying, we have laid down the boundaries, we have decomposed the entire uh code base or the uh code base into the uh service boundaries, it's time to put everything together. So this is like the final agent that comes into the play, right? And all of this This orchestrated using your cloud and explorer agent and sub agents that whatever we do, right? So now the chronicler it doesn't need to read the code at all. It's just looking at the outputs of the previous stages and it's putting together the final knowledge base which contains architecture, domain glossary, your final domain model, integration map. Even we created cloud.md from this agent only. Like we also got our because when we tried to do cloud in it, you know, the MD file that got was was not really like a I would say a good MD cloud context MD file. You want but afterwards when we created a module wise context MD file it actually contained very useful information, something that we could use. Even we created onboarding guides for cloud for example, I want to add a new restaurant point, where I should go, right? And all of that. Even your service topology and everything. So it created a a pretty rich knowledge base that could help us to uh actually think about like what part we should address during refactoring, what and if you want to rewrite, okay, what would be the estimate and so on. So let's uh we we saw all the five layers of this pipeline, right? Now let's take a look at the complete pipeline. So it's simple, right? You have scout, cartographer, domain then specialist, then surveyors and chronicler. Right? Each stage, right? is works on very with very specific goal. It has very specific instruction. It produces some output. That output is immutable, right? No stage is allowed to go and change the output of the previous stage. So like that each stage works on a very specific contract. So basically each stage works in isolation. All of the agents work in isolation. They are not They are not aware of other things. They are only aware of the contract and they look at that particular folder, read the things based on that contract. Cool. So, this is your like complete pipeline. You might be wondering like who watches the workers if you have so many workers, right? Some of them may fail, right? Some of the One of the cartographer just crashes or something happens. Like there is a network interruption or something. Who will Who will look after the errors, right? Who will identify like uh what should run in parallel? How many cartographers I should start? So, the answer to this is one final agent. So, that's our conductor which orchestrates this entire pipeline. So, basically, when I say I want to run this pipeline, okay, I just spin up the conductor. Now, the conductor is holding the pipeline state. So, it knows at what point point it can uh at what point the current pipeline state is and what agent should be spin up. So, you get basically a resumable pipeline through this. Right? Moreover, moreover, like uh it is uh it's not Though it's like orchestrating everything, it's not actually executing anything. Everything is executed by your workers, right? So, this is our classic uh hubs and spokes uh hub and spoke architecture where your hub is the pipeline state. It's central and spokes are the different agents which basically work in isolation. All right. So, that gives you like the complete orchestration pipeline that when you run, it will produce the final knowledge base. Now, let's take a look at what this pipeline actually produced. So, when we ran this, okay, we got a reusable index, something that we can query again and again. So, we were able to map all 10,000 plus classes and methods uh in that knowledge base. Then, the documentation artifact that we got was like 200 plus different documents, like uh some of them were uh you can say lower-level documents, but on the top like we got uh uh all of the knowledge base that we are talking about, the architecture diagrams and everything. We were able to map the complete data layer. Like, there were 400 plus database tables. Then, the API catalog that contained 80 plus REST endpoints and 60 plus SOAP endpoints, which were identified through this. We found 15 plus service boundaries, like clearly identified, reasoned about, documented, and we can now decompose them later when when we plan that work. We created a workflow library where we tracked 80 plus workflow traces with the sequence diagrams. Finally, the architecture and the most important thing is this knowledge base is a living database. As I said, your pipeline is resumable, right? It is also keeping track of the fingerprints of the file it processed. So, for every delta, I can just run this pipeline, and for that delta, only the part of the knowledge base is recreated. Right? Cool. So, getting this output, you know, it wasn't that easy. Like, there were some hard lessons that we learned while building this. So, first thing, validate early. If you hand over 200 documents to a development team and say like, "Hey, validate this. They're not going to give you a a good feedback on that one. It's very hard to review everything. So what we did was like step-by-step each stage output was actually validated by the developers who were actually working on that particular thing. We showed output to the product owners and all and we got every stage properly tuned and validated. So validation needs to be very early in this step. Okay. Evolve the tools. Now I talked about LSP, right? But LSP is a service. Calling LSP was actually it it took a lot of time. Imagine like there are 10,000 files you're calling LSP on each of the file and each LSP call takes somewhere around 50 milliseconds to 100 milliseconds. So that made our pipeline very very slow. And that was our mistake. We were thinking like like in IDE I use LSP, why can't I use it in this pipeline? Well, you can't because in IDE you work on a single file, right? Here you're working on 10,000 files. So you can't use LSP over there. You can't rely on the service. You can't waste like calls to a waste time in calling the other service. So we replaced it with a tree-sitter parser that can produce AST that can produce the same symbol index. But this time the tree-sitter parser only took 10 seconds. Like we used multiple instances of that Python process. But yeah, in 10 seconds we were able to like create like index the entire code base. We got the exact output on which the remaining stages can work. And that's where the third point came in. Since we had very structured contracts between all of the stages, when we replace the core of the cartographer with completely different logic, right? The specialist layer did not get affected at all. We just evolved the part of the code and we got the cartographer the specialist layer was still working. Finally, respect the economics. This is all about uh the question is around uh which tool to use, you know? Use LLM for for reasoning, not for structure. Similarly, when I should use this? Uh with this when we tried using this knowledge base for a different purposes. Like we we have used it for creating uh blast radius check. When we are making a code change, we can infer the blast radius very easily from here because we have the call graph. So, from that we can infer the blast radius. But, when we tried to use the same to replace the typical grep and uh grep and glob tools of the cloud, we found that it's actually running slower as compared to the the typical grep and glob. So, we need to measure the things. We need to respect the economics. All right. So, coming towards the end, right? So, here I just want to share the the the core principles behind this entire solution, right? We started off by acknowledging that the context window is a finite container, right? We cannot design around it. We cannot circumvent it. We have to design within operate within those constraints. We partitioned the entire pipeline all the tasks into different stages. We even created specialized agent competent to execute very specific goals. So, that's where we achieved our role isolation. Partitioned all of the tasks into different buckets, right? Finally, we had special agents which synthesize the outcome of all of these agents into a single cohesive output, right? We had a conductor which orchestrated everything. So, this when you compare it with the previous a single prompt that we tried like this this design sounds pretty elaborate. And the only thing, you know, that that pushed us towards this that like we paved our path through this code because of the only thing which was the constraint that is this context window is finite, right? We have to We have to maintain it. We have to keep it relevant, right? So, that constraint actually helped us like was not our enemy, but it helped us to come up with this final pipeline that produced the knowledge base that we got. All right. That's it from my side. Let me know if you have any questions. Yes, sir. Yeah. Okay. So, here uh basically on the when we are doing static analysis, it's pretty hard to get what code is dead. For that, if you are able to And also one of the thing was like the tricky part for about this code was like it contained lot of feature toggles because this was a highly customizable application. For different clients, we were actually flipping some of the switches and distributing it. So, identifying dead code is it's it's pretty difficult. Okay. So, we kind of started off with like creating a library of the feature toggles. And based on that, we considered like, "Okay, if this feature toggle is on or off or or some of the code like we since we had the call graph, we could now identify the code like uh I'm talking about the agent that I never discussed, we never implemented, but we are thinking about something called pathologist, which can identify code smells, and one of that was unreachable methods and unreachable classes, right? So, we have the symbol index, we have the call graph, so based on that we can find out which classes are unreachable or which methods are conditionally conditionally reachable. So, based on that we can identify what is a dead code. We did not do it here because our uh main focus was to decompose everything and get the domains out like the functional analysis, not really the uh tech due diligence. We did not do it, but we could do it using this. We just need one more agent. Uh uh which calls? So, uh we we also you tried to use a one more strategy where we wanted a historian which could look at the history of the code base, and based on that it can find identify in last couple of years which part actually received most of the changes and everything. Uh we could use a telemetry and all, but again, the thing is like our focus was a functional decomposition here. Uh if I was doing uh any sort of, you know, bug fixing or something, I could use the logs and everything, even the telemetry to identify which part of the code is used most, but we did not use it. Um yes. >> So, uh Harshad, very nice talk. Thank you. Um I have a slightly provocative question, right? >> Sure. So, >> I think very interesting talk in the sense that the central premise is that don't don't sort of just go and read all the code but you know the way you it's it's essentially like applying software engineering first principles which is separation of concerns and tools that have always existed in software engineering right like for example LSB dependency graph reading palm files building using those to figure out and your you know dependency injection and stuff like that right so the question is this that what I mean what would what do you think might happen or did you try this if this itself was given to an agent as a problem saying you know do this go spawn off separate agents to do these tasks right so that's it's almost like saying that the conductor that you have is well I write a you know agent spec for the conductor and I identify these roles and I tell tell this agent spec that go and spawn off separate agents to you know go and accomplish this and then come back and build this up did you try this or what do you think might happen I mean >> so in initial attempts we actually relied on that one and uh what I remember is like uh in March Claude wasn't actually creating explorer agents or anything like it was just using the >> unless you ask it >> yeah but now it's doing it yeah Claude has become more smarter it it is kind of creating its own explorer agent when you ask like okay read this much part it says let me spin up explorer agent and then give you the output it does that only thing is like where you need to provide help to it is the recipes of the particular analysis that you want to perform for example how you process a spring bean graph, how you we we we did not had just rest controllers. There was also some of the soap endpoints which are lying out there, right? There was a very home like a home-grown middleware which which we had to tell Claude like this middleware works like this. There was a camel routes. So, they require a separate processing. So, we cannot fully rely on this one because then we don't get that deterministic output. We wanted the knowledge base to contain the specific thing and we realized some of the things like this only because what we did we created the knowledge base. We asked it to like analyze a defect and it was not able to find a path based on that. Then we asked the developer, "Hey, why it's not able to find that path? Where is the issue?" So, it showed the issue is in this file because this was a solved issue. Okay, it is in this file. Okay, I asked Claude, "The issue is in this file but you are not able to go to this file. Why?" So, he said, "The actual thing is that route is connected through a camel graph camel camel route that basically file is connected through a camel route. So, I need to process that." So, we added one more recipe. If you come across camel configuration, process it like this and connect all the processors and the endpoints. >> Okay. >> You know, so you have to help it there. So, entire I what I feel is like entire orchestration you can rely on that Claude completely but some of the core spoke part which is maybe like on which Claude is not trained on or something, we have to provide. >> that's that's like saying that, you know, that would be the equivalent of a scalar. That would actually be a scalar. >> That's a scalar. Yes. >> Like you just go and do this. >> Yes. >> I have one other slightly tactical question. Like as you were presenting this, a lot of this seemed like at least the first few parts like you know and this is in the context of a large or a extra large code base right it seemed like a lot of this part was like pre-processing like that it would have to go do that sort of you know it would have to go do that and then complete that for it to be able to answer you know questions against that code base is that correct or is that basically till till you know for example if you go back I think you you started with your first layer right with the scout the scout layer yes so it particularly layer zero and one right this is scout and the cartographer because the cartographer is literally like I mean it's like map right so it's map building and logically right yes so it seems like that this is like a pre-process thing before and unless this completes like this is like a pre-processing that you would have that it would have to do over a code base maybe that takes time like a fairly long time and only when that is done then you can then it can sort of serve these interactive queries >> Yeah means initially took a lot of time but a later when we replaced the cartographer layer with tree sitter it was pretty fast and by the way now you have GitLab orbit which which performs the same thing it creates a duck DB based index out of your code and so you can use that it also provides I think semantic search over the code but when we were doing this not graphify was not present GitLab orbit came like one month or two months ago >> [laughter] >> and even the the Serena MCP is out there even the code base memory is there all of them are like pointing to the same thing that you just said like yeah they are creating the a queryable graph knowledge graph which can be queried and enriched into the next level of information. >> Thank you. Thank you. Any more questions? Okay, so uh