Science in the Agentic Era: Structured Experimentation With Ado - Alessandro Pomponio, IBM

So hello everyone. Uh thank you for uh coming taking the time to attend. Um I'm Alessandra Pomp. I'm a research software engineer at today we're going to be talking about agents but maybe from a perspective that is not as popular. And today we're going to be talking about agents in science and how a tool that we developed and open source called ADO will allow you to do science with them. So let's start off with a fairly basic question that has been answered many times already today. Uh what is a coding agent? And today I'm going to be referring to agents as coding agents. So please keep that in mind. So a coding agent is um an autonomous AI program that has been powered by an LLM and that can reason and act. So it has agency uh towards the goal and that can access tools and correct itself in a loop. So basically a coding agent is just an LLM that has an harness which is some scaffolding that gives it access to tool usage manages prompts contexts state and control flow. I assume that given the uh kind of uh conference that we are at uh many of you will have used at least one of these uh coding harnesses and they're very good. They're really really good. They allow you to do software at an incredible pace. You have to keep them in check and not produce some AI slop. But um they're very good. And the the reason why they're very good is that they break down problems into four stages. These stages are explore, plan, execute and verify. These steps are not just belonging to these agents. Uh they are everywhere. Um we can see them even in something that we all know as well which is the DevOps loop. It's uh a framework that gives us uh instruction to do development in a an iterative way. And if we look at these eight steps, we can recognize what I just talked about. So we have an explore phase which is the discover phase where we gather information about our current issues or feature requests that we may have. We have a planning phase where we define the requirements, create road maps and define tasks. We have an execute phase where we write the code, we commit it, we test it, we release it and we deploy it. Of course, there's a a test phase here which brings in pieces of validation already. But for a product I would say the most important verification is in the actual monitoring that we have in this verify stage. Right here we manage the infrastructure. We monitor that our product is behaving correctly. We collect feedback and that drives the next iteration of our DevOps loop. If we try to map these things to agents, we have these again four points. Um, so the agents can perform these steps thanks to the harness and the tools that the harness provides. So for example, agents can use these tools to gather contextual information from various sources. for example, um the the code repository that you're editing it, seeing the git history from this repository to see how the repository has evolved over time and maybe a commit that was made recently introduced a bug and that kind of information in addition to logging etc can really help these agents. This information in fact is is it is planned. It is used in the planning stage [snorts] to create these u these plans that are structured and identify the intermediate steps that the agent will have to follow for implementing these fixes including uh potential risks and validation mechanisms. The harness also provides some kind of level of uh isolation. Um if you go to the sandboxed stage uh you can have more isolation but in general these harnesses provide ways that these agents can use to um write files, read files, execute API calls, execute commands and this really is what the agent is all about nowadays. Right? Of course, we then have a verify stage in which the agent using the features that are provided by the harness can verify that the changes are correct by using language servers, finding uh using you know tools that the project provides for example the set of llinter tests and monitoring instrumentation to then uh verify that the changes are correct. But until now I've just talked about DevOps. So where where's the science, right? Well, it's been here all along in a way because if we really think about it and we ask a question, what do scientists do? They explore, they plan, they execute, and then they verify. Right? So if I just change the slide that I've been showing you uh with the DevOps loop just a little bit and I call it a different way. Let's say I call it the scientific method. We see basically the same things, right? The scientific method starts from a research question. Let's call this the explore stage where we research the state-of-the-art of the problem that we're trying to tackle. It goes to a planning phase where we make a hypothesis on how something is supposed to work. Then we execute on the hypothesis by designing an experiment, validating it and running an experiment campaigns that is actually meant to test the hypothesis in a scientific and rigorous way. We then validate the hypothesis by collecting the data, analyzing it and drawing conclusions and seeing whether the hypothesis is confirmed or it is not confirmed and then this information will guide our next research efforts. So if the question isn't right, if the agents can do the DevOps loops and the scientific method is basically the same, what's missing? So what is a science agent? It's probably a coding agent plus something. And in our vision, this tool is ADO. So ADIO also called the accelerated discovery orchestrator is what we believe is a toolbox for agents in science. It provides schema based uh a schema based approach to experimentation. So it is a python framework and we use for example paidantic to make to define rigorous models uh for all the configurations that we use for our exploration to make sure that the agents have a framework in which they need to stay and we validate all everything that uh is created to make sure that nothing uh goes ary. Then we have provenence tracking because we're talking about science. So results must be reproducible and must be trackable. So we track what kind of version of ADO or what kind of experiment was executed to measure a point for example so that you have this kind of you can trust the results. We also uh automatically collect results in a structured data store with support for collaborative projects because science is something that is not done on your own most of the time. Of course, we then have the usual bits. We have an extensible core architecture where you can add your own plugins, your experiments, your samplers, whatever you want. Of course, we then have a scalable execution back end powered by another open source project which is Ray. You can use it on Kubernetes as well for even more scalability. Um for example, we will you will see that we can run like um 2,000 optimizations uh concurrently and this is possible thanks to Ray of course. Then we built an intuitive CLI. We started off a few years ago. We we were calling this a human- centered CLI because we followed uh the best practices as much as possible for this. But we saw that this translates to agents being able to use it better uh as well. Then so the what we think here is again science agent means coding agent plus ADO and the way we map the usual explore plan execute verify with ADO is is here right so ADO again provides a structured data store with methods to store and retrieve these kind of plans configurations results in a way that is persistent of course and quarable which is the most important bit. Agents can then use this information to analyze the results, discover trends in the data and find the gaps in the research. These plans and reports are then written back to the data store because the evolution of the plans through times through time is very important for the agent to understand what's going on. Then we also allow the execution of experiment campaigns through the CLI and through our uh API of course. Well, the orchestration layer automatically tracks prominence information and the sampling trajectory because in um when you sample points you can have uh ve very much information hidden in the trajectory of the sampling that can help with troubleshooting. For example, the verification is done through for example pyantic schemas for all our resources and through accessing the execution logs while the CLI is working and um the traces through the sampling trajectories. The agent can then perform some advanced troubleshooting to verify why for example a bug was there or what happened during the optimization run. In general, the execution loop that a human or an agent can do with ADO is this one, right? It's again colorcoded in a way that uh maps to these stages. We will retrieve the existing uh plans, experiments, and configurations and then create a plan using the existing information to define what are the next steps in our research. We then define what we call discovery space which is the parameter space that we want to explore in our research and the experiments that we want to run on these points. We then run the experiments on these points and we store the results in this data store. The results are then analyzed with the exploration and used to draw conclusions. So if it's that simple, they lived happily ever after. Thank you for your attention. No, I'm sorry. It's not the way it is because agents are good, but they still miss something. Coding agents work well on established frameworks, but when dealing with ones they haven't been trained on, they often time fall into these five traps. They don't know where to look. That's the biggest issue. During their exploration step, we will they will try to search by keyword, but often time they will fail to find exactly what they want. This leads them to hallucinate commands, classes, methods, whatever. And if even if sometimes they manage to scavenge through and find something to hold on to and get to a solution, they will take a long time and burn through your token budget for the session. They will often also end up with these creative solution trying to circumvent the problem that they haven't explored the codebase well and they will also create many single-use scripts that are impossible to keep track of and you will end up um losing. So I'm not a doctor but I think the diagnosis for this is just a skill issue. So we need to upskill these agents and the way we do it is through agent skills. I'm assume that most of you are familiar with these as well. They were introduced by Anthropic. They can you can define a simple markdown file with instructions basically that the agent will follow when matching to a certain set of keywords and intents. With these, you can really prevent agents from guessing to reduce their hallucinations. You can save them from going down the wrong paths by defining clear to-dos and to to not do. Uh, and I think this is like really important because often times when an agent starts going down the wrong path, you can you can hardly ever get them to go back to the correct path. You can also provide step-by-step workflows with decision branches and stopping conditions to make sure they act reliably in a more trustworthy way. So here's a small examples of the skills that we have added to ADO um to to match to to what I was saying before. So to prevent agents from guessing, we have a skill that's called query ADO data that shows them how to efficiently query query our data store to find existing plans, matching spaces, and more. some of the instructions that we have over here. For example, just tell the agents use a JSON path query to find information instead of retrieving everything and do it in Python because then we can they they just waste tokens. They get a lot of information and struggle to parse it. We also have a skill. It's called formulate discovery program problem uh which is just based on examples because just like humans they really appreciate example-driven um skills where with this methodology we tell them how to choose the appropriate experiments how to define the search spaces and the experiment campaigns for our discovery problem. Then we have conduct empirical study which is a step-by-step workflow on how to run experimental studies with ADO. This is a skill of skills. It'll call where appropriate all the other skills and implements the full execute explore plan execute and verify loop. So uh we're going to see a demo. In this demo, uh we are going to use ADO and well we'll show you how we use ADO to find optimal solver configuration for uh a program called CLEX that developed by IBM which is used in a mixed integer programming problems. We will start off by seeing real life case where somebody else in our team had been using ADO to run their experiment campaigns and we want to see where we are at with our optimization. So we just we just start asking examine the ADO pro uh the pro project and suggest me the next steps on what to do. The agent will use the skills and query our database to find all the relevant information. The agent will then synthesize this information creating a structured report and recommending next steps. So I'll stop it here for a second. We will see that we are running some mixed integer programming problems from uh a library of problems come called miplib. We have run 27 operations which is our experiment campaign on 15 parameter spaces. We see what has been happening and we also see that it found some issues. So here the agent has seen that this sorrel 3 operation failed because all of the uh six submitted trials reported nan and it suggests maybe this is why it happened but we are not uh satisfied with maybe. So we have another skill to investigate for example these failing experiment campaigns. Why did our optimization fail? So we let the agent loose. We ask it to examine the operation. The agent goes in a planning phase. It retrieves the important information using a step-by-step workflow just to keep it in the correct path. Get the operation yaml. Check for an existing report. Review what the operator what the sampler does. Check the definition of our parameter spaces. get the outputs, get the measurements, and then analyze. Do not just guess. So, the agent does all of this thanks to our strict schemas. It has an easier time doing this. It writes a report. And it says there are two potential causes. One of them is that a file is missing. Try this. Try this. run a diagnostic experiment, sample just one point, wait it actively, gather all the logs and identify the exception before rerunning the whole experiment campaign again based on data. So let's ask it to run this experiment campaign, run the diagnostic operation. It'll look at other example of existing experiment campaigns. It'll create some definitions using our schemas and it'll launch it on our cluster. Now the agent has started this onepoint experiment to see if everything was working. It looks at it. It says it's solving well. I waited a minute. It has a 20 minute time limit for the optimization. Let's let's leave it be. I'll check once the exploration is complete. But it seems that things are working. So let's see the operation finished. We have the diagnostic worked. This is what we know. The root cause has been confirmed and it says you can now I fixed everything that needed to be fixed. You can now rerun the full 200 trial operation. So let's resubmit it. And also while we're at it, let's create another operation on a different space to run another problem, a lecture scheduling problem. It'll fetch similar spaces to see if something already exists and break down the problem. So the operation has been submitted. It didn't find anything for the other space. So we will create a new one. But we see it made a mistake. But we at the top we can see it dry run the creation of the space. So we gave it a way to verify that its changes were going to be correct. They weren't. And thanks to the error and our thanks to the schema, it it figured out what was wrong. It then fixed itself. We create we tried run the validation again. It worked and it submitted. So both explorations are running and we can proceed with our science. Now, of course, this is just a a demo, but it's a demo that happened in our research. And if you use ADO, you can try it on your own. So, I'll use my skill to conclude the presentation. I'm just going to say that we're listening. Uh we're very much looking forward to contributions and if you want to try it, please get in touch. This is our repository. It's github.comibibbmad. Six letters quite easy. And of course, there's a a link to LinkedIn if you want to get in touch with me directly. What's very important is also your feedback on the presentation. So, if you were so kind to submit a feedback for the presentation, it would be much appreciated. Thank you very much. If you have any questions, I'll be here for you. [applause] Feel free to raise a hand if you have any question. Um, do you want to come closer? I'll repeat the question or I can just give you the the microphone. So this is really interesting. Um, one of the things that LM struggle with is a lot of this like planning and contextual things across um, long runs. So obviously you're taking a step in this direction to improve it. Um, can you say more about the limits here? Is this just something where you feel you could feed it the proceedings from a journal or something like that and just tell it to make computational improvements on the latest like file system results from a file system journal and sit around and wait for a year and it would produce a bunch of research papers or where where are we going with this? >> Well, this is just a tool, right? Um, you can I think a good thing to do with it is like find a research paper that you're interested in and ask it to implement it in in here. Like we had for example some quantum uh research papers that some colleagues of ours had uh had been looking at and we could ask our agent just reimplement this because the code wasn't published and reimplement it with this framework and it worked right now can you do automated science with it? Well, maybe not, but it gives you a framework to run these experiments in a way that's probably more reproducible. And I think the real value here is that you don't have to reimplement the wheel every time. You have an orchestration layer that takes care of, you know, storing your results, quering them back. And this is for free, right? You just write your own experiment. You plug it in the framework and it works. If you do a lot of research and you have to reimplement it every time that gets taxing even if you have your coding agent which maybe can do it fast but it'll have to do it every every time. So having a framework in which you can execute things I think it's very valuable. Not sure if that captured your question but I I'll be I'll be here if you want to ask more. Any more questions? We have like a few minutes left as well. Well, if not, thank you very much once again. See you next time.