Report Agent: An LLM-Powered Framework for Automated Shell-to-Code Report Generation

Duration: 10:07 1087 transcript words EN summary

Video summary

Lucy John, a senior analytical data scientist at Genentech with eight years of experience in clinical trial data analysis, introduces the Report Agent, an LLM-powered framework designed to automate the generation of R code from shell documents and CSV schemas. Her motivation for developing this system stems from the repetitive and expertise-heavy nature of manually creating TLGR scripts, which often requires deep knowledge of shell structures and data schemas. While large language models offer potential solutions, Lucy highlights that directly prompting an LLM for end-to-end results is unreliable due to frequent hallucinations and structural mismatches. To address these challenges, she developed the Report Agent, which breaks down complex tasks into specialized, manageable steps controlled by dedicated components rather than relying on a single, monolithic AI call. The core of this framework is its agent pipeline architecture, which processes inputs like DOCX files and CSV schemas to produce a functional R Shiny application through five distinct agents: passer, planner, mapper, code generator, and validator/reviewer. The process begins with the passer agent, which extracts CSV schemas to create dropdown selections for user input and parses shell information into machine-readable JSON files. Next, the planner uses pure logic without LLM intervention to build a structured generation plan that maps each row group, setting placeholders for subsequent processing. The mapper then introduces the first use of an LLM to fill these placeholders by determining data types and identifying corresponding SAS view columns, resulting in an enriched plan ready for code generation. In the final stages of the pipeline, the code generator uses the detailed mapping plan to create a complete R script, while the validator and reviewer ensure that all labels and rows are correctly mapped without missing elements. If any issues are detected during validation, the system triggers a revision loop back to the mapper to resolve discrepancies automatically; if problems persist, it generates a to-do list for the user to address manually. Once the pipeline completes successfully, users can interact with the generated R Shiny app via a control panel that allows them to review or run the code, view real-time status updates on each processing step, and download the final script along with logs detailing token usage and execution results. Looking toward future improvements, Lucy identifies two key directions for enhancing the system: implementing earlier error detection to reduce computational costs and evolving from processing a single document to handling sequential table processing workflows. These enhancements aim to further streamline the development process while maintaining high consistency and reliability. By combining structured logic with targeted LLM usage at specific stages, the Report Agent effectively bridges the gap between complex data requirements and automated code generation, offering a robust solution for clinical trial data analysis that reduces development time and lowers the expertise barrier for generating reproducible reports.

Read the full video transcript

Hello. Thanks for being here. Today, I am excited to share my work about a report agent. Um this is an LLM-powered framework for automated shell to code report generation. I am Lucy John. I'm a senior analytical data scientist at Genentech. I have been working in clinical trial data analysis for about 8 years, spanning the medical device industry and early-phase pharmaceutical drug development. My focused disease areas are cardiovascular and metabolic diseases. Um as for the motivation, uh during my work, I found out that generating TLGR script is repetitive and sometimes time-consuming. It also requires uh understanding of the shells and the data set schema, um which might be a high expertise barrier. Uh with the evolving uh GenAI, um you may ask, why not directly ask a LLM to give the end-to-end solutions? Well, in practice, I found out that directly prompting the LLM is unreliable. Sometimes, it will give me something unreal or not exist. It's prone to hallucinations and structural mismatch. So, my solution is called TLG agent. It is a based LLM agent. Basically, it will make the whole task and separate into a few tiny specialized steps. And each step is controlled by a dedicated component. The proof of concept confirms successful table reproduction. The preliminary data suggests reduced the development time and improved consistency. The system architecture act like this workflow. In a high level, this system will take input the docx draw and CSV schema. And output is a R shiny app. It has three component. One is a drop-down selections, which I will show you more details in the next page. And the second one is a generated R code that you can download or review. And the third component is a R output while running the R script on the shiny. And the process between the input and the output is this box. I call it agent pipeline architecture. It consists of five tiny agent, passer, planner, mapper, code generator, validator, and a reviewer. In the next a few slides, I will introduce each of them in more detail. The first one is passer agent. So, what the passer is actually doing is two things. One is to extract the CSV schema and to create a drop down selections in the R shiny app. This is example. Once a CSV is uploaded, it will quickly to parse the columns names and make a drop down list. So here, the users can select the treatment groups that they want to use or any variables that you they would like to use for their analysis. Other than this, the parser agent will parse the information in the shell, like the titles or the contents formats, and the store them in a machine readable file, we call it JSON. And this will be used for the next step. Parser is done, then we'll go to the planner. The planner takes the parsed doc shell, CSV schema, the treatment group name, and it builds a structured generation plan. Here, we don't involve LLM, we just use the pure logic and to make a good plan for every things that included in the input. Like this. It will create a mapping task per row group, and it will uh leave uh as a placeholder for each of them as null, and uh it will be used for the next step, which is mapper. This is the first time in this pipeline we use LLM. If you still remember the output from the planner, which is uh mapping task, but have all the fields as empty. Here, it is trying to fill in the informations by calling LLM. It will determine whether the row group type is continuous or categorical. Or um it will uh and it will identify the corresponding SAS view columns and then map it uh from the doc apps. At the end of the mapper, it will return a enriched plan for the code generation. That is for the next step, code gen. This is the second time we call LLM in this pipeline. Remember, we have a very good detailed mapping plan. Then we will use that to call LLM >> [clears throat] >> to generate a complete, very detailed R script. The feedback also will help here if there any issues are found. It will also give us warnings or unresolved mappings if it cannot resolve by itself. So that the users will have a chance to update their input and run the whole pipeline again. At the end of this step, we will have a R script. And the final step is a validator and a reviewer. It will check if all the labels, rows are mapped without missing. If If there are some missing, it will trigger a revision loop from the mapper again. And uh it will help to solve the problems. If not, it will generate a to-do marker for users to solve the problem by themselves. This is the last step of this agent pipeline. And when you're done with this, the users will interact with the R Shiny app. They can either review the code or they can also run the code and then see the output. Here is the example of the R Shiny. On the left panel is a control panel to control the input and also have some buttons that users can play with like a generate R code or run R script or even download the R script. On the right side is showing us the process of each um of each step's status. Here is example. We can know once it's done or it's running or it's pending. So, if everything's looks right, we will expect everything run from the beginning to the bottom and then we will have a clean pipeline log. Um and then we will also have a message about how many tokens used during the whole process. If everything's runs good like here, done is done and there's no error, then we can view the generated R script and we can see the run log which here is a output. For the future directions, there are two sides. One is earlier error detections. Um, this will help with um, the saving the uh, computational cost. And another side is um, move from a single docx to a sequential table processing. Thank you for listening. This is all I have today. Um, let me know if you have any questions. Thanks.