Care and Feeding of Your Biostats Team: Scaling best practices in a large hybrid SAS/R team

My name is John Erlinger. I work in the cardiovascular outcomes registries and research. We're part of the Heart, Vascular, and Thoracic Institute at the Cleveland Clinic. CORE was started in 1997 with the arrival of our director, Dr. Eugene Blackstone. We started as a pure SAS shop with a small team. Guiding principle of CORE from day one has been to have the ability to hand off any study easily. These hand-offs require a social contract of practices. These are held together by the discipline within the team. Our primary in-house method started with Proc Hazard, a C program with tight SAS bindings. Over time, R entered by the methodological evolution. Dr. Hemant Ishwaran developed random forest SRC in R and CORE in 2007. Random forest for survival, regression, and classification. I wrote the GG random forest package to simplify plotting these forest outputs using GG plot. As we've grown, we found that discipline alone doesn't scale easily. We are concerned with the scale that this social contract has to support. 19 staff driving 150 data requests delivered directly to researchers for their publications. We also directly produce about 100 presentations and 50 manuscripts a year in collaborations. All of this feeds into the 1,100 HTI peer-reviewed publications produced last year. Every inefficiency in our system multiplies across that. This talk, I'll be directly addressing what operating at scale means. These 19 staff are split across three disciplines. Data engineers focus on executing data requests and then getting data from the medical systems into our data warehouse, primarily using Python. Stat programmers assemble analysis data sets primarily using SAS. Biostatisticians build an analysis report using SAS and R. Our researchers collaborator convert the analysis report into manuscripts and presentations. Data flows through the systems. People transform it at each step. The biostatistician sits at the core taking the analysis data set and producing the analysis report that the researchers can use to construct the manuscript. Since this is an R conference, we can focus on the biostats component of this pipeline. The main deliverable of the team is the analysis report. The analysis done in SAS and R jobs are an implementation detail. What our customer cares about is the report that comes out of the other side. Researchers bring the questions formed as hypotheses. This drives the process. What data is required to answer these questions? Core determines the statistical approaches in collaboration with the researchers, refining the questions, figures, and tables communicate the results to the end audience. Scaling the team means making every step of that pipeline reliable and handoff ready. We still maintain the goal of any analyst can pick up any job. We call this bus insurance. Can another analyst step in and pick up wherever you are and continue the work? There are three areas where reproducibility breaks down in practice. We'll address each one of these areas with a Here's the failure mode, and here's what we're doing about it. Make the safe path the easy path. We currently do a lot of mental work to compensate for gaps in each of these areas, remembering which data set is current, keeping track of which version of a function ran, manually assembling figures into a report. The goal isn't to add process on top of that, it's to replace that mental overhead with automation. Using checksums instead of your memory, package versions instead of tribal knowledge, port or cut it instead of copy and paste. Long-term division is an audit process that runs automatically, not a checklist someone fills out, but a system that tells you whether the analysis is reproducible before you have to ask. We're not there yet, but every piece we put in place moves us closer. Starting with data ingestion, making the safe path the easy path. We have a standard folder structure. That isn't the problem. That works. We But we've all seen these naming strategies. Data file naming might work. We also use the system the file system modified dates implicitly, but those are fragile, changing uncontrollable ways. We want to standardize our data with a simple automatic approach. Start by making the file date part of that standard. Adding the date to the file name when the file is created means it won't change if the file is moved. That doesn't solve the whole problem. We still need to track what happens inside the file. What if the file gets overwritten? What if a data extract gets updated? Naming and modified dates weren't designed for that job. The folder structure is already there and it works. Jobs go in the analysis folder, reading data sets and writing models and results to an estimates folder, and then generating graphs. The documents folder contains our deliverables. Inside the data sets folder, file names can now include the date they were written programmatically. A data set manifest can be created with a checksum registration before we first use the data file. This must be separate from data set creation and done before we build the individual analysis jobs. We are building a data audit trail checked inside the analysis jobs. Importantly, this can be done regardless of the analysis platform. Checksum is stored in the manifest at registration. This is the deliberately separate from file creation. If the file is overwritten accidentally, say a data extract is rerun without a date change, or the data corrected in place, the old checksum stays in the manifest and the job should fail till someone consciously reregisters or recreates a new data set. Social contract now is never overwrite a dated data set. This makes rebuilds fail loudly by default. The date stamped file name tells you when the extract was pulled. The checksum tells you whether the file you're reading now is the same one that was registered. It doesn't tell you if a new data set file exists. It only indicates that the registered one hasn't changed. Model building, making the safe path the easy path. In model building, we have a few problems to address. First, regardless of the platform, SAS and R versions are not locked into the study. We work on a shared disk with our studio server and a shared SAS instance. Neither platform captures the environment well. SAS's shared server setup makes this invisible when you run a job. R studio can help with projects if we use them. Global package library in R makes using package versions fragile over time. Failure modes look different, but the root cause is the same. No one picked the envi- pinned the environment to that study. Can you pick up a job 6 months from now? Will it still run? The key here is we want the handoff. Python has virtual environments. In R, we can use the RENV package to track the environment. On day one, we initialize RENV and then take a snapshot of the package library and R version we are currently using. This makes the jobs transferable, not just reproducible on your own machine. The lock file travels with the project. The library itself gets rebuilt locally when we create the invoke the RENV restore command. We update packages intentionally with RENV install whenever we need the new versions, and then create a new snapshot so that those are also stored with the study. SAS has no equivalent that I can find. It is less volatile than R, but the environments are still an open problem. Our strategy for SAS and R shared functions surface two different failure modes, both equally bad. We use a SAS standard centralized macro library, but we don't control when chances changes land. Every study gets every update whether you're ready or not. We also use a master template folder. This copies the current version of R and SAS shared functions into the study at creation. This isolates studies from each other, but also from improvements. Fixes typically stay local to the study until they're copied into the master folder, if they're copied into the master folder. And then these functions often diverge over time because we rarely propagate changes back from the master folder into an existing study. Neither platform has a deliberate version release process. Version packages are better than scattered copies, but the governance problems shift over time rather than just disappear. Combined with RENV, the version choices become explicit and deliberate, which is exactly the point. There is still a maintenance issue. Anyone can currently edit the map to SAS macro library. The package development and takes a different skill set. There are new disciplines around release management and communication. Still, controlled versioning helps maintain stability of a study over time. The next standard problem is the code and results are not connected. This again transcends the platform. SAS, R, and even Python all have interactive execution modes. In SAS, we solve this by mandating that you have to run the whole job before leaving it. Does everyone do that every night? Maybe. Probably not. In SAS, the list file is the output. Python has Jupyter notebooks. R scripts have no explicit equivalent, so we turn to Quarto. Quarto can be seen as being analogous to a SAS job and the list files. Single unified report that includes R code and the output. The report has the advantage of including real figures in line and renders to a deliverable report in HTML, PDF, or even Word. The pitch to a SAS analyst isn't just learn this new tool. I tried that. It didn't work. It's this gets what you already have in a list file, but with real figures in line and a structure everyone agrees on. We're working on a four-section template structure as a standard. Short enough chunks to be understandable, consistent enough to navigate, flexible enough to cover every study type. We have data ingestion report that passes clean data to the methods for model building. The model is then passed to the explainability and the results reports for communicating with researchers. Longer term, we're working on a book down approach to possibly replace the manually assembled analysis report deliverable that we rely on. We still have an underlying copy-paste problem. Both SAS jobs and R scripts are copied from the central templates folder, meaning in-flight studies don't implicitly get the updates. R 2 and R packages are the foundation we're building first before tackling that propagation problem. Figures and tables, making the safe path the easy path. We spent an extraordinary amount of time and effort to make SAS graphics behave with our SAS plot.sas macro. We bend SAS graphics into publication quality for graph for manuscripts, but we never solved the presentation problem. The methods we used to use we used to use broke over time, so every presentation now requires a lot of manual work. ggplot is more flexible, but customizing the output for manuscripts and presentations still requires a bit of effort. Can we capture that effort and make our figures consistent? Our internal HPTI plot R package addresses this. Capturing the effort in a package and enforcing shared standards through an S3 pattern. The S3 pattern matters because it separates concerns cleanly. The transform functions do the statistical work and return typed objects. The plot method knows how to visualize each of the typed objects and returns a ggplot object. The analyst's job is then decoration, choosing a scale, adding annotations, not rebuilding the figure from scratch. We provide different themes to target different end targets. One for manuscripts, a couple for presentations if you're using a dark background or a light background. Formatting is all controlled within the included themes. The package examples end up being the documentation and the quality standard. If your figure matches the example, it's correct. This is what we will cover and improve consistency in how we deliver publication quality graphics in R. GGRandomForest was a proof of concept born from that internal need, built as a package, and eventually adapted externally in other organizations. It was removed from CRAN because of a dependency drift, not a design failure. The pattern still holds. I hope to get it back on CRAN soon, but there's still a bit of work to make to bridge the gap. HVTIR plotR applies the same model to a core specific plot types. These packages are always in development. Science keeps moving, and the package has have to keep up. That's a governance problem as much as a technical one. This is the type of output that motivated GGRandomForest. Random Forest SRC produces individual predicted survival curves for every patient in the cohort. What you're seeing here is full distributional picture of survival across two groups, not just the mean. GGRandomForest turned that forest object into this figure with very few function calls. Handle The package handles the extraction and the scaling. And ggplot scaffolding, the analyst then controls labels and themes. We're not done. I've used R table packages when communicating with collaborators, but we need to tame the GT and GT summary outputs to align with HVTI publication standards. We have SAS table macros, so the framework's there. HVTI R tables package is on a long to-do list. This table is the honest answer to where we currently are. Folder structure and naming conventions are long-standing and solid. That's the foundation everything else builds on. R and V is the easy win that we're rolling out now. Methodology and graphics packages are in use, but actively changing. I'm also actively working on in development items. And the aspirational items are real and planned, but not yet standard practice. This is a real production environment with real constraints, not a clean academic case study. To wrap up, I've been working on these pain points in core for a long time. The reproducibility framing named failed. Reproducibility framing failed. It reads as an academic virtue. Something you do for reviewers when asked. The production reliability framing lands differently with the team. It's about whether someone else can pick up your job on Monday morning when you're out sick. Same tools, same practices, but the reason to adopt them shifts from good science hygiene to your teammates won't be stranded. The second framing can help with buying. Make the safe path the easy path means the standards have to be lower friction than the workaround, or people will always take the workaround. Being transparent, it still takes work for people to change their ways. Even if the path is hypothetically easier, rollout is still a learning experience, and it's being done under production pressure. I'm hoping that this talk was informative and might start some conversations. I want to know, what's your experience? And can we all benefit from the knowledge as a community? Tell me, please, what have you done? My slides and other resources are all available on GitHub. Additionally, I'm hiring, so please reach out. Thank you.