From Prompt to Proof: Governing LLM Intelligence in Pharma with Snowflake and Shiny

Hi, my name's Tony Kaskasali. I'm the CEO and founder of TCB Analytics. We're a Boston-based data consultancy that's been around since 2015. Um an avid R user, R lover for over 20 years, and I'm excited to be here with you today. So, today we'll be talking about the use of LLMs specifically in pharma, and some of the challenges we came across, how we're solving them, and overall how we're building a competitive intelligence tool for our client. So, the client is a rare disease pharma company, and it's really important that they stay on top of all of the specific trends happening across the rare disease landscape, understanding what other clinical trials are happening, what other drugs are being created, and so on. So, here's just an example of some of the questions that they wanted to answer. Some could be very specific around trial readouts that aren't even published anywhere yet. Uh for example, when is this randomized part of trial X expected to read out? Or what clinical trials evaluate drug A versus drug B as well as a combination strategy in breast cancer? And so on. And some of these could be pretty basic or very, very specific. And these aren't just look up questions. They require understanding relationships, complex relationships between drugs, trials, outcomes, organizations, and so on. So, the natural question is can't we just ask an LLM? And unfortunately, the answer is no, we cannot just directly ask an LLM reliably. So, our LLM here is the suspect, and you've probably all heard of large language models hallucinating and making up data or evidence. They have invented studies, uh mechanisms of action, citations, even DOIs, digital object identifiers that don't exist. One analysis uh it showed that of 18,000 papers published in 2025, 2.6% of them had at least one potentially hallucinated citation. So, that's a problem. We can't rely solely on LLMs due to this natural hallucination problem that they have. And here's where things fell down particularly strongly when we tried asking an LLM directly. We asked, "Which trials evaluate drug A and drug B in an indication?" Let's just say stomach cancer. So, what the LLM did was it pulled papers uh and trial data on drug A and then on drug B separately. And then put the results together and fabricated an answer. And when we went and verified, there actually were no trials found that have looked at drug A and drug B as a combination strategy. So, the LLM answered very confidently that this has been studied and is a documented trial. So, this is obviously a huge problem. And the reason this happens is that LLMs infer relationships through patterns and probabilistic reasoning. So, they are not deterministic in nature. And in medicine, we cannot be probabilistic only. We need to have explicit relationships defined. And since those relationships Sorry, implicit. We have to force the LLM to understand explicit. So, here's an overview of the full pipeline to give you an idea of how we go from user-generated questions to natural language answer. So, a user can come in to the shiny front end and ask any type of question. And the first thing that happens is the question gets classified. And we extract entities from that question. and I'll get into the details of that next. It [snorts] then goes through a duplicate detection process. So, has someone else already asked the similar question? Has it already been answered? Has that answer been verified? If yes, we can skip that entire blue section on the bottom and return the answer and the user just sees it immediately in the front end. If it's not a duplicate, it has to go through this whole process. So, we go to the knowledge graph, which I'll talk about as well, but a knowledge graph essentially forces those explicit relationships I just mentioned. It's an ontology and it's how we'll ground the truth in it with our data before asking the LLM. We embed the questions and store them. We then provide that ground truth from data sources we trust to the LLM, get an answer, embed that, store that, and pass it back to the front end. And then qualified users are able to now verify whether that answer is correct or not. So, that's our human in the loop. And once it's flagged as correct, it's in the database flagged as verified and we can pull it for future use. One of the first things we do when a question comes in is what I mentioned, entity extraction. And this helps us understand the question, helps with duplicate detection, but it also allows us to pass the right parameters to the knowledge graph so that we get the right data back. So, you can see, for example, are there trials testing a TKI combined with chemotherapy versus chemotherapy alone? And it groups it into these different things. So, the drug class, the combination if there is one, the type of comparator and the arms of the trials, chemotherapy alone, and so on. The next step is categorizing the question. And there's a lot of reasons we do this. This is not an exhaustive list of categories, but it allows us to route the question to the right data sources. So, for example, if it's a mechanism of action or molecular question, then we'll likely go to Campbell or DrugBank. Whereas, if we're looking at trial design and operations, we'd probably go right to clinicaltrials.gov. Now, embeddings are really important for a number of reasons as well. And if you're not familiar with what embeddings are, it's essentially a way to turn words, strings of text into numeric vectors that the computer can understand and compare them. So, this is really important for our duplicate detection when we're looking for similar questions. So, you can see biological and biology are very similar, obviously. But then with biology, it's very complex. The search space is huge. We need bio-aware embeddings. And there's something called BioLinkBERT that we tested out. It worked really well. So, for example, if you give a regular embedding model, TKI versus tyrosine kinase inhibitor, it's going to compare the strings and say these don't mean the same thing. However, if you give it to BERT, it's going to know that TKI is a tyrosine kinase inhibitor because it's been trained on PubMed and and millions and millions of medical documents and literature. And of course, this wouldn't be an R medicine talk without showing some R code. So, here's just an example of how we wrote the embedding function. And you can see we're actually literally just calling the Snowflake embed model. We ended up replacing that with the the BERT model because it's bio-aware. And then you can see how we're doing vector cosine similarity between the two question embeddings to to get a similarity score back. Now, we wrote this all in R and the front end's in Posit. We've got Snowflake on the back end. However, we are actually migrating all of this into Snowflake itself. So, if you want to use custom models like BERT, you can use what's called Snowpark Container Services and it'll allow you to basically query that model from anywhere. So, really this becomes platform agnostic. If we want to build, say, a React or Streamlit app on top of this entire question and answer pipeline, we could because it's all in Snowflake. It's really just API endpoints at that point. But, of course, with R, we're able to prototype this up and and build it very quickly. Now, I want to talk about one of the most important pieces in this pipeline, which is the knowledge graph. And if you're not familiar with what a knowledge graph is, knowledge graphs are ways to define explicit relationships. Uh it gives you a standardized ontology. It allows for all kinds of cool graph querying. But, in this case, one of the most important things is that we ground our data in truth and we pass that to the LLM before doing anything else. So, for example, to avoid that problem with drug A and drug B, where the LLM just hallucinated a fake study, this lets us go look at these nodes first. And we have relationships like a trial has a drug. A drug treats an indication. A drug is owned by company X, and so on. So, for that particular query where the LLM just fell apart, we would ask the knowledge graph, "Give me all trials where drug A and drug B were used, and only trials where a combination therapy was used." So, we're able to sort of guardrail before we ask the LLM with with data from trials that we know are valid and exist. So, I want to talk through now a specific example of how we query the knowledge graph. If we go back to that original question, "Are there trials testing a TKI combined with chemotherapy versus chemotherapy alone. Remember we did that entity extraction? That's what gives us the specific parameters to pass to the knowledge graph. So, in this case, we would query the graph and say, "Give me trials all trials with two groups of patients. One group gets a TKI plus chemotherapy and the other group is chemotherapy only. Make sure both groups are in the same trial, right? So, we avoid the drug A and drug B problem that we saw earlier and it doesn't fabricate some trial IDs. And then the knowledge graph returns those NCTIDs. So, now we go and we query clinicaltrials.gov with those NCTIDs and provide that data to the LLM and then ask our question and limit it to only using that data. And at the end of the day, we always have a human in the loop. The human is the final judge. I've like to say AI creates a draft, the human ultimately decides. So, what we end up with after going through that whole pipeline and process, are these what we call knowledge cards. And they'll have the question, the answer, some tags and information about the trial or drug, whatever it is. And you'll notice there's probably hard to see, but there's a little eyeball and an edit button. And so, the people that are allowed to verify answers can come in here and see that it hasn't been verified, click in, verify the sources, the answer and press approve. And then that goes back in the snowflake as a verified knowledge card and answer. So, the next time someone asks a specific question, we do our duplicate detection, we determine, "Hey, this has already been answered and verified. We can just return that verified answer right away." Here's what the final product looks like. The UI portion in shiny allows the user to ask a natural language question. We also have the ability to do structured queries, which provides a set of drop-downs to pick an indication, a company, and so on. We get back an answer with the different sources, as well as an audit trail of how was this answer actually generated. For example, here are the concepts in the knowledge graph, the key nodes that were used, and the relationships, the various data sources from CT.gov, or whatever proprietary internal sources, and then the LLM used to generate the answer. For the final verdict, ultimately, the goal isn't to create smarter AI. We need to make it harder for AI to be wrong, especially in medicine. We need to deliver trustworthy and grounded intelligence. That grounding comes from the knowledge graph and verified data sources. We need to answer questions at scale, uh improve the answer quality by routing it to the correct data sources, some of which may not even be publicly available, and need to be accessed internally only. They could be internal PDFs or poster abstracts. And ultimately, you're building up a database of institutional knowledge. You can maintain a continuous improvement effort to update those knowledge cards, update with new results that come out, and so on. And all of that with auditability, governance, and scalability. Thank you for your attention and for listening. I know that was a lot in a short amount of time, but I wanted to leave enough room for questions, and it's hard to go through too much technical detail in 15 or so minutes. So, happy to answer any questions. Feel free to reach out. Here's all my contact info. And thank you, once again, Our Medicine for having me.