Peter S Kim - Steenbock Lecture II - Fall 2026

Okay, welcome everyone. Welcome to the second day of the Harry Stebach lecturesship uh in the department. So just repeat the uh introduction. The uh steam lecture is one of the most prestigious honor bestowed by our department. Harry Stebach was a pioneering leader in the field of vitamin metabolism and a member of our faculty in the early 20th century. During the time he discovered vitamin D deficiency is a sore as a is the cause of ricketetts and thereafter he played an instrumental role in establishing the Wisconsin Alumni Research Foundation namely WARF which is now among the oldest and most successful technology transfer offices in the United States. Okay. And then again uh for our speaker today uh we have Professor Peter Kim from Stamford. So Peter is now the Virginia DK Lupic Professor of Biochemistry at Stanford. He started his career as the whitehead fellow at MIT and then uh all the way became associate head of the department of biology and Howard Hughes medical institute investigator. And then he went on to become the president of the Merc research laboratories from 23 to 2013 where he oversaw the development of of over 20 new medicines and vaccines. And those include Gardil the first vaccine for prevention of cervical cancer. Januia the first DPB4 inhibitor for treatment of type 2 diabetes. Uh KUDA the first FDA approved antiPD1 inhibitor for treatment of cancer among many many others. uh while he was known for discovering the spring-loaded mechanism for how proteins cause viral fusion with host cells for which he was awarded NAS award in molecular biology and this discovery laid the foundation for the critical aspect of protein engineering namely stabilizing the metasta metastable state of bofusion proteins now found in many different vaccines including those for SARS KV2 among many many awards he has received I have to say he's a member of the National Academy of Sciences, National Academy of Medicine and National Academy of Engineering. And finally, Peter has mentored uh dozens of students and postocs and remarkably six of his trainees has become National Academy of Sciences members. And then one became the president of HHMI and many others in leading positions in research institutes, biotechnology and pharmaceutical industry. And finally, thank you for Serrof Fisher for sponsoring uh the symposium. And then without further ado, uh we have Peter talking about protein language model today. >> Thank you D for that nice introduction. >> Oh, that's loud. >> Thank you for that nice uh introduction. Can this be turned down? be pretty loud. >> Okay, testing. One, two, three. That sounds pretty good. >> Great. Thank you. Still pretty loud. Is it? Let's >> try the other pack. >> Okay. testing. One, two, three. >> Okay, everybody can hear me? Great. Um, thank you. I'd like to talk today about using language protein language models uh to understand and evolve uh proteins. We're all familiar with language models uh thanks to the revolution that's going on uh starting with chat GPT where uh if you take a large corpus of English language and you feed it into a computer and uh the computer can build a neural network and what that uh computer has learned to do is to be able to predict the next word or a masked word in um a sentence. So here the cat likes to sleep in the and then what the computer will do is give you an output with the probability that the next word will be a certain word and so it's giving you probabilities and uh what people first in the uh Facebook group at the time did was to produce a very large language model. The language model that I'll be focusing on is called ESM1B. And this was uh as I said first done by the group at Facebook when they had a a group uh looking at protein language models. And the net result after you've trained this uh language model with a large amount of protein sequences non-redundant protein sequences is you end up with a model that when you mask a particular amino acid. So if this is this protein sequence, if this is a sequence of your favorite protein and you mask one of the letters and you ask the computer uh what letter belongs there, it will give you an output and again it's an output that's a likelihood and it will tell you that there's in this case a 20% probability that a likelihood that the amino acid that belongs here is an isolucine. And the remarkable thing is that uh very often the um the model gives back the amino acid that was actually masked by uh you when you did this. Um Brian He an amazing uh posttock at the time when he joined my lab wanted to ask the question about whether or not protein language models such as I just told you about could be used to predict the the directionality of protein evolution. So if you have a bunch of protein related protein sequences uh that had are related by uh evolution and you put each sequence in as a node and the edges between the nodes correspond to a transition from one protein to another. You can end up and you put them in a what's called a K nearest neighbor network. You create that K nearest neighbor network. What that network doesn't tell you is whether the directionality of evolution is from this protein to that protein or from that protein back to this protein. And so what Brian's hypothesis was when he started this work was that you could use the protein language model to predict directionality. And his hypothesis was that if you put two the two sequences into the language model and let let's say to make things simple that sequence A differed from sequence B in only one amino acid position. If you ask the language model which is more likely A or B, his hypothesis was that the protein that the language model predicts is more likely is the more recent protein. Okay. So if protein B scored higher than protein A in the language model, then the hypothesis is that the direction of evolution was from A to B. So the more recent one was the was the one with the highest probability. And if you do that in a in a k nearest neighbor network the question was do you end up with uh these local directionalities aligning uh with uh in in the same direction and indeed that's what he found and he called what he was observing evolutionary velocity or evo velocity. So from that very simple hypothesis, he was able to take a look at evolutionary trajectories. And here is a sampling of a a K nearest neighbor network of all of the known sequences of the nuclear protein from influenza A. And um what he found was that the evil velocity projections often aligned the individual local vectors often aligned into very long vectors going in the in the same direction. And when he uh when he for example looked at a path from a 1968 flu virus that had been studied by Jesse Bloom, the trajectory down to the 2007 virus. And he ran um a uh a simulation using the pro the uh probabilities uh predicted by the transitions for each of the possible transitions in this network. what he found and ran that simulation, you know, a thousand times, he got what's shown in the blue lines here. And you can see that it corresponds quite nicely to the path that Jesse Bloom had worked out, except in one place where perhaps uh if the language model had been used to help guide the predictions, uh there might have been a different pathway. The thing about this approach is that because you're assuming that the project the uh movement of uh evolution direction of evolution is indicated by these individual vectors. What it allows one to do is to order the different sequences in what's called pseudo time. That is the relative time for the transitions such that those sequences that were early in evolution are indicated in this darker uh shading and those that were most recent are indicated in this light shading. So you can actually order the sequences in pseudo time. The other thing that this type of analysis allows you to do is to actually go back and predict the roots of the evolution evolutionary pathway. And it turns out the predicting roots of um phoggenetic trees among evolutionary biologists is is often a difficult problem and sometimes a controversial task. But this is a way of uh without bias just going backwards and saying where are the roots of the evolutionary of the phoggenetic trees. And I'd like to highlight how this uh really could be used using um uh to identify roots using the influenza pandemics as an example. So over the past hundred years there have been uh four major outbreaks of uh influenza pandemics which have caused uh in some cases enormous numbers of deaths. This 1918 flu uh pandemic caused over 50 million deaths and that's when the world was a lot smaller than it is today. And the way in which these pandemics occur is that a virus that's in an animal jumps into humans and starts to infect humans. And so for influenza, this is particularly a problem because influenza virus has what's called a segmented genome. So each of the different genes are encoded by a different piece of RNA and their individual segments inside the virus. And when a person gets infected with um a seasonal with a normal a human strain of influenza shown here in red, but then also gets infected by a virus from another species such as a a bird here and so they get co-infected with the two viruses. What can happen is a process called reassortance in which the individual segments of the influenza virus uh reassort such you end up with a virus which is almost all human adapted but which contains one uh one segment which is from the bird. for example, if it corresponds to the hemoglutin and because the humans have never seen this virus before and the entire uh community has no pre-existing immunity, what this virus can now do is grow out of control in a pandemic. But the point is that this is a jump. It's a jump of a new virus into the humans. It's not evolution of a pre-existing virus that we that is circulating. If you take uh the sequences of viruses and you use the standard phlogenetic evolutionary tree and you take the uh outbreak that occurred with H5H7 in uh the 2010 time frame, what you end up predicting is that this virus evolved from a pre-existing H1N1 virus that was already circulating in humans. Okay? But of course, that's not what happened. As I said, it's something that jumped. If you use evil velocity to analyze the roots, what you see is that you can identify that the roots of the 1918 pandemic were around here. The 1960s pandemic was around here. And this, which fortunately didn't turn into a pandemic, but was an outbreak. This H5H7 occurred as a separate route over here. And so this is a demonstration of the power of using evil velocity to identify roots and to identify correctly that this was a new jump and not the evolution of a pre-existing uh virus inside um humans. He also looked at SARS Kovv2 and importantly uh EVO the the uh ESM1B language model was trained with protein sequences before the pandemic. Okay. So the model did not actually see the sequences of the spike protein that I'm showing you here. But what this shows you is a UMAP projection of the sequences of the spike protein from SARS KV2. Over on the right hand side is ordered in real time. This is like the first 18 months of the pandemic and this is in real time. So what you see is you start here with wild type dark black and then what happened as you as many of you know is you got the D614G variant that occurred next that turned into this dark purple. Then uh we went up to B.1.1.7 which is the orange and then we went to B16.7 as the most recent which is the yellow. And if you look at the pseudo time ordering using EVO velocity what you see here's the uh pseudo time. Oh excuse me I did that backwards. This is the real time. This is the pseudo time. So I did that backwards. In real time you start wild type go 614G you go to 117 and then you go to 11 1 617 and in pseudo time what you see is that you reproduce that order of the evolution of the SARS KB2 spike protein. So there's a pretty good correlation here's the spearmman between real sampling time and this is all the sequences of the COVID spike protein versus pseudo time. Even more remarkably I think is if you go to cytochrome Cytochrome C is a classic model in protein evolutionary biology because it's such a it sequence is so highly conserved among ukariots and if you analyze the sequences of pro of cytochrome C what you see is again you have evo velocity predicting very strong directionality of velocity and if you look at the uh pseudo time plotted as the y-axis here versus the taxonomic classes going from ukareots to to mammals. What you see is that you're getting the pseudotime ordering uh in the correct order. And indeed, if you do that and you look at that, what this is saying is that EVO velocity is able to predict the pseudotime progression of the evolution of cytochrome C over 2,000 million years of evolution. So it's really quite remarkable uh I I thought at the time when Brian got these results that just based on the amino acid sequence of the protein and the assumption that the uh language model is uh able to give you evolutionary direction that you're able to uh reproduce things such as I've just shown you. And I want to emphasize it's done from protein sequence alone. No other information went into the analysis than what I just told you. So then Brian asked the question, if Evo velocity can reconstruct evolutionary trajectories from the past, could you actually use it to predict evolution moving forward? Right? Could you actually say here are uh the my predictions for what's what would be better and guide laboratory evolution? And it turns out that the answer is yes. Here is an example using a um an anti the what's called unmutated common ancestor or UTA of an anti of a monocone antibbody. So this is essentially the germ line of the monoconal antibbody the starting point for the uh where this monoconal antibbody started. This monoconal antibbody mab114 happens to be a monoconal antibbody against the Ebola virus that is actually an approved FDA therapeutic and this is the germ line for that antibbody and if you use evo velocity to predict or the language model rather to predict the top 10 substitutions to try. So you ask the language model in the entire sequence of MAB114 where when I cover the amino acid where does the language model tell me that there's another amino acid that belongs there and it says so with a very high likelihood where the top 10 likelihoods of that and then if you make those 10 point mutations you can't see it very well here because the scale is so high but if you make those those top 10 m predictions roughly uh four of them I think are better than wild type. And then if you make combinations of those four and you test them, what you find is that one of the combinations is 160fold better affinity than the starting point. So with a remarkably low throughput screen using just two rounds of evolution in a laboratory he was able to take an antibbody uh precur an antibbody and to evolve it to be 160fold better. He went on to do this with several other antibodies including uh several of these that were FDA approved and in some cases these are antibodies that had been uh created by professional antibbody engineers and evolved uh first in a in a person through sematic hypermutation and affinity maturation but then secondly by protein engineers and in each case using the same strategy of two rounds of evolution solution testing less than 20 new variants. In each case, he was able to improve the affinity of these antibodies, including several which are FDA approved, ranging from 1.7fold up to the 160fold that I showed you. So, it actually worked again in a very low throughput uh and highly efficient manner. So, what do we think is going on? Well, if you consider on the left hand side all the possible mutations of a protein. So this is the full mutational space in white and you consider the high fitness mutations. So those that are going to be better in in the case that I just showed you higher affinity for example. What we think is and and then some there's going to be some small subset of this full mutational space that are going to be plausible mutations because most of the mutations in this mutational space are going to lead to proteins that are not fit. They're going to be unfolded. They're going to aggregate. There's going to be all sorts of problems with most of the proteins that you create in this space. But some of them are going to be plausible. And those that are plausible will include those which are high fitness. We think what the language model is doing is the language model is telling us what sequences have a high likelihood of being plausible mutations. So it's allowing us to not sample this mutational space which is going to lead to bad proteins but it's saying instead here are the plausible mutations that you can try. So severely restricting the space such that when we test those plausible mutations, a high a high fraction of them are actually uh going to be better. Or to put it another way, you know, if you're one if you're if this is the uh ramp for plausible mutations, random guessing, you're going to be guessing all sorts and the likelihood that you're going to actually go in a direction of a plausible mutation is very very low. Most of the time you're going to fall off the ramp. But if you have a guide as to what's plausible, you're going to very efficiently climb up this ramp as you move forward. So that was quite a remarkable uh finding as well. Namely that using just the amino acid sequence of the protein, you can actually predict with high u efficiency mutations that will improve the protein's fitness and that when you combine them, you can evolve things to be uh much better. So the next part I'd like to tell you about is um work to use language models. But the when I've just been telling you about language models where you cover for example you mask an amino acid and you ask the um the model to tell you what amino acid belongs there. The way in which these language models uh work if you look under the hood is they look at what the uh computer scientists call attention. attention maps. And so if you ask what amino acid belongs here, the way in which these models are able to predict what amino acid belongs there is of course based on all the other sequences, all the other residues in the sequence. and and some of the residues say here residue five uh if this is the attention map as it's called some of the some of the uh residues are attended to by residue 5 to different degrees. So some amino acids are more important for predicting what residue goes here than other amino acids. And it turns out here on the bottom that if residue 5, let's say, attends highly to residue 3, uh, often it's because residue 5 and residue 3 are in contact with each other. Okay? And this is inferred even though you don't have structural information. You're just using a sequence of all of these proteins, but the model somehow figures that out. And indeed um the Facebook group showed early on that you can use these attention maps to actually predict the contact map for a protein. You can predict which residues are near each other. So the remarkable finding here from these guys was that just using the amino acid sequence of a protein in a trained model looking at where the attention is you can predict the contact map. And now people have gone on to use this to predict the structure of a protein based on the attention maps. Okay. So the information is there. What a graduate student in the group goal Kanan did was he asked a question for those situations where you have high attention but they're not contact maps. Could that be due to alossteric uh interactions? And so alsteria as many of you know is when uh uh ligan binds on one part of the protein and it affects the uh activity of a catalytic site. So it's action at a distance communication at a distance. And so the hypothesis was so here I've already shown you that residue 5 attends to residue three and that is explained because the two residues are in contact with each other. But what if you have a situation where residue one attends to residue 7 very highly and residues one and residue 7 are not close to each other. Could that be because residues one and seven have an alossteric relationship? Okay. And so what uh goal's hypothesis was was that when the active site attends highly to a particular residue the hypothesis is that that residue might be alossteric. Okay. So to test that hypothesis uh what goal set out to do was first to create a data set an alossteric protein benchmark data set and he did that by going to a collection of proteins that had already been published um uh in the literature in 2025 a benchmark called Alobench and he went through that large set of aloseric proteins and he he went through with a very strict criteria. He said where do I have an alossteric protein that I know the threedimensional structure and I know the threedimensional structure of an alossteric modulator a small molecule bound to the protein. Okay. So these are examples where we we refer to as the gold standard of alossteric data sets because we know where the active site residues are and we know exactly where an aloseric modulator binds due to a high resolution crystal structure. And then what he did was to say for those active site residues and for the alossteric uh modulator sites uh he created the the benchmark data set and then he scored the attention uh between the active site and every other residue in the protein. And so for example here with this if we had a catalytic histadine diad what he did was he would score the attention to those catalytic residues for all the other residues in the protein and then he would rank them by their attention. So for in this case against the catal diad residue M2 is high and residue P3 is high and so they would be up here high and his prediction was that these would be the high these high attention uh residues would be alossteric and in some cases it works amazingly well. So here is lactate dehydrogenase which is a classic tetromeic alossteric protein and uh the active site residues are indicated with these green spheres and then the alossteric prediction the prediction of alos of uh the attention. So the attention going from low attention to high attention is colored on the protein from uh yellow to dark purple. And you see so these residues out here are low attention uh and these residues here are higher attention and then these are even higher attention. The spheres the colored spheres are the known aloseric modulator contacts colored by this same attention scale. So what you see in this example which looks really good is that the residues that are far away from the active site that score very highly in tension turn out to be exactly those residues that when you look at the crystal structure are making contact with a small molecule alossteric modulator. So it worked really well here. It didn't work everywhere. But with this benchmark set of 109 different proteins that he made, he was able to compare it to the two uh best models for predicting alister. One of them is a structure-based model called M. The other is a uh model based on multiple sequence alignment called EV couplings. And so each dot here is a different protein and it it's we're plotting the uh AUC the area under the uh receiver operator curve. Uh and so higher score is uh better. And what you see is this is the score for the language model. This is a score for uh the structure-based prediction of alisteria. The best prediction for structure-based which is called M. And each dot is a separate protein showing you the the score on each case. And so in blue are those cases where the language model does better at predicting the alossteric site than uh the M and in gray is where M does better than language model. What you can see is that with high statistical significance uh the language model is doing better than M. And here on the right hand side is the same comparison using a co-evolutionary analysis called EV couplings. Again, blue and gray. And here we're we're still statistically significantly better than EV couplings, but not as much as so as with M, which probably makes sense as co-evolutionary analysis is close closer to what uh the language model is actually doing. It's looking at the relationships between residues. So it looks as though using this language model approach, focusing in on attention, we're able to actually come up with a method for predicting alossteric residues in proteins that is better than the state-of-the-art uh prediction methods that have been out there and that has many practical implications. The most important of which is that in drug discovery to be able to get high specificity targeting alossteric sites versus targeting a catalytic site is often a very effective strategy because when you target say the active site of a kynise you're going to inhibit a lot of kinases but if you inhibit an alossteric site of a particular kynise you're much more likely to have a specific uh interaction and indeed um the type of protein that is most highly drugged uh with current drugs are G-proin coupled receptors seven transmembrane groin coupled receptors or GPCRs and it turns out that in this case there's a GPCR called a beta 1 adinuric receptor which is the classic uh ad classic uh GPCR the adrenaline uh receptor and what this group here did was an alamine scan of the beta Beta 2 adinuric receptor. So they replaced every amino acid in in beta 2 AR with alanine and then they measured the shift the signaling response as a function of agonist concentration. And what they observed and this was known was if this is a wild type uh receptor some of the mutations caused a shift in the potency that is it became less effective. it required more agonist to get uh the response but you still got a full response and they called that potency shifted. Some amino acid changes caused efficacy shifts that is you got approximately the same agonist concentration to cause the shift but you didn't get as much of a signaling response. And so here is a plot of each of these dots is a different substitution to alanine of a particular residue in beta 2 ainerturic receptor. here plotting uh the normalized amplitude that is whether it's efficacy shifted or not and here plotting the log of the EC50 that is what is whether it's potency shifted and what you see is that most of the alanine substitutions don't affect things it leads to wild type like signaling some of the amino some of the alanine substitutions lead to low potency some lead to low efficacy and some lead to both low potency and low efficacy the color of the dots here. So that's that's the data from from this paper. The color of the dots here is what goal added based on attention. So those residues which had low attention to in this case the adrenaline binding residues known from high resolution structures that had low attention to the adrenaline binding residues are shown in yellow. And you see there's a prepoundonderance of yellow over here. And those with high attention to the uh adrenaline binding residues are shown in dark purple. And you see many purples out here. Even more remarkably, if you ask what are the top five candidates with attention to the adrenaline binding site, those five are shown here. One, two, three, four, five uh dots. And here this is segmented into those in the first quadrant with wild type se signaling. and those with altered signaling. And what you see is that four out of the five of the top scoring uh residues are in the altered signaling class. And in fact, they are um all four of these are what are called switch residues in the GPCR field. And switch residues are residues that change their confirmation between the activated and the inactivated state of the GPCR. And so for example, this uh this right here is a residue that is part of the switch called NPXXY. It turns out it's the N of the NPXY in the seventh helix residue relative 49. This is part of the DRY switch and this is a D of the DRY. Uh here's the W of the CWXP switch and here's the P of the P IF switch. And so it identified residues that are known to be critical in the confirmational change for this G- protein coupled receptor. So again uh what we're able to show is that the language models are able to predict pre-existing known aloseric relationships. And so what Gokul then did was to ask whether or not we could use this whether he could use this method to actually predict unknown alesteric residues. And so he did this first with a protein called DPP4 depal peptidase 4 which is an important diabetes target drug target. And um he identified the top. So here's DPP4. It's a dimer. in green are shown the catalytic residues in that dimer. And he predicted he looked at the top five residues uh with regard to attention to these catalytic residues in the protein. Only two of those five expressed well and those two were this serarine here and this thrienine here. He made those substitutions to alanine and studied kcat km and kcat over km. And what you can see is that both of those individual alanine substitutions change KC cat one of them changes KM and both of them change KC cat over KM. So basically he was able to take a protein which was not known to be alossteric and to use this method to identify residues that when mutated to alanine change KC cat over km uh with a single alanine substitution. He then went on to look at a known alossteric protein and this known alesteric protein is ACE2 um uh which is become famous because of um COVID because it's the receptor for the COVID um spike protein but it's actually uh angotensin converting enzyme 2. It's involved in blood pressure regulation. It's known to be an alossteric protein. And here in this protein in red are the catalytic residues. In uh purple are the top five putitive alossteric residues as predicted by attention and in green is where chloride binds. Chloride is a known alossteric modulator of ACE2. Okay. And so what you can see again these are residues that have attention with respect to the active site highest attention. There were five he took the top five of these predictions. Of those top five um one of them uh one of them did not uh have activity but the other four did have activity when they were substituted to alanine. And what you can see here is that when you plot kcat over km for each of these four mutations, three of them have a statistically significant difference from wild type in kcat over km. But more impressively, when you look at the effect of chloride ions on the activity here from the wild type, it's shown in black. What you see is that one of these mutations shifts that chloride sensitivity, that chloride coupling dramatically to the right and another mutation shifts the chloride sensitivity substantially to the left. And so again using just five um substitutions and studying the top four he was able to identify two substitutions that have a significantly altered chloride response curves in this alossteric protein. So um it's quite remarkable that these language models are able to allow us to track evolution to guide evolution moving forward and to uh predict alteric residues. What's truly, I think, amazing about this all is that everything I've been telling you requires only the amino acid sequence of the protein. You don't have to tell the computer what the protein is. You don't have to tell the computer what it structure is. You don't tell have to tell it what it binds to. The only thing you give it is the amino sequence, acid sequence of the protein. And there's sufficient information that's been captured by the language model when you fed it these millions and millions and millions of protein sequences that it can do what I just showed you. But of course I'm a structural biologist and I do believe that sequence determines structure and structure then determines function. So the question that Brun Shanker set out to ask is if we add structure to what we're doing here can we do even better? Okay. And for that he turned to uh what's called inverse folding. You're all familiar with uh structure prediction based on for example alphafold where you take take a multiple sequence alignment. So you make take many sequences that you align them and you use that to predict the threedimensional structure of a protein. What inverse folding is is the opposite of that. You start with the structure of the protein and you ask what sequences can fold into that structure. Okay. And this was first done by uh Bill Degrad using um rational design principles to design a poor helix bundle. Uh Steve Mayo's group at uh Caltech was the first to do a computational redesign of a native structural motif at high resolution. And then Pier Harbory when he was a graduate student in my lab at MIT was the first to do a computational high resolution design of a protein fold that had never been seen before in nature. He designed at high resolution a protein called RH4 which is a coiled coil with a right- hand superhelical twist which had never been seen before. All the coiled coils in nature before had been leftand superhelical twisted. And in each case what uh what was done was to take the structure that you wanted and to make a sequence that folded into that structure. And so what Verun set out to do was take advantage of another language model called ESM if or inverse folding one which is also produced by the Facebook group. And what this language model did was to take amino acid sequences of proteins but it started with amino acid sequences of proteins that had very good highresolution crystal structures 16,000 of them. And it inputed not just the sequences of those proteins, but it also inputed the Calpha coordinates, the the backbone coordinates of that protein for all 16,000. It trained a model based on that. And then it added about 12 million sequences that were predicted by alpha 2 and the sequences of those 12 million proteins in a non-redundant analysis. And so basically they trained a model that uh would use not only the sequences of proteins but also the backbone coordinates of the structure of the protein. And so what Verun set out to do then was to ask whether he could use this approach of inverse folding to now ask when you mask a a protein uh residue. And you take into account not only the sequence but the backbone coordinates to predict the likelihoods of what sequence what amino acid residue should go here given the constraints of the C alpha coordinates given the constraints of the structure to predict what's going on. And so what he did was to do that and to ask whether or not he could um actually predict evolution going forward. He work first looked at known evolutionary data sets. This is a data set um from Phillips and Desai and it's a data set where they did exhaustive deep munogenesis deep scanning immunogenesis of uh an antibbody called CR 9114 which is an anti- flu antibbody. It turns out that this antibbody has 16 substitutions from the germ line from where it started as com. And so there's 16 substitutions in the heavy chain. And so what this group did was to make all combinations of these 16 uh substitutions. So that's 2 to the 16 65,000 different uh mutations. And they measured the um on this axis they measured the uh affinity against the H1 uh hemoglutin and against the H3 hemoglutin. So there's 65,000 dots here. Each dot is a different uh mutation. uh uh which is a v variant of these two to the 16 uh sequences and uh on the y ais is the log likelihood predictions using the structure in form the inverse folding language model and on the x-axis is the experimentally determined affinity and in blue triangle is the starting point the germ line and in green is the mature antibbody and what you see is that there's a very good correlation here between the affinity of the antibbody and the uh in structure informed language model prediction both for H1 and for H3 uh in terms of using this approach when you look at and and what you can see here is that uh those well you can't see it here those correlations are shown here and the correl the if you the correl correlations that I just showed you are this 65 and 0.5 for H1 and H3. And importantly, this is when he used the structure informed language model using the structure of both the antibbody and the antigen. So he used both the antibbody and the antigen. When he used just the structure of the antibbody, he does significantly worse than when he uses the combination of the two. And the remarkable thing about that finding is that the inverse folding language model was trained on monomeic proteins. It was not trained on complexes. And yet what he's finding is that when you use the complex, you're much better at predicting the two to the 16th variant correlation that I just showed you on the previous slide. And so it's as if uh so incorporating the structure of the antibbody and the antigen improves the performance and it suggests that the structure informed language model can implicitly learn features of binding from the structures of complexes which is really a pretty surprising result for us that the complexes actually do much better. And so what we think is going on here is that again if this is full mutational space, this is the high fitness variance here and this is uh what's plausible. If you use just the antibbody sequence, you do pretty well. But if you use the antibbody sequence plus the antibbody backbone uh you do better. But if you use the antibbody sequence plus the antibbody backbone plus the antigen sequence and the antibbody backbone. If you use structure and sequence of the complex, you actually narrow things down much more effectively. So again the question was can we use this method to evolve proteins moving forward and for that uh verun turned to uh an antibbody called beeplovamab. So during the pandemic there were six antibodies that were FDA approved and as the variants came along uh one by one the antibodies lost their emergency use authorization from the FDA. The FDA pulled them off the market. The last antibbody standing was an antibbody called Bealova. It was an Eli Liy antibbody that had been highly evolved. But then here in December of 2022 when BQ.1.1 came along, beepovam lost its emergency use dose authorization. So Verun took beepamab and he used the structure informed language model to predict here the top 10 or 15 mutations that would improve it. He tested them as singles and what you see is that approximately half of them do better. uh and here he's measuring neutralization directly. He's not measuring affinity. So he's measuring virus neutralization and about half of them do better. And then when he made combinations of these that did better as a single amino acid substitution, what he was able to do was identify a variant that improved the neutralization potency by 25fold. As you can see here, bringing beepomide, which was no longer effective against BQ.1.1 into the range where it was be uh clinically effective against BQ.1.1. And I want to emphasize that the screen that he used here and the data I'm showing you is for neutralization. It's not for affinity. And the reason why I want to emphasize that is when one does huge screens, it's difficult to do a huge screen an assay on the final activity. So if you if you made 10 to the 3 10 the four different mutations, you'd have a hard time measuring neutralization for all 10 to the three to 10 to four. So instead what people do is they use a surrogate first say affinity and they find things that bind tightly and then they take the ones that are tight binders and they go now in a lower throughput manner to neutralization. But because this is so efficient and you don't have to make more than 30 antibody sequences to find something which is good. You can go directly to the assay that you want. And in fact, when he took these mutations and he looked at the affinity, what you see plotted here is the neutralization activity and the affinity change. And what you see is that about a third of the mutations actually have a higher affinity than the winner. Okay, so about a third of the mutations have a higher affinity. But in fact, by screening directly for neutralization, he was able to see that this one was actually the one that was best. So the reason why that's important is we think that the killer app for this sort of uh uh use for this sort of method is one in which the final assay is very cumbersome. Let's say you had a final assay which you had to use a mouse model to find out whether or not the mutation was good, right? Well, you can come up with your surrogates, but at the end of the day, you have to go to the mouse to see if it's better. here with 30 different antibodies, you could actually go directly to the mouse model and see which ones actually work. And so it's a because it's so efficient in low throughput manner, you're able to think about uh screening applications where you can go directly to the final uh answer. One more thing I want to say about beepovam. This is the structure of beeplovamab in ribbon shown bound to the receptor binding domain the RBD of SARS KB2 and in the the spheres indicate those mutations which I've been talking about. What you can see is that many of the spheres are nowhere near the receptor binding domain. Many of them are far away and they're in what are called the framework regions of antibodies and such framework regions are normally left alone by protein engineers. What protein engineers do is they mutate the CDR loops which are in contact with the RBD. And yet you can see that many of the mutations that were helpful were outside of that and would never have been predicted by the protein engineers. But here's the here's the more important point. in in pink are those amino acid changes that improve the neutralization activity against the original Wuhan strain of SARS KB2 and in blue are those amino acid changes that improve uh the neutralization potency against BQ.1.1 and they're different. Okay. So what the language model is doing is it's telling you which amino acids are plausible which ones to try and then if it happens to be one type of vir one strain of virus one set of mutations might work better. If it's another set of another virus another set of mutations might be better. Or if you want to think in terms of evolution you have the plausible mutations. If the world gets warmer maybe evolution chooses these residues. If the world gets colder, maybe it chooses a different set of residues. So, it's telling you what's plausible, not what is going to be right for any particular circumstance. And again, what we think is going on here is that if this is the full mutational space, uh the structural restriction is is limiting the space to those which are structurally tolerated and then the language model is telling you which ones are good uh to use forward. So the last point I want to make is that we think that this is not just for antibodies uh but rather it's a general method for proteins. Here is um an example where this group here Brennan Johansson carried out a deep mutagenesis experiment on map K1 which is important uh cancer target. And when Verun runs his protein structure informed language model and he picks the top 10 predictions based on that language model using the structure of map K1 and he asks in the for those top 10 predictions what fraction of them are in the top 20th percentile of this DMS data set. What he finds is that five out of the 10 are in that top 20th percentile. In fact, he finds that two out of the 10 are in the top five percentile of this exhaustive deep mutagenesis experiment. So, it's able to predict in very with just 10 predictions mutations that are very good candidates coming out of this deep mutagenesis data set. And when you look across a variety of different proteins here on the bottom is the list of proteins that have deep mutagenesis coarance and also crystal structures. And you see they include enzymes, they include binding proteins, they include ubiquitin conjugating enzyme proteins, different types of enzymes. In blue is is what's shown when you use the structure informed language model. In gray is what happens when you use the language model without structure. But what you can see is that in many cases the not always but in many cases the blue is doing very well at predicting the high scoring percentile scoring uh mutations that came out of these two mutenesis experiments using just 10 uh predictions uh efficiently. So just as the language model has been trained on all sorts of different proteins, we think that what I've been talking about today is applicable to all sorts of different proteins and indeed people are using these methods now to try and uh in the laboratory evolve proteins with all sorts of different activities. And so I talked I've mentioned the people that were uh primarily responsible for the work that I talked about today. Brian, he is uh with the guy who got us started on all of this and he's gone off. He's now uh got his own appointment as a faculty member at Stanford is going on to doing all sorts of wonderful and crazy things. Um Gokal and uh Verun are two fantastic graduate students who together with Brian carried out the work and these are the other members of my group that are doing uh terrific stuff. With that, I'll stop and be happy to answer any questions. Thank you for the talk. I was um a little bit intrigued by the part where supplying the antigen structure helped the uh prediction better and um briefly you mentioned about some enzymes at the end of your talk and I was thinking if um for example the purpose of biocatalysis or sometimes antigens are not proteins. So if we supply maybe the chemical formula like smiles would that help the prediction better in a similar manner? >> That's a great question and I don't know the answer but you should you should do the experiment. Uh that would be very interesting actually if uh a small molecule uh did help uh with that. Um one thing we are playing around with is truncating the antigen so that we use not the whole antigen but part of it. And uh what we're finding is that you don't need the whole antigen to get that benefit. So I think I would guess that you might be able to get a positive result there, but that would be an interesting thing to try. Yeah, you should try that. Right. I should say everything that I've talked about. Well, the Alistister is about to be published, but we we make all our code available uh once we publish. Yeah. So you could you could go do the experiment, right? >> Uh great talk. Um two small questions. The alististeria results were very interesting and um it seemed like in the aggregate the the language model outperforms like M or um MSA best based methods but there were some cases in which one performed better than the other or they disagreed. >> If you look at those examples do you get any insights into like what types of aloseric mechanisms are really strongly captured by attention versus other kinds of >> um mechanisms. >> Great question. Uh we we haven't I should I should have I should tell Google to do that. We we haven't really looked at that very closely. Um but it is true that we've looked for example are there are there interface alteries that are better predicted than ones that are not and we we haven't been able to find a correlation. We haven't done though uh a careful analysis in terms of types of alisteric proteins which I think would be uh maybe perhaps along your lines. So far we haven't found a a distinguishing feature. I I think it's fair to say predicting Alistister is really hard. Um and uh you know we're still far from perfect but we're we're better than than any other method right now. Yeah. But there are still those examples, right? >> Absolutely. Yeah. Um and then the other small question I had was there are drugs like gleeback for example which exploit sort of differences in alossteric potential of like sarkc versus able and I wonder if these models >> like is there do they predict those types of sites to be alossteric regardless of the the rest of the protein or does it actually potentially tell you um kind of like local hotspots in evolutionary space? >> Good question. Uh I don't know the answer, but I do know that uh SARKC does do pretty well with this math. So you could go go take a look at that. All right. Yeah. >> Hi. Um so I was also curious about some of this kind of alossteric prediction things. Um I think either last year or the year before I think it was Amy Go's lab published a really interesting paper on also using these inverse folding models to parse DMS data sets. And what her lab did was um run I think inverse folding scoring across two protein structures for the same protein. And I was curious if you'd tried also that in combination with these evolutionary scale models and the inverse folding >> to see if you could better predict like certain protein dynamics or engineer some. >> Yes, that's a good question. I think that those are interesting results. Um using uh basically looking for residues that prefer or bias things towards one confirmation versus the other. Uh extending that to the Alistister is is analysis is is non-trivial. Um but it's something that somebody should figure out how to do. It's a good question. Yes. >> Fascinating talk. Uh I was curious for the initial evolution results if um when applying to the already evolved like human evolved proteins if these models had to backtrack in certain ways to then find a different ma uh maximum point in the fitness landscape or are they only able to scale monotonically towards the local maximum in the fitness landscape? >> Yeah. Uh very interesting question. Um, so far it looks as though uh it's just going forward up until present time. The the analysis is is complicated by the fact, full disclosure, the analysis is complicated by the fact that we're only able to use extant sequences. So we can only look at the sequences we know about. We don't really know what the sequences were 200 million years ago, right? But we know the sequences of the proteins that are in that classification today or in recent history when we sequence things and we're using those sequences. And so um that's that's a caveat here to what's going on. But to the answer to your question, it looks like it's still progressing up until uh present day. And what's interesting is it looks like we can go future uh as with with the evolution at least. Um, yeah, I think that's probably all I can really realistically say at this point, right? >> Yes. Hello. Great talk. Um, so traditionally, prior to these language models, if you wanted to sort of cut down your mutation list to find um more rational mutations, you might use an MSA where you look for conserved residues and maybe residues that um are have equal conservation to the wild type residue. Yes. >> Or or EV mutation where you might take multiple sites. Do you find that the language model which is sort of MSA it doesn't know the MSA of the protein but maybe has captured that information. Do you find the language models outperform those approaches >> or um maybe they perform similarly? >> Yeah. So first of all I think the language models um have incorporated the MSA information because they've learned so much from all the sequences. But um here uh I didn't I didn't spend time on this. The MSA methods are in purple and so these are antibbody MSA methods and you see that they don't do well and the language models alone in in this case the language model alone that's sequence only but this is a language model which has been trained for antibodies are are there and what you can see is that you know really the um when we especially when we use the complexes the structure informed language models are putting us in a completely different category of of what are able to do, right? >> Why why do you think the language models perform better than these like MS? I know that's sort of philosophical, but I >> I think they're learning much more than MSA, right? Um I think the MSA is sort of the of course if as there's co-evolution, it's learning that, but it's all incor but then it's learning much more than that. Um yeah. Yeah. And and much more distant sequences as well. Yes. Yes, John. >> Yeah. Very very interesting talk. I had a question about predicting perhaps co-evolution. >> Yeah. >> Uh of protein protein interactions. >> Absolutely. >> And especially if you think about virus surface proteins that you can be projecting into the future whether you might also be projecting what the receptor might look like in future hosts. >> Absolutely. And um that that's definitely a very interesting question. In the case of virus hosts, you know, one also has to take into account the immunity aspect of things and development of antibodies. But in fact, Brian independently has gone on to do some beautiful co-evolution work um using uh systems which are uh bacterial toxin antitoxin uh where you have really nice data sets in the literature to look at and um uh and it works right. Yeah. Yes. I'm curious about kind of in the best examples you have of the alistister predictions are you able to see almost residue level um binding or I noticed you were mostly plotting kind of like rankbased metrics there um but in those really good examples are you able to see >> the the answer is yes and the the the example I I took out a couple slides but the example that I highlighted where the switch residues in the GPCRs, those are particular amino acids that have been beaten to death by the GPCR groups uh because they're so important and they're known both from through mutagenesis as well as through analysis of structures of different GPCRs to be what they call the switch residues. And and these switches are combinations of typically three to five amino acids that all change confirmation in concert. and it was able to identify specific residues in those switches that are responsible for the alister. Um it's also able to identify residues in the pockets of alossteric modulators. It doesn't identify all of the residues in the pocket and we actually don't know if all of those residues are are getting if you will alossteric signals from the from the catalytic site but often at least one of them is very strongly predicted right and so could it be used as a way to uh predict pockets that might be potential drug targets we think yes we hope so right now Yes. >> So, I'm curious about your uh Bea, sorry, >> yes, >> those framework mutations um and I guess why you think they're increasing potency and then kind of on the other side if you can look at why they might be not evolutionarily selected for, for example, in a host, are they polyreactive? Are they destabilizing the B cell receptor? um and and if that you know those types of predictions would also be helpful. >> Great questions on poly reactivity. We we are um actually quite paranoid about that. We've tested all our antibodies and they are not polyreactive. So these affinity increases or these activity increases that we're seeing are not due to making the thing stickier. Uh they're due to real changes in in activity neutralization with with respect to the framework mutations. FRA it framework mutations are are known to occur during sematic hypermutation and affinity maturation and they change the angle uh the angles the orientation of the VH and the VL and that leads to changes in affinity and this has been well studied unfortunately only in a couple cases um there's one case where uh Pete Schultz studied it with a haptinbinding antibbody another case where Steve Harrison studied it in HIV evolving antibbody but there substitutions in the framework residues that change the orientation of the VHVL and increase increase the activity. So it's not as though these mutations don't occur in nature. They do and you can see them when you look at phoggenetic trees of of antibodies. Where they don't occur usually is when you hand that antibbody to a protein engineer and you say make it better because the last thing the protein engineer is going to do is change residues that are uh 30 anctions away from where the res where the antibod is binding. they're going to change residues that are binding. And so what my point was that these are uh residues that would not have easily been found by your typical protein engineer even if they were doing exhaustive deep munogenesis. You just don't have enough u uh combin you don't have enough uh room to make that many combinations. Um it's these framework mutations are really fascinating if if you use prediction programs to predict the structures of antibodies like alpha fold or rosetta fold. Uh you can predict the structure of the VH and you can predict the structure of the VL very well. What you can't predict very well are the CDR loops because they're the parts that are binding. But the other thing you can't predict well is the orientation. And so uh the the framework mutations are actually an important part of antibbody structure and evolution, but they're they're just not captured by uh your traditional protein engineers, right? But they are captured here, right?