Voices in the Code: A Story about People, Their Values, and the Algorithm They Made

Um, I'm very pleased to welcome David Robinson today, who is going to be giving a keynote talk on Voices in the Code, a story about people, their values, and the algorithm they made. David Robinson is part of OpenAI's safety systems team, where he focuses on building trust through shared understanding of OpenAI's technical safety work. Throughout his career, he has focused on interdisciplinary sociotechnical research and practice. Earlier, he built the policy planning team in OpenAI's global affairs organization, served on the faculty of Apple University, and co-founded the civil rights and technology nonprofit Upturn. So, welcome David. We're very happy to have you here. Um, the format will be he will speak for a while, and then at the end we will have time for questions and discussion. So, please stay muted throughout the talk, and then there will be the ability to ask questions at the end. So, thank you so much for being here, David, and please go ahead. >> Thank you, and thanks, Emily, and and thanks to everyone here. Um, it's an honor to be with you. Um, I uh am excited to be sharing this in a context with statisticians because uh biostatistics are are are are important in the story. I will confess up front that uh my own background is in uh law and policy, and so the way that I came to this was a little bit oblique. I'll just say before we begin in earnest, I had been interested for a long time in how uh algorithms that make important decisions about people are governed. How do we know that they're working correctly? How do we know what values are inside of them? Uh and so on. And I actually was was teaching a class where some students identified the organ transplant system as a case study. Um, and uh the the presentation you're about to see on the book that it describes are downstream from uh those students not wanting to write a paper about that, but uh me getting very excited about uh this topic. So, uh let's get right into it and and and I really am looking forward to the question and answer and conversation piece. So, um you know, uh you know, and definitely like uh like to to get into that. So, okay, uh let's get started. So, our story actually begins at this airport Marriott in Dallas, Texas on a chilly morning in February of 2007. A crowd of several hundred people is gathered in a conference room inside for the first preview of a new and life-saving software product. Something that's been in the works for several years already at this point. Uh these people have come together to consider a new version of the algorithm that whenever a donated kidney becomes available anywhere in the United States, uh this is the system that decides which out of the more than 100,000 waiting transplant patients will get an offer to their transplant team that uh amounts to a chance to receive that organ. Uh the previous generation of this software is already in use and saving many lives, but people in the room are harshly critical of what's already been shipped. They think that this time they can do better and they're passionately dedicated to getting things right both ethically and technologically. And they know that in order for their efforts to succeed, a large number of non-experts, people who are busy living their lives, are ultimately going to need to understand and trust the way that this system works. So, um the uh the the background here uh perhaps familiar to some in the audience. This is a moral challenge that really began with Teflon. Uh the uh the synthetic plastic which is the material that is used for the tubing in this diagram of a shunt that can be left in someone's arm. So, uh, kidneys, uh, filter the blood. Most healthy people have two of them. Dialysis is the name for filtering the blood outside of the body by machine. Uh, the very first time that was done was actually during World War II in Holland. Um, but there was a problem, uh, which was that, uh, you needed a large bore needle to get the, uh, the blood in and out. And so, uh, much the same way that injecting drug users eventually run out of injection sites, uh, people after about a month who who needed dialysis, they could be they could be kept alive, uh, and really revived from an a deathly ill state, uh, but only until, uh, the they ran out of places to put the the needle in. This tube, uh, this this Teflon tube changed everything because it can be left in permanently and used repeatedly to get blood in and out of a patient's body. Um, and when this tube, uh, was pioneered, uh, it was in the office of a man named Belding Scribner, a physician in Washington state, who, uh, suddenly found that because of this technology, he could keep alive any kidney patient, uh, that was in kidney failure, essentially, or many of them. Uh, but, uh, he only had four machines to do the dialysis. So, he was inundated with calls from physicians, nephrologists around the country, who who wanted their patients to have access to the technology. Um, uh, but of course, uh, the doctors, uh, orientation is to fight for every patient. And rationing care is really a moral challenge. And so, Scribner decides to do something pretty radical. He creates a committee of lay people that you see pictured here from a 1962 Life magazine story. And the idea was that this would happen in phases. First, doctors would say which patients were medically eligible for this dialysis treatment, and then these lay people would select from among the medically eligible patients. Um and um you can see who, you know, kind of the these people are anonymous, but you can see there's one woman in the group. Uh so it's mostly men. There's a clerical collar visible. Um and to put a long story a long story short, this did not go over well. It was seen as very ad hoc. Uh the committee uh it seemed tended to like income earners with families to support. Um a couple of scholars uh publishing a critique of this wrote, "The Pacific Northwest, cuz this was in Washington, the Pacific Northwest is no place for a Henry David Thoreau with bad kidneys." Right? Um so there's lots to dislike about Dr. Scribner's approach uh here, but there's also something profoundly honest about it. And I think it actually set the foundation for uh the algorithm that allocates kidneys even to this day. And the idea was, "Look, the technology might be complicated, but the hard ethical choices that it presents us with belong to a wider community and not only to the technical experts." And by the way, in order to get the ethical input that we need, it's necessary to bring lay people into the fold and get them up to speed on how the technology uh works. And so I'll skip over a bunch of the story, but this essentially becomes uh these become foundational premises for the transplant system. Eventually, Congress funds dialysis for most of the people who need it, and then transplant emerges as a superior therapy. This up till this point, of course, we've been talking about rationing not organs but dialysis treatments, but eventually transplant emerges as the as the first-line therapy, but they can't do that for everybody either. And so, a national system through the use of computers is what the Organ Transplantation Act calls for. Um And um there is there is an impulse uh o- over time the the system evolves into something that heavily prioritizes waiting time. So, it's not quite wait your turn, but it's something um that's uncomfortably close to that from the point of view of many critics. And so, by the time people are getting organs, sometimes uh they are uh older and sicker and less able to benefit uh from those transplants. And so, back to that room in Dallas uh where we began, the basic idea of the new plan is to save the most life. And roughly in this context, what that means is matching each donated kidney to the person who, as near as anyone could predict, would gain more years of life from it than any other candidate would. And they call this concept LYFT, l y f t, for life years from transplant. Uh uh and the math behind this was was really uh complicated because you have to estimate several different things for each person. Number one, how long would they survive if they didn't get a transplant? Two, what if they get this kidney? Uh uh how long is it going to last? And by the way, if it failed, how long would they survive after that? Um uh so each of those questions had uh an intricate uh debatable quantitative model uh behind it. Uh one of the critiques of the lift system was that uh overall the C statistic uh if you compare two candidates, which um and I'm in a room full of statisticians, so I should I should tread carefully here, but my understanding is that this is if you pick two candidates, um you the question is how often does the one who has the higher lift score, that is to say the longer predicted survival uh from transplant or predicted gain in survival from transplant, how long is it is it how often is it right about which which which candidate that is? And the answer was uh 68% of the time. Um uh and so one of the criticisms was this is, you know, uh uh asking people to put a lot of moral confidence into a system that is uh as one of the critics uh construed it, not much better than flipping a coin. Um uh But the the basic idea here, the appeal was we're going to save the most life that we can. And I think at a values level, from the interviews of participants that I did and the reviews of the archives, um this seems to have been a particularly strongly appealing idea for medical experts. So when I went and I interviewed people who were part of the working group that had developed Lift, I mean, you could hear passion in their voices of the doctors, the clinicians who worked on this telling me about its virtues. Um and I think part of the enthusiasm for this approach comes from the values that people are socialized into as they acquire medical expertise, right? A doctor may think in terms of fighting for the most life possible for each patient and typically wouldn't want to regard any one patient as more or less deserving than another. So, there was a certain moral logic to saying a year of life is a year of life. It's equally valuable no matter who gets to live it. Um and we can make more years of life in total and that is better than fewer. And the promise was a lot more years. If the forecasts were right, then in each calendar year that the new system Lift was in effect compared to the old one that focused on waiting time, um there would be more than 10,000 additional years of life lived across the United States relative to the old way of giving out organs. But, Lift was not just going to change uh how many years of life got lived. It was also going to change who got to live them. Uh one of the most important factors in the Lift calculation was the age of each potential recipient. That factor alone was about a quarter of the Lift score. And what you're looking at here is a forecast of how the age distribution of organ recipients would would change under the new system. And what you can see is the existing system is in pink. The uh proposed new one is in blue and it's showing the percent of all transplants going to recipients at each decade of of age age position. So, the fraction of transplants going to people in their 20s was forecasted to triple and the fraction going to people over 50 uh was going to decline to less than half of its former level. And I want to take us to a very specific moment in Dallas just after noon. All the expert presentations are over and a man uh named Clive Call, who you see here pictured with his daughter, takes to the stage. He's 54 years old. He's the first person without an MD or PhD to address the group. He's a traffic engineer from Los Angeles. Um, and his kidneys are on the brink of failure. He has a rare genetic disease called polycystic kidney disease, which causes a gradual decline in kidney function over the course of one's life. He's a patient advocate for group of patients with this disease, um, who, uh, who can live patients with this this disease can live, uh, with their natural kidneys for many years. But, eventually, if they live long enough, they'll need a transplant or else, like other people who have kidney failure, they'll need to rely on the risky and life-dominating ritual of thrice-weekly dialysis to stay alive from from one day to the next. Understandably, um, Clive is focused here on the big picture and not the details. He only gets 5 minutes to make his case, and he starts out with a methodological objection against the new system, um, one that actually makes makes it unclear whether or not he fully understands what's being proposed. So, this is actually a slide from his presentation. Clive starts with with this objection. He says, "Age isn't always a good proxy for health. Sometimes a candidate in their 50s would actually live longer with a transplant than one in their 40s." Um, Clive seems to think that the, uh, proposal is to use age as a bright line so that younger patients would always come first. Um, in fact, the formula was more complicated than a bright line. Age was the most important among the factors, but, uh, there were also other factors like diabetes status, and sometimes the formula did, in the end, favor a, uh, healthier candidate in their 40s over a less healthy one in their 40s. And so, uh, you can imagine that the experts proposing the new system might want to dismiss what Clive is saying because if we really want to be exacting about the details here, uh, he's wrong. Um, but having studied this and having interviewed Clive a number of times, um, I think that even if, uh, he was, uh, mistaken on some details, he was right on the gist. If he'd had a team of data experts helping him prepare for this moment, they probably would have helped him sharpen, uh, the objection. Sure, there are no bright lines based on age, but age is being used as a proxy for health, apparently because the system doesn't have more detailed data, for instance, about each candidate's, uh, heart health or their lifestyle. I mean, one might think that that's not fair and that the fair thing to do might be to go out and gather more data. Um, so, a- as he put it, "Getting a transplant is something I prepared for by taking care of my health for 50-plus years. To not get one now is demoralizing and seems unfair." Um, and in fact, it would be a perverse incentive for his daughter who has the same disease and might, uh, benefit if her health declined when she was younger, uh, enough for her to need a transplant and be basically healthiest among the sick at the time she needed, uh, a transplant. Um, uh, and people noticed other values problems, too, with, uh, with Lift that apparently had not, uh, loomed large for the technical and clinical experts. Uh, healthier transplant candidates are, of course, generally more efficient at converting, um, a kidney into years of life saved, but who's healthier in the United States? I don't have to tell this this group. Uh, pe- people of means tend to be healthier, people with good access to medical care tend to be healthier, and on average, of course, structurally fewer people of color tend to be healthier. Um in the end, Lift was scrapped, and today's algorithm strikes more of a balance, where the healthiest kidneys get sent to the patients who are likely to live the longest. Well, it's still the case that the rest of the waiting list gets a reasonable chance to receive other organs. Compared to what came before, this does increase the amount of life saved, but it also improves on the narrows the the gaps in transplant rates between racial groups and keeps the door open for older patients to get transplants. Uh so um let me uh give you here um just one of the examples. This comes from the annual report by the Scientific Registry of Transplant Recipients, which is a separate body from the operator of the transplant system that exists just to audit its performance. And you can see here, this is from its kidney chapter from a few years after the new system was introduced. Uh you can see a comparison here of deceased donor transplant rates among adult wait-listed candidates by race. And so, the narrowing of this um sentiment basically means that it's nearer to being true that you have roughly the same chance of getting transplanted regardless of your race. There are a whole bunch of other equity problems in the uh in the transplant system, including things about, for example, who even gets to be wait-listed in the first place, which is not uh itself a fair process necessarily. And I get into that a bit more in the book, but incrementally at least as to this part of the picture, this does look like an improvement. Um, okay. So, I came to this as Emily mentioned in her introduction, not as a clinician or a biostatistician, but as someone interested in the question of how values decisions inside of software get made. And uh, there are a bunch of things about this story that uh, are real-life examples of the kinds of techniques that um, are often discussed for newly introduced systems today. Transparency about the decision logic, participation of key stakeholders in designing how the system is going to work, uh, forecasting of what its impacts would be before it gets deployed, auditing after the fact like that slide we just were looking at, uh, of uh, how the system is performing uh, for different groups. And basically the kind of transplant that I wanted to do was a transplant of lessons learned from the from the kidney domain into other areas, whether it's uh, you know, housing or hiring or uh, other areas, criminal legal processes where software's being used to make important choices about people, assignment of school seats, uh, all these kinds of places, um, and uh, so I think I I want to give a half dozen sort of cues here about places where the lessons of the transplant system may be may be useful. The lessons of this story may be useful to uh, the design of other systems. I think cases that depend on public confidence in a system's competence and fairness are a special category of opportunity here because trust is hard to gain, uh, uh easy to lose, and very, very useful, right? Uh so, in the transplant context, people are organ donors. They can't sell uh their organs or buy organs, but uh so so people's willingness to donate and their confidence in the fairness of the system becomes really uh really important. Um uh I think high-stakes private sector systems uh also have this shape to a degree. Hiring, insurance, lending, uh proving correctness to regulators, and building confidence among customers and the public are perennial challenges. Um and I think so transparency and auditing in particular, including independent audits like what uh AST TR does for transplant medicine, may be indicated when when public confidence um seems to be uh a key ingredient in the success of a system. Um a second a second thing uh that was happening here and also happens elsewhere is the rationing of um uh public of public resources. Just as there are only so many kidneys available for transplant, there are also, for instance, only so many uh seats available in each public school. When a resource is public, there's a strong presumption that a broad polity should help decide how it is shared. Um in some cases, there may be a question about whether or not uh to even regard the supply of a resource as fixed. Like, for example, shelter beds for homeless people. But where where wherever one does believe that a public resource needs to be rationed, that framing tends to concentrate discussion and clarify the moral stakes in ways that parallel the challenge of allocating kidneys. Another question is do simpler processes work? Because if so, if it ain't broke, don't fix it. Um, in some places traditional institutions seem to work pretty pretty well for governing algorithms. Pardon me. For example, there's an elaborate regime of stress testing of financial algorithms known as model risk management. Um, which uh uh while not perfect could plausibly be said to acquit the institutions involved pretty well. And in the case of insurance pricing, uh legislators and others did did actually succeed in opening up the moral workings of a high stakes algorithm. There's wonderful work on this by my friend Barbara Kiviat who studied state-level insurance regulations and the moral logic that is embedded in them. Um uh another uh lesson. Oh, somehow they're all lesson number one. Uh another lesson here is that algorithms uh Oops, sorry. I don't These somehow got a little bit out of order. Okay. Algorithms shift our moral attention. This I think is one of the key pieces. Is is because there was an algorithm, because there was a waiting list, because people were being pulled off of that list, the focus was on how people got from the waiting list to the to the transplant operating room. And other questions like, for example, who gets onto the list in the first place or how much can people access transplant immunosuppressive medication after they get their transplant we're relatively off stage. >> Right. >> Second thing is that I think again cuts across the different areas in which we might think about using techniques like these. Participation creates opinions. Sometimes when you think about people will talk about public input and it'll be like a measurement problem like we go out and collect as if some natural resource the opinions that people hold but in this case over a a years-long period people went back and forth and gradually changed their opinions in response to one another's arguments. And that's part of what made it possible for a system to exist that made sense for everyone. Um Uh another lesson fourth out of out of the sixth uh deliberation is costly. It takes time. People have to travel if they're going to be face-to-face. The details matter. It's not something that we can do every day. This process went on for years and so I do think one of the lessons is that we need to be selective about when and where and why we make the investments that true participatory governance requires. Um fifth is quantification can be a moral anesthetic. Um so what I mean by that is that uh when once you have a number there is a sense of authority that the number achieves that makes people feel like there's a right answer even when at some level the choice is tragic because not everyone who needs an organ is going to be able to get one. There's there's a literature on what are called tragic choices and part of the idea uh uh of that literature is that um people need to be able to kind of move through the world without constantly confronting uh uh the felt sense of disaster that happens when someone's not able to get something that they desperately need. And so, even if you look back at the statute that created the transplant system, um it uh focuses primarily um on uh it it says the system should focus on medical criteria. Which, if you think about it, is a little bit of a dodge, right? Because although there are many medical facts that are relevant to the outcome of a transplant, uh the those factors don't tell you whether it would be better to um maximize life saved by giving to the young and the healthy or give everyone an equal chance or so on. Like there are there there are uh inescapably moral questions, uh whether we look them squarely in the eye or not, that uh that are part of this. Um another example uh of such a choice and of such uh anesthesia um uh is Oh, I'm sorry. Are you guys seeing Sorry, I think I'm off by one on which slide is being is being displayed. I apologize. Quantification can be a moral anesthetic. That's where we were. Um is is it is uh it is always uh uh the case that uh there are moral pieces underneath when you really dig. Even in the life-years from transplant formula, the um uh the math had to include some kind of way of looking at the fact that surviving with a transplant is better uh for the patient than being on dialysis and spending 12 hours a week in a in a clinical setting getting dialyzed. But how much better is it if you are effectively restored to normal health with a kidney? And the answer that they that they came up with was that uh a year on dialysis was was worth about was worth about the same as 0.8 years of full health in terms of its effective value to the person who got to live it. But that was deep into the numbers and certainly not written on the tin. Finally, no matter how elaborate the process may be that you go through to design a high state system, the reality of uh uh things that are really really important to people is that they often are going to use whatever tools they have to get the outcomes that they want. And so although there was a year's long participatory process, one part of how the system got resolved was that eventually patients started suing over the fact that the system had not been improved and the compromise that ultimately happened was in part the result of the threat and the reality of some court proceedings. So I will I will conclude there with my gratitude and get to what I think is the most interesting part of this, which is your questions. I I see that there are a bunch of them, but I haven't had a chance to look. Emily, can you help can you help guide us here? >> Yeah, thank you so much. That was really interesting. A lot to think about. I took a ton of notes. I think some of the questions were about zoom issues so far, but let's um ask people right now if they have any questions about First, let's thank our speaker. And then um if people have questions um about the talk, please either put them in the chat or the Q&A and I can read them out. Or if you want to, you're feel free to come off your microphone and ask the question. >> And I'll just interject. I'm sorry about the slides not having been been clearer. I didn't realize that that was happening. I didn't mean to mean to intend that. >> Well, while people are thinking of their quest Oh, here. Okay. First question. Do you think big techs are doing enough in terms of algorithmic fairness practices to gain public trust? >> Um I I mean, that's there's a lot of different uh moving pieces there. Uh and I think really that's a question for the public rather than for the companies. I would I would say that uh uh there's a lot I you know, the the one of the things So, at at work, one of the things the main thing that I do is present uh in detail the technical work that we do on safety uh incident to our deployments of our models. And one of the challenges is that uh the systems themselves change quite rapidly. And so, uh the metrics that are the best available metrics for understanding their safety performance uh and this I think is also true for fairness, uh themselves tend to rapidly evolve. And so, I think that um Let me just also say, I think one of the important questions is how and where do third parties uh that have expertise to kick the tires, but whose incentives are different from insiders play a role in in helping to build that trust and one of the one of the areas of work at Open AI and you'll see this reflected in the reports that we publish is that in various ways we bring in what what we call we call this external assurances work but basically it's various kinds of auditing and various kinds of expert review of our internal um systems that touches on fairness but often focuses on safety and so I actually think you know trust is the joint product of a big system and I think a lot about trying to nourish that that ecosystem. >> I think one of the things that sorry I was just going to say I think one of the things that is a common issue between that question and also something you mentioned about you know who gets on the transplant list and also comes up in like AI and these LLMs and also comes up in my work which I analyze a lot of electronic health record data is like who is in the database that we're drawing from to build these algorithms, right? Like what is the data source and how do you account for what's missing when you don't even know what's missing? That's can be one of the biggest sources of bias rate in an algorithm like this kidney transplant algorithm if you're looking at the historical data on who's been on a list on the list and using that to develop something like who's not on the list um did is that something that they thought about a lot? >> Um >> You know how do I how do you statistical methods or other ways to you know try and correct for that bias of like who's not in that data set that you're using to even get started? >> There was I mean there was some some thought uh about that. I think though that um that like I was saying a little bit earlier, it it is um natural to focus on the levers that you have. And so, when they were designing this system that was about moving from list to transplant, uh questions about who was who was on the list were were were basically kind of off stage. Um it's not that no one in the community was thinking or caring about this. There definitely were and are all along people uh who were thinking about this, but it was kind of it it just wasn't the main focus as far as I could tell of the discussion. And I mean, it is you know, to the point about it being you know, complicated and and costly to wade in to how these systems work. There is a kind of um you know, just as there's a scarcity of the ability to help patients, there's like some kind of scarcity of the ability to improve systems that is also in its own way a tragic choice because we can't sort of make everything all better at the at the same time. But um uh you know, I think actually since this system was implemented, there've been some important steps uh forward uh like there was there was an explicit uh consideration of race in the formula for kidney function that was disadvantageous to uh patients of color, African American patients specifically. And um uh that was removed uh uh shortly after actually my my book came out um due to some other efforts besides this the story we're counting here. Um Yeah, does that >> Yeah, I think that that all makes sense. Um I see another question in the Q&A asking were there other confounders such as rural urban patients, larger smaller transplant centers in the algorithm to help make it fair. >> Um uh there were I mean there were some factors, but I don't think any of of those were among them. And you know, it's interesting thinking about it at a center level as opposed to a patient level. Um one of the things uh that uh perhaps perhaps the sort of clearest wrong that I that felt that I found when I was working on this at a at a personal level. I mean, there are many things that are good and many things that are bad about the system and many opinions about it. But one of the ones that was really seared in my mind once I uh learned this was so transplant centers are in general evaluated at least for kidneys on 5-year survival. So fraction of patients who um after they get a transplant 5 years later, they are still alive and the transplant is still working and hasn't failed. That's the that's the the one of the primary success metrics. Um at the same time uh Medicaid, which pays for dialysis for approximately everyone uh and also pays for immunosuppressive medicines for people that do not have uh other health coverage, which is expensive and critically important to keeping a transplant healthy. Um uh but but Medicaid only pays for 3 years of immunosuppressive drugs. It doesn't pay for an indefinite duration of immunosuppressive drugs, which means that if you're a transplant center judged on 5-year survival um then uh you you basically are are are taking a hit every time you uh transplant into someone who's going to lose their immunosuppressive medications and of that group subsequently uh experience a failure of the transplant before 5 years are out. Um and so that gives transplant centers um a real sort of strong push not to put these people on the list and not to do transplants for them. Pardon me. >> I mean, that's kind of a shocking thing to learn. I mean, it seems very short-sighted of Medicaid to not pay for the thing that makes the procedure work indefinitely. Um is is it I don't know a ton about kidney transplants, but is it just destined to fail if you don't can keep taking the immunosuppressive drugs? >> It it it very much depends on the patient and the and and the organ and the degree of immuno uh compatibility that there is. Uh but uh I mean, in general, they're not giving them out, you know, recreationally. It's it's it's it's clinically helpful to have these to have these drugs. Uh >> Yeah. >> And and so I mean, yeah, if I could sort of wave a wand to change one thing, I think I might start there, personally. >> Are there other questions from the audience and Okay, I see another question here. Going back to the question of who is in the data for training the algorithms, are there methods or efforts to try to get more of those underrepresented groups into new training sets or algorithms? >> Yeah, there's there's a a significant amount of work uh on this. Uh I think, you know, generally across the health space, and there've been there've been some uh cautionary uh uh tales. Uh I people may know the um uh the Obermeyer et al. uh paper about um uh I see Emily smiling uh about uh predicting who was going to need care based on past utilization, which uh you know ended up channeling care toward those who have better access to care. To summarize a very complicated story there, but you know, so that has led to some efforts to uh I would say I mean part of it is gathering different data, but then one one of the most important parts of this is thinking critically about the data you have because realistically um you know, it's interesting data in Latin actually means given and it kind of is the givens, right? It's kind of typically the data is gathered incident to some other process and then you know, we're thinking about an algorithm or you know, like factors were gathered because they are important for for care and then you know, OPTN the transplant network is sitting there and they they have you know, certain fields in their database and they don't have other fields. And there there are efforts to to expand this and certainly the cost of gathering additional data is decreasing. So I mean I think the direction of travel is toward higher resolution pictures of of what is happening, but it's also um you know, it's it's it's a slow process. These are complex systems that take a lot of work to update. >> Yeah, that makes sense. Okay, here's another question. You mentioned one patient who spoke of his concerns at a conference. Are there any efforts to make the process of donor selection more patient caregiver centered or to incorporate that into algorithms? >> You know, um uh so so yes, one of the things uh is important is for patients to even know and understand that transplants are available and are first-line preferable treatment. Particularly patients that are experiencing kidney failure. And there so so the places that provide uh dialysis care are supposed to tell patients about how they might be able to get a transplant. But the incentives there are very very poorly aligned because the dialysis centers um DaVita and others they actually make uh you know, it's a business that uh is more successful the more people demand dialysis. So, that's one of the one of the frontiers of this is educating patients about the availability of of transplants and partic- particularly patients that you know, don't already have excellent access to medical knowledge and care. And so, for example, one of the uh uh uh physicians that I interviewed uh uh works in Texas helped to create a large uh transplant program there and African-American patients were underrepresented. And one of the things he did at one point was to institute a program of having past transplant recipients uh from that community, so African-American recipients, uh go and talk at local churches about how this worked, how it was valuable, um and how people should know about it. Uh the the incidence of kidney failure is much higher in the African-American community. I forget if it's three or fourfold uh the national numbers uh because the factors that contribute to kidney failure include things associated with socioeconomic status like uh stress and diabetes and obesity. Um uh So so that's a piece where there's there's a lot uh you know, there's a lot more that that can be done and in part it's cultural, even though once you're on the list it's data, but kind of getting there is cultural to a degree and sort of and political really. You need the resources. >> Um >> Yeah, says this question says, "I'm reminded of Cathy O'Neil's Weapons of Math Destruction. I think we often rely very heavily on a very small asterisk of AI um can make mistakes, be careful. Can you speak to more proactive accountability when things go awry while balancing model and technical progress? And does this lean on the third-party vetting you mentioned?" >> Uh yes, I I'm a huge huge fan of of Cathy's and we've worked together for years. Um I it it it So I think uh I mean, there's sort of our two pieces to it. Uh one is clear understanding of what's happening in a system and how the pieces of it fit together and you know, who's benefiting or being harmed. Um and then there's a piece about, "Okay, given that who should be accountable? What should we do? What should be changed?" Uh my my my primary focus is on the first piece, which is building shared understanding of what's happening and then I think that creates the building blocks for a broader, you know, discussion of uh of accountability and and kind of for people to reactively be accountable. But you know, another really interesting thing about transparency is that um it doesn't just It's not you know, you could think of it as like, oh, we find problems through transparency and then we fix them because transparency showed us that like there was something, you know, behind the curtain that was bad. And that can happen and that is important. But another really important part of this like uh it is how it changes the incentive structure of the thing that is transparent like in the first place. The fact that people know that they will be subject to scrutiny and review is a very I think is a very healthy incentive alignment. Uh you know, and I'll It's a little bit analogous. So, I work on a safety team and you know, one way that you could ask about safety is you could ask your question like, well, how often does safety pump the brakes? How often does something almost, you know, go out the door and then safety says, "No, it's not safe and we need to change it." Um uh that can happen. Uh I've been part of, you know, uh a few of those moments where, you know, particularly for something that's sort of baking right until we're taking it out of the oven. It's like wait, at the last moment it may become clear that there's a problem and then we take a step back and uh you know, build new safeguards or or decide that we need to uh do something else. But another and probably in the long run the more important aspect of how you make something safe is uh people know that it has to pass muster at the end in order to go out the door. And that changes the planning all the way back to the very beginning. And I think that's um that is one of the pieces of uh of proactive accountability is uh so, it's not just reactive when things go awry, uh but also, you know, creating conditions where people uh are encouraged to build in a way that is less likely to go awry in the first place. >> Yeah, I think that's really interesting point. Um >> [clears throat] >> new question. What advice can you give to those of us also working to develop shared understanding on technical topics? This person says they work in public health informatics and it's a constant challenge. >> I yes, um uh I'd be curious to hear a word or two more about what that the shape of that challenge is. Is that Is that a thing we can do? >> Yeah, of course. Do you want to come off mute and and >> Yeah, I can come off mute and talk about that. So, um the space that I work in really sits kind of between public health programs and the folks in those programs that do the data work for for the programs and then the IT folks that sort of support the programs with infrastructure and stuff, right? So, um those the those IT folks don't always talk the language of the what the programs epidemiologists and those folks need and then uh you know, being from a statistic I have a statistics and system science background, so I like I can kind of see both sides of things and so I'm constantly doing a translation of trying to help each each side understand what the other folks are trying to do and what they're trying to get, you know, um what their goals are and what they're you know, trying to create this like shared purpose in in projects and across, you know, strategic uh plans and things that we have that in public health, right? >> So, so I I I can say a few things that are are part of my compass for navigating um situations where you have like colliding complexities of different kinds or different specialties. I think that um you know, one of the things that that is really important is to be intrepid about what I don't understand. Like part part of uh part of my job is to sort of be the person in the back of the room who says, "Wait, I don't get it." Like, can you how would you explain that to a family member? Or you know, it sounds like it's like this. Is this what it's like?" And then you sort of offer something that is a more accessible formulation. I think part of the nature of being an expert is to sort of not have the point of view of someone who comes at an issue naively. And so, it's almost like naivete is a little bit of my stock in trade. Like, I'm among these engineers and I'm sort of uh able to be the person who's a proxy for those who, you know, don't have all of the background and you sort of end up um trying to find very, very accessible ways uh that that use, you know, avoid uh specialized terminology and really uh address the essence of of what's happening. Of course, this is I think, you know, it's difficult to do when the the questions on the table are themselves detailed and and technical. So, I wouldn't point to this as being easy. Like, I think this is one of the hard problems living in in modern civilization where we have, you know, so much complexity that we sit on top of every day. Um I do also It is one of the areas in which I'm excited about applications of AI is the idea of being able to meet people where they are linguistically, conceptually, educationally, and unpack for them in a way that that is tuned for them, uh you know, how some public program works, what they need to do to avail themselves of a benefit, how to, you know, navigate a process. Those those sorts of things. Those are the sorts of things that in general um AI can do pretty well and it's very very high availability. Um So, uh oh, and someone says I ask so many Abigail in the chat saying asking lots of questions. Yes, this also is uh there's kind of an aspect of slowing things down. I think people sometimes uh they communicate at high bandwidth and high velocity inside a cone of expertise and that's that's natural, but um shared understanding is is something that um it sometimes takes different different pacing as well. >> Yeah, I I love everything you you pointed out here and I find that I'm often working with mental models, you know, and I try to uh uh map out what those mental models really are by listening to the the the way that someone's talking about, you know, what they're trying to do and and I've even it's funny you brought up AI. I use Copilot and I will go and take these sort of mental models and and ask it for the the problems and the questions that these folks are facing or the questions they might ask and and sort of try to get at it in that way too to to to um you know, write up a an FAQ or something like [laughter] that, you know, like >> Yeah. >> to try to like you know, illustrate what the other is trying to bring out and then I'll share it with the whole group, you know, that kind of thing. So, um thank you so much. >> Of course. >> Yeah, it's it is that conversation so interesting and it reminds me of something you said earlier that really struck me that like you said something like the number gives things a sense of authority and I think that that is true on actually so many sides of these kinds of problems where like once we as, you know, people generating numbers, you know, I'm a statistician, so I work on the side of like I'm making the number exist. Um once that's out in the public, people think, oh, that number is the truth, right? And that could that's a problem because that's not the case. The number is an estimate and it's a moving target and it comes with its own biases, but I think we as people working in um statistics and data fields can also make that mistake, right? Where we think that the number is is more important than it really is and it's easy to kind of forget what it means and what's behind it and what the moral interpretation of that is, what it will mean to an individual who's facing that number. Um and actually like something that you were saying before, I I think that's another thing that AI is good at helping with. Like you oh now now from the patient side, oh, I get this number, this prediction. Well, what what is that actually mean? And doctors aren't always good at explaining that to patients necessarily or people don't know what questions to ask to get the information that they actually want in the moment, especially if you're dealing with like a stressful medical condition. Um that kind of thing. So it it's a hard situation, I think. >> Yeah, for sure. >> Um I see another question in the chat from the information science perspective, uh what are some areas you think need more future study to make our applications safe and accountable? >> Um so um I think the kind of thing it's funny I'm seeing um this comment about walking around and seeing patients. I think groundedness is so important and is so is so easy to sort of float away from in our informational world. And it's it's it's so I think uh much of the best work in information sciences that I've encountered is is is work that involves, you know, not just like what did sending alerts to the, you know, nurses on this ward, you know, do to the sepsis rate, but like, you know, what did they say about these? And were they, you know, was it helpful? Was it overwhelming? Like, really uh a high-touch aspect to it. Um and I it's it's sort of ironic that in, you know, a field that is defined around these information systems that uh I find bringing the human element in is often um among the most uh useful things we can do. And often in an interdisciplinary context where you've got, you know, people that might have, you know, methods backgrounds that involve talking to people, and you've got other people that can sort of see the innards of the system as the as the gears are turning. And and put those understandings together. I'll take a last, you know, curveball. Um and if there if there If there isn't one, I'll just I'll just say uh thank you all very much not only for being here, but for the uh but for the work you do um that is that is so important um ultimately to all of us. Thank you. >> Yeah, and I I will also tell people if you haven't read David's book that this talk was based on. I did read it in advance. Um and I think you that you would be pleased to read it yourself because you will get a lot more details that of of what this story is about. So, I would encourage you to to read the book. >> Thank you. >> as well. And thank you so much. Um if there aren't any other questions, I will just thank you again for being here and for this wonderful talk. I think it gave us all a lot to think about and important things to think about as people who are kind of behind some of these numbers and thinking about these issues, the the issue the topics of, you know, the moral side of things and the shared decision-making and accountability, I think are really important to keep in mind. >> Thanks. Thank you. Thanks, everyone. >> So, thank you so much.