Statistical Rethinking Lecture A09 - Modeling Events

Welcome back. We're at lecture A9. Are you excited? Yes, we're almost done. I'm exhausted, but happy. So, um we're going to do modeling events today. It's uh two big themes in this lecture. The first is teach you uh uh the general structure of generalized linear models. So, we're going to move past purely linear models to generalized linear models as a category, focusing on the example of uh binomial and Bernoulli models. Um and then topically, it's about modeling events. Events are things that get counted. They're not continuous measurements. Uh and that's a big category of things in science, so you really need to know how to do it. Uh and then the um context I'm going to do it in is in the context of the statistical modeling of discrimination, which is an interdisciplinary topic that I myself am sometimes uh obsessed with, people have told me. Uh but I think it illustrates the real value of some explicit causal inference and structural modeling to make sense of of the data. Uh there's a metaphor I want to start with. Uh this is uh Lord Kelvin's tide prediction engine, one of them. I think it's the second one he built. Lord Kelvin is a Scottish uh scientist engineer. I forget his real name. Uh Kelvin is a river near Glasgow in Scotland, and Lord Kelvin was just the you know, the lord of that land. Um I forget the guy's real name. This was back when all scientists were rich people, cuz it's the only way you could afford to do science, right? Cuz it didn't generate much income unless some royal person wanted to pay you for it. Uh but he built mechanical computers among many other things. And uh uh this is what's being traced out on the left part here by that weird arm is a prediction of the tide levels. And there's a bunch of gears which interact and add together in complicated ways to produce that continuous prediction. This is what generalized linear models are like. In fact, many statistical models, once you get outside of very simple linear regressions, are the internal guts of the thing are on some scale that is not the outcome scale of interest. Uh linear regressions are nice and cognitively convenient despite their difficulties because the uh scale on the right-hand side and left-hand side are the same. Yes, it's they're the same thing. So, it just passes through. You have a direct model of the mean as an additive combination on the left that measurement. This is not true of any other type of statistical model. Sorry. And you have to model the other kinds of events. It's just true. Uh good news is it's not that bad once you get used to the idea. Um these machines can be constructed with LEGOs, too. Here's a LEGO-type prediction engine, which also functions, just to show you that uh it doesn't have to be as complicated as Lord Kelvin's, as well. Um generalized linear models are meant to map uh additive combinations of predictors, like the sorts you've been doing before, uh which again, why do we like additive combinations of predictors? It's one of the simplest ways to do statistical conditioning, to stratify. Why do we want to stratify? Because it's the only way to get causally valid estimates. Yeah? Remember the adjustment sets and all that stuff from the DAGs. Uh but we need to map those uh additive combinations, those those stratifying variables, onto outcome distributions uh that appear in nature, which are not simply Gaussian. Uh so, we've been working with Gaussian before, which is in the upper right here, and this is my uh sort of uh relationship diagram for for the distributions that are most common in applied statistics, and they're most common in applied statistics because they're everywhere in nature. These are the fundamental distributions of generative processes in nature. Uh there's a whole chapter in my book uh about these and the generative processes that generate them. I'm not going to spend a lot of time in this lecture. Unfortunately, I love this topic, but there's just not enough time. Um these are these distributions are not made up by statisticians to annoy you. All right, these are things you find in nature, like the Gaussian distribution. Uh so at the middle here, uh this is called the exponential family. It's a family tree. And right in the middle is um the the or distribution. Right? So if uh what do we call that in English? The great-great-grandparent, the uh primal primal distribution. That's a better translation. Uh which is the exponential. Um and then there are various ways that you can transform exponentials into all of the others. Uh these things are there. They're related to generative models um uh and therefore to causal inference. Um uh and I want to give you a glimpse into how we use them productively and interpret them. Uh this material is I've said this before, I just want to say it again very quickly. This material's complicated, so if you're feeling confused, it's only because you're paying attention. Right? That's true. It's perfectly normal. You just have to understand enough to flow forward a bit. Right? So you're not sinking, and that's normal. Don't feel like you're the only one. Um it's it's a real problem in teaching applied statistics that uh everything comes at you all at once. You've got the scientific models, research design, and and coding, and and uh everything else. Uh inflation. Right? It's just hitting you at once. I'm very sympathetic to that. Uh but you don't have to get it all at once. You you will be able to effectively do applied statistics before you understand it. I know that's that's a good thing. I'll say it again. You will be able to effectively do applied statistics before you fully understand it. But you the understanding will come with practice in the contexts, the scientific context where you have the scientific background to understand the statistical output in light of the generative models in your discipline. Okay? It really happens that way. So you need to practice it. It's like learning kung fu. You cannot watch Jackie Chan movies and learn Kung Fu. Right? You got to go, you know, throw some punches and get hit a bit. Right? So, you have you have to train the motor memory. And the same is true uh with statistics. Okay. Oh, this this cartoon is just to say like I know sometimes the homework seems hard, but you should try research. Yeah, that's even harder. I'm just trying to build a ladder for you that gets up to that, right? Uh but that's just how it is. Of course, you have more time to do your research than to do the homework. But uh uh I'm trying to I'm trying to find that middle ground in the homeworks where it represents a real challenge and applied that is not trivial, right? Like I'm really teaching you skills you need, but it's something you can practice. It's not as hard as as uh real research. And sometimes I overshoot. I apologize, but I'm just trying to figure out where that spot is the sweet spot is every time. The data set we're going to work with today is this uh in statistics a somewhat famous uh small data set cuz it's um from a paper from the late '70s, um and it's been uh used a bunch in teaching in statistics to to illustrate something called Simpson's paradox, uh which which I'd be happy to talk about, but I don't plan to talk about today. Um it is uh around 4,500 PhD applications for the 1973 UC Berkeley um uh admitting class. And uh here I'm showing you the whole data set. There are 12 rows. That's what I like about teaching data sets like this. You can just put it on one slide. Um and what we've got is uh a number of departments. Uh 1 2 3 4 5 6 departments. And in each of them, we've got the total number of applications that were submitted that year, and then the number that were accepted, and the number that were rejected, and then we've uh stratified those counts by the uh uh self-reported gender of the applicants, male and female. And these data were uh used internally in UC Berkeley. This is how it came to the attention of the uh, statistician who wrote the report on it because there was a question internally by UC Berkeley's administration. This is a very healthy thing, whether there was gender discrimination in admissions. Yeah? So, they internally did this. They weren't forced. Right? And so, they they compiled this data set for that reason. And what I want to do is walk you through um, modeling these data. These are counts. These are obviously not normally distributed. Maybe I shouldn't say that. They're not normally distributed. It may not be obvious yet, but it will be. Uh, so how can we model counts? That's one of the extremely common form of data. If you're an archaeologist, like almost everything's a count, right? Um, I don't know if there were if you agree with that, but it's like counts of stuff, minimum numbers, uh, how many toe bones, things like that. Um, uh, how can we model uh, counts? And then how can we causally model something like this? And actually ask a question, is there discrimination by admissions officers? Um, this kind of research, this is still contemporary, it goes on. As I said, I've been accused of being obsessed with this kind of statistical problem before cuz it transports across domains a lot. And there are lots of really naive papers every year published by specialists in particular fields who don't know the general statistical literature on this. So, uh, I I It's like I don't mean to pick on these two papers, but I'm going to pick on these two papers. Here are two papers that came out in the same year, 2022, using the same data, about the same population, which is uh, members of the National Academy of Sciences, and they reach almost opposite conclusions. Um, neither of them are right, in my opinion. Or rather, I should say neither they're not even wrong because there's no causal framework in either for us to figure out what the estimates mean. This is causal salad. I know I'm very critical of these papers, but it's just you get people who are rightly interested for very good reasons in the the problem of gender discrimination or discrimination against some other protected category, they've got a data set at their disposal, they think we can analyze this without noticing that there's a hundred years of statisticians talking about how hard it is to do this in cross-sectional data sets. Uh so I'm going to lay that out for you today so you can understand. Uh I'm convinced there's a lot of gender discrimination in these domains, but not because of cross-sectional data sets, it's because of the testimonies of individuals who have suffered it. All right, I'm an ethnographer. That's the heart you talk to people you know, you've got rich stories. The cross-sectional data sets are very hard to model in these things, but I'm going to show you how we could try to do it uh with the goal of transparently laying out the assumptions that would be required to reach different conclusions. Yeah. Okay. Sorry for the rant, there'll be another. >> [laughter] >> Um modeling events. So, what do I mean by events? I mean discrete unordered outcomes. Uh things that can happen. Yeah. Um in the other section last week, uh the most recent lecture I gave was also about modeling events. It was black cat adoptions. If you're interested in that, it's up on YouTube. Uh that's a kind of event. The cat is adopted or it's not. And we modeled how long it would take uh how long black cats wait to be adopted compared to other coat colors of cats. And yes, black cats are discriminated against. Support thanks. Adopt a black cat. Okay? They're they're good cats. Um Okay. Uh the observations of events are counts of the particular unordered discrete um events. And uh uh the unknowns though are not counts, they're probabilities. And this is different from all the models we've done so far in this section where we were modeling a measurement on the left with that is distributed normally, and uh the mean that is the the the units on the additive predictor are the same thing. They have the same units as the outcome. Yeah. Could be where there's kilograms or miles or whatever it was. And that's not true anymore. The parameters in count models are probabilities or log odds. I'll explain what those are when we get to it and the outcome is counts. And that just has to be that way cuz that's the way the nature is. Okay? You can't observe probabilities. They're fictions that humans make up. Yeah. At least if you're Bayesian. And but counts are things that you can measure. Yeah. We want to relate these together there's a very justifiable and powerful way to do this. In models of this type generalized linear models everything interacts whether or not you explicitly encode an interaction effect. So in linear regressions many of you will know there's this convention of multiplying predictors together to construct a linear interaction term. I showed you ways to do it also by subsetting by index variables create interactions. Those are all great strategies but in in purely linear models you must explicitly construct the interactions. Make the effect of one thing depend upon the other by explicitly encoding it in the prediction equation. That is not true in a generalized linear model. You can still do that. Right? And there are good reasons to. But everything will interact anyway because of ceiling and floor effects. So say we're modeling the probability that that salamander dies. I know it's macabre but bear with me. This is also the example in the book. If one thing kills it nothing else can. That's an interaction effect. Right? You can't kill the salamander twice. Yeah? Does that make sense? Okay, good. Sorry, I work a lot with ecologists so I'm often thinking about things like this. But um Okay, and then we're going to deal with this weird thing called log odds when we get there and I'll explain how that works. Okay. Yes. Um The core Bayesian workflow I'm going to reference this just a couple times today cuz I think you're used to it, but we're going to start with the generative model and the estimate as always work our way into the count data, but we're going to start by thinking about the generative model and and the different kinds of estimates, different kinds of causal queries we could make about it in the context of these admissions data. So, the the vague estimate is was there gender discrimination in graduate admissions. Now, I want to problematize this a little bit. I'm going to build up a simple causal model which which is uh uh in this case sort of unarguably right, but incomplete. That is it's missing variables. I want to explain that to you as well, and then show you the different kinds of estimates that relate to it. So, the first thing here is the idea that gender influences admission. Um and the background to this of course is this it's gender as judged by the admissions officer, the person reading the application, right? It's not the direct effect of the gender, but the the gender identification in the context of judging an application may affect admission. Um and if that's true, that feels like discrimination, right? We don't want that to happen. It's not that there's something else that's that's correlated with gender that's doing it. Uh there are things which are downstream of gender um which also affect admission, like which department you apply to in this data set. Um it should not be a surprise to any of you. Most of you are anthropologists, right? Uh >> [laughter] >> anthropology does not have an even gender ratio. It has not for 50 years, I think, something like that. I participate uh a lot in two different disciplines. One is anthropology, other is statistics. These have opposite gender ratios. It's like going to the different conferences like entering different universes for me. It's just really strange. I feel much better in anthropology, to be honest. Maybe I shouldn't say things like that. But, um uh anyway, so uh the gender ratios in departments affect things, and so um and that also affects your admission because some departments are, let's say, way more popular than others. They get way more applications than they can possibly accept. Uh and therefore, if if that effect the saturation of a department is correlated with the gender ratio in the department, there will be uh what I'm going to call structural discrimination in the total system, right? Uh not direct discrimination by judging gender, but just by the choices of the individuals of where they apply. Um and we can have a an interesting discussion about whether whether we are comfortable with that or not. I am not. I will just say right [laughter] now. Uh structural discrimination is still discrimination. Yeah? Um okay. This relates to other kinds of discrimination, too. Wage discrimination, people study it with the same kind of DAG. You've got some protected status. Uh it could be cultural background, ethnicity, gender. Um uh and then that uh uh influences occupational choices, and there may also be direct effects on wage through direct discrimination. Uh uh so, taste-based discrimination uh as well. Same kind of structure. Um I've I've uh published with some of my colleagues a few papers on um police discrimination, mainly in the US. Uh some of you know about this topic. Um also has a similar kind of causal structure. Uh ethnicity influences probably you get stopped by the police, and there may be indirect and direct effects of being uh assaulted by the police as a consequence of that as well. Trying these things apart in observational data sets, it's hard. Uh but it's worth trying, uh is what I want to argue. Okay. Uh this is a long walk. Let's get to what is discrimination. There are There's lots of discrimination in here, and you just have to be clear about what you're talking about. There's direct discrimination. Sometimes this is called status-based or taste-based discrimination. Um where the person judging the application makes a judgment about the person's gender, and this influences, maybe unconsciously, maybe consciously, um their probability of accepting uh the individual. Uh there's also indirect discrimination, sometimes called structural discrimination, where it's not the admissions officer that's doing it, it's just that um uh individuals of some genders uh tend to preferentially apply to departments which are saturated in applications, and so there have a lower probability of their application being accepted. And so in the total population across all departments, there's discrimination. Does that make sense? This happens with grant applications, too. And that will be your homework problem. So I have a data set from more recent times uh that's that's grant applications. Um Does this make sense? Yeah? And then there's the total discrimination, right? Where we add up all these paths. And this is the total discrimination effect as well, which is what individuals across all of the departments experience. Um and the question of which is discrimination depends upon your research question, or what kind of interventions or policy issues you're interested in, or which subjective experiences. And that's that's They're all legitimate kinds of questions in different contexts. Yeah, question. So what you didn't like about the two papers you showed us earlier is that they they didn't define well They've got no DAGs. >> [laughter] >> They've got no DAGs, and they've got no estimands, even. It's like I challenge you to read those papers and figure out what they were thinking they were estimating. It's really And I don't again, I keep saying I don't want to pick on those papers in particular, but I clearly am. No, but there are lots of papers like that that come out like And I follow this literature, so I get kind of kind of exercised about it. Um in a lot of fields, it's just a general symptom of the causal salad approach to regression modeling, that but it manifests in an area that I care about, so I get more angry about it. That's all. But your question's a very good one, yeah. There's different flaws in the different ones, but it's just very hard to piece together in those papers exactly what their regression strategy is trying to achieve. They've got some variables, and they make some regressions, but they don't justify them in partic- in light of any particular estimate or any particular definition of discrimination. And then the papers use the same data to reach opposite conclusions. It's really frustrating that sort of problem. Um Anyway, I hope I answered your question. Yeah? Does that make sense? Okay. Uh sorry, I I I yeah, I'm very invested in this topic, and you can get me to go off on a tangent about it so with very little effort. Um Okay, point I want to make that's about statistics now. Uh the total, as is often true in these causal systems, the total causal effect is something you can estimate. Just stratify by gender. Right? If we're interested in total discrimination in the whole system, that's easy to do. We can confidently estimate that. There are no arrows into gender in this data set. Right? So, it's not confounded, we can estimate that. But, most people don't want that. Right? So, the the thing you can estimate is not what you want. And the things that you want are hard to estimate. By hard, I mean you need to make strong assumptions, and those are the two on the right, the indirect and direct effects. You have to make strong, sometimes implausibly strong assumptions to say that you've estimated the causal effect without bias. I want to say um even if you can't eliminate bias, it's still worth doing the estimate. Yeah? These are important problems, and if you're waiting for the perfect data set and experimental design to study them, you're going to be waiting your whole life. Right? You we just have to be open and transparent about the risks of bias, try to walk out uh transparently what direction we think the bias goes and what its magnitude may be. Um And and I think in the next lecture, I'm going to pick up this and maybe talk about that and like adding bias to these models. Uh if that that might be interesting to you. You can rebel at the end of this lecture and let me know. Um, does this make sense? Yeah? Uh, in particular when I say requires strong assumptions, let me bring up one. One strong assumption is there's no unobserved confounding. I think it's very, very likely in these situations that there's unobserved confounding of this type, that there's a common cause of choice of department and admission chance. All you need to believe is there's heterogeneity in ability among the candidates. For whatever reason, their educational background, how rich they were when they grew up, um, uh, uh, natural talent, whatever that means. Right? It could just be their perseverance in studying. I think personality variation like that, your ability to A lot of us as professional academics, right? What is our What is our What is our great skill? The ability to sit and do boring things for countless hours at a time, right? It's like our superpower. Yeah? It's just an incredible thing. Yeah, just keep fighting with that data set, right? Most Most rational people would give up and start another job, but we are into this. Right? So, those sorts of things, uh, people are heter- heterogeneous in that this affects ability, this will affect, uh, choice of department because of the nature of the work in different disciplines, and it appeals to different people. Um, or how competitive different disciplines are, and people of higher ability will be more likely to apply to the more competitive disciplines because they have a comparative advantage. Uh, and that will also affect their acceptance probabilities, right? I don't think this is a difficult, uh, thing scenario to understand. Um, I think it happens a lot in these circumstances. Uh, so, given this, um, you can estimate the total causal effect of G in this graph, but once you start stratifying by D, note that D is a collider on the path between, uh, G and and A. Yeah? So, once you stratify by department, you've opened up a biasing path. The unobserved confounding on D and A does nothing if you just want to estimate the total causal effect, but you turn it on when you stratify by department. Okay? So, again, next lecture I want to talk about this more, but I'm going to ignore it for the rest of this lecture. Okay? But, I wanted to get this out in front of you to think about it. Good? Yeah? There's nodding. I appreciate your nodding. Um I feel like I'm ranting because I'm just so like invested in this topic. Right? Okay. Uh now we have our little DAG. Isn't it cute? There on the left. Um I've removed the unobserved confounding. Cuz we're not going to think about it again. And I've written down an estimate for direct discrimination, which isn't to say this is the only thing worth estimating, but I want to show you a workflow for going through to that. Okay? Um It reminds you yeah, this is the the distribution of A of doing G holding department constant. Right? That's the direct effect. Does that make sense? Good. Um now let's get into the mechanics of generalized linear models. Okay. So, it's just the thing I Here's my dank meme for the day. So, um it's a thing about modeling counts is that the internal mechanics of the machine of the tide prediction engine are on a scale called log odds. Uh and if you're a psychologically healthy human being, this is not how you think. But, soon with practice, you too can can think in log odds. Right? It's a very kind of natural scale. Uh and it is actually it emerges naturally from the fundamental modeling of counts in nature. It's not something that statisticians invented just to torture you. Um So, we observe counts, but we estimate probability on the log odds scale usually. I'll say what that is of the events. And on it the first model in this course was act also a count model. Remember? Uh and we worked with probabilities back when we tossed this globe. That This is why I've left this in the lecture halls cuz I wanted that I was waiting for this lecture where I would pick it up again and remind you about the globe tossing model. This is the same kind of model. We're going to do it again today, but I'm going to talk about the mechanics much more cuz we're going to have predictors on the right hand side. We need to stratify by things. Okay? Um Oh, yeah. The maximum entropy classifier in this meme, this is what machine learning people call the model we're going to do today. Because you can derive it by maximizing entropy. Uh which is something I explain in the book if you're interested in it. There's a whole chapter on entropy and how it relates to statistics in the book, which I love to death, but I I I would need an hour to do it just by itself. Okay? I apologize. Um All right. Generalized linear models have a structure very similar to linear models you're you're familiar with. In linear models, the expected value is some additive combination of the parameters. Uh in generalized linear models, the expected value is a function of an additive combination of the parameters. So, here I've just made some anonymous function f. Now, so we we have this sum, an additive combination of the parameters, and we're going to transform use some function to transform that into a probability scale. Um so, let me break out the anatomy of this a little bit more. Um Here we have a a 0 1 outcome. Uh we model that as Bernoulli because that's the only distribution you would do this as. Typically, Bernoulli is just a coin flip distribution, a biased coin flip distribution named after some Swiss mass- Swiss mathematician. I don't know. There were like nine Bernoulli brothers, and they were all mathematicians, and I can't keep them straight which one this is named after. Right? Um I don't know what the Bernoulli are doing these days, but I'm sure they're rich. Yeah. They probably make chocolate. Or hoard illegal money laundering or something like that. Whatever the Swiss are doing these days. Sorry. It's all done. Uh p sub i is the probability of this event y sub i, right? It's an expect- it's an expectation of a type. It's a probability. Across many trials with the same sets of predictors, we'd expect um um, this to be the uh expected value of the outcomes. Uh, and then we have our, um, additive combination and just like in linear models, this is any real value. The whole real number line is legal here. Yeah, between negative infinity and positive infinity, although we don't usually want to go that far. We want to stay somewhere around zero. Ideally. Um, and then we have this function F, which maps the probability scale uh, to the linear scale. It's a transformation. And in in Bernoulli models and and count event models, there's, uh, very conventional and natural. It emerges naturally from the maximum entropy classifier approach. I there's a box in the book that proves this for you if you're interested in the mathematics. Um, it's called the logit link. And, uh, the logit link, we write it as logit P sub I is equal to the log odds. Uh, so let me explain what's going on here. We call this log odds because PI over 1 minus PI are the odds. It's the probability of the thing over the probability of the other thing. Yeah, this is called the odds. If some of you are gamblers, you're very familiar with odds cuz gamblers talk about odds all the time. Uh, if you are a gambler, shame on you. No. >> [laughter] >> Um, but you may be familiar with this. Yeah. Uh, in my youth I used to go to Las Vegas and I am familiar with odds. Yeah. Um, and, uh, it's log odds cuz there's a log in front of it. Yeah. Uh, this is a very convenient transformation, the log odds. Uh, actually it has it has symmetry. It has lots of nice properties, uh, that we want in a transformation, uh, of a scale. Um, and the inverse, when you convert it back, is called the logistic, which population biologists are familiar with and I'm going to draw it on the next slide. Right? The logistic curve is this S curve. There are lots of S curves, but the logistic is the best. Yeah, it arises naturally from population growth of certain idealized type of population growth. Um and the log odd scale, so what I'm doing on this graph is on the horizontal, we've got the log odds of the probability, so we've transformed um the probability scale to that additive scale. So, the value on the x-axis is the sum of all of your parameters and predictors multiplied by predictors, okay? That's what the horizontal acts the the x-axis is. And the vertical axis is the probability that corresponds to it and it takes this S-curve. So, zero means a half, right? Right in the middle. It's important that this is symmetric around zero um and four means nearly always and minus four means hardly ever. And you just kind of bracket that in your mind and get used to this. Yeah. Um increasingly bigger numbers are still always and increasingly smaller numbers are still never, which generates uh certain identification problems in in non-linear models, which are it's important that the stat model have these these inconvenient properties because that's how nature is. Yeah, remember the salamander? One thing kills it, nothing else can kill it. Right? It's It's just how it's supposed to work. Um let me show you some animations and then we'll continue with the mechanistic part. This is just a prior predictive simulation of an anonymous uh logistic regression on the on the left-hand side here, I've got a prior distribution, very loose prior distribution for alpha and beta in the linear model on the log odd scale. Right? So, alpha and beta are on log odd scale. Uh zero means a half, right? Beta zero means no effect of the predictor. Um and I'm just taking random samples and animating them around as before uh in there. The little red cross uh on the left graph is the data generating model. Um and I'm going to start taking samples from the data generating model soon, okay? And then we're going to do Bayesian updating. On the right, we've got the prior predictive space for the logistic curves. And then what I want you to see is that they can go either direction. The prior has no bias for a positive or negative effect of the predictor, which is on the horizontal axis, and the vertical axis of this is the probability of the outcome of the event. Does this make sense? Yeah? Um now I'm going to start taking samples. We're going to add one data point at a time and do Bayesian updating, and we'll get the Now we have a posterior that's slightly shrunk. There's two data points, three data points. Isn't this satisfying? It's just very look. Um our machines are learning. Yes. We get closer. Uh See how fast you get contractions? It's the power of sample size and simple models. If you want to estimate the mean of something, like 12 data points is incredibly powerful actually for estimating means. Um if you want to stratify it, you need a lot more, but estimating means and we get contraction. Uh and it moves gradually. Um this is a consistent estimator, so as the sample size increases, the posterior will converge to the truth um with probability one. Uh and you see how it's working there. Yeah? Okay, so this is like the previous thing. Bayesian updating works exactly the same way here. It's a nice, I think, relaxing thing about being Bayesian is there's only one kind of way to process the information. You declare the generative model and then you do Bayesian updating. So we don't have to choose uh an estimator here. Good? Does this make sense? Yeah? All right. Um let's talk about priors now. So I did a prior predictive there. Uh the thing about uh Bernoulli binomial regressions with uh logit links, just like all generalized linear models, is since now the the parameters are on a different scale from the outcome, you have to think a little bit more about the prior scale. But I've already taught you how to deal with this. You just have to do prior predictive simulation. You The implications of the prior count on the outcome scale, not on the latent scale. But you must choose them on the latent scale. I'm sorry, this is just how the world is. Yeah, it's like this all the time. Uh here's an example. So, you can think about we're going to take alpha, which is the intercept, remember in the linear model, here like the world's simplest logistic regression logit piece of I equals alpha. Alpha is just the average log odds of the event. If it's zero, it means a half. If it's four, it means always, almost always. Minus four, almost never. And let's imagine putting a Gaussian prior on this, which is often a reasonable thing to do. Okay, so the top one is uh what you might think of as a default really flat prior, normal zero 10. That's a normal distribution with a variance of a hundred. It's very, very flat. Um lots of textbooks uh in applied Bayesian statistics will tell you that that's a good prior cuz it's agnostic. It's very flat. I'm here to tell you it's a terrible prior, and I'll show you why in a moment. And then we're going to consider two narrower ones, one that's normal uh zero comma 1.5, and normal zero one. This is the latent scale. So, how do we project these these distributions to the outcome scale? Well, we take each of them and we do the transformation. You can take a sample from take a bunch of samples from each of these distributions and just transform those samples to the probability scale using the the logistic function, right? That's all that the logit means. And when you do that, um this is what you get. So, for the first one, it turns out this very wide flat prior on um alpha on the log odds scale implies that uh uh uh an outcome distribution where things are almost never happening or always happening. Why? Because almost all the probability mass is outside of four and minus four. Right? A normal zero 10 is extremely wide on the log odds scale. So, it's saying that we expect almost all of the events to be on or off, and that's what you're seeing in the upper right. Yeah, challenges your imagination because you think that the middle point is somehow special and it's going to show up, but the whole distribution is what matters and where the mass is distributed. And that wide Gaussian distribution has a bunch of mass out at like 20, which means always. 10 also means always. Right? So, if you integrate over that thing, you get this very weird bimodal distribution on the probability scale. This is not a good prior, cuz that is not usually what you think. Right? Sometimes it might be, in which case go ahead, knock yourself out. But, it's not what you think. Uh Uh the other priors are more reasonable. If you want a flat prior um on the outcome scale, normal 0 1.5 gets you pretty close. That gives you a pretty flat across probability prior. Yeah? There's no way you could intuit this. Right? That you just have to do the prior predictive simulation. And as you start adding predictors, uh that's going to increase the variance of the prior, so you need to make all of these priors narrower just to keep this thing flat. Yeah? But, you can do that. I've already taught you how. Just do the prior predictive simulation. You can totally solve the problem on your own. Um I prefer things even a bit narrower because I like regularization. So, the bottom would be my conventional choice quite often, right? I'm skeptical of extremely high and extremely low event probabilities before I see the data, right? And And there's a whole chapter in the book. Um I'm sorry I haven't haven't gone into it as much in this in in lecture about overfitting and how much of a hazard that is. Uh you should definitely read that chapter. Uh regularization is a good thing. Okay. Let's talk about mechanics. Um Uh here's uh doing the prior predictive simulation to show you uh I keep telling you you should do the prior predictive simulation, so I would be an if I didn't give you code. Right? So, here's some code you can modify. Right? And then put in your own linear models into. Uh how do we do prior predictive simulation? Uh we sample from the priors as we define them. So, we sample uh here I'm doing uh uh 10 to the fourth samples from for alpha and beta. That's way more than I'm going to draw on the graph, but just to show you. Then I make a sequence across the X values that I want to simulate over. Um And then I just do the inverse logit. Inverse logit is another name for logistic. So, inverse logit of the linear model transforms onto the probability scale. Now we have probability curves, and then I just plot them. And we get these plots here. So, for alpha zero normal zero 10 and beta normal zero 10, you get these really strong relationships in the prior, is what the model thinks. It's like a prior that's saying X is really strongly positively or negatively related a priori to the outcome. And that probably isn't where you want the prior centered, right? You want something that's closer to zero. Yeah? But, you know, knock yourself out. So, here's here's how what you see when we change it to zero comma 1.5 and zero comma 0.5. You get things that are much weaker a priori relationships between the things. And then the data has to overcome the skepticism of the model. Yeah? Which I think is a better situation to be in. Yeah? Does that make sense? Okay. Let's return to Berkeley. Um We're going to model these data now. Uh These data are not disaggregated into zero one counts, but they're you could do that. I'm going to show you in a few slides what I mean. They're aggregated into counts, and so we're going to model them using something called a binomial. So, here's the distinction. When people say logistic regression, uh they might also mean they might just mean it has a logit link, but usually what that means is the outcome is a zero one binary, on or off. That's usually what logistic regression means when people say it. Um and that's when that corresponds to a Bernoulli distribution for the likelihood. Uh the The regression is when we've aggregated for all of the applications that have the same predictor values, that is the same department and gender, we just aggregate the counts. We count up all the ones. Yeah, and that's then we have an aggregate count and that's a binomial variable. A binomial is you do a bunch of coin flips and you count how many heads you got. Right? So, that's what it is. Uh but in this type of modeling case, we need to make sure that all the right-hand side variables are the same. You'll get the same estimates. This is This is just about how you format the data, but you're going to get the same posterior distribution modeling the same data set either way. You can always move between them and do it both ways. Um here's In this data set, here's the difference. Uh we think about an aggregated data set on the right where G is is gender and D is department, just for two departments, and then we have um the number of acceptances in the A column and the total number of applications in N. We can disaggregate or explode each of those four rows on the right into um N. So, row one on the right implies 355 rows on the left. Yeah, with zero and one. Right? There will be 31s and uh 325 uh zeros. Yeah, so it'll be a really long table which goes all the way into the basement. Uh but uh it's totally fine. And it'll still run fine, too. Uh it won't run as fast. It'll be slower, but it's it's you're going to get the same posterior distribution. Does this make sense? We're working with the ver- version on the right cuz I think it's better for teaching. But if you come across a data set which is already zeros and ones, you don't need to convert it. It's totally fine to to work with it that way. Okay, here's the code. Um Uh I think the only thing I've commented here is to note in the Ulam code uh the binomial, I have the word binomial there now, and you literally just write the link function in front of P. Yeah, when you when you're writing Ulam code, you're writing functions that are going to get processed in Stan. So, you could just write these things in like that. Um the first thing we're going to estimate is the total effect. So, the design of this model is just to stratify by gender, and the linear model is just an alpha subset by G. So, that means there's two alphas, there's a vector of length two cuz there's only two self-reported genders in this. It was 1973. Right? And uh uh and so, that's all it does. Does this make sense in how it corresponds to the estimate for the total causal effect? We're just going to stratify by gender. Um the direct effect uh since you are all masters of D-separation now, you know what you need to do. You need to stratify by department. Um and I'm going to do the full interaction stratification because uh that's my default. Um so, we're going to have a matrix of alphas now uh indexed by G and D. So, we're going to have four uh or sorry, more than four cuz there's like six departments, right? So, we've got two genders by six departments, so 12 alphas. Right? For each department, we estimate two alphas. That's the full stratification. Yeah? We're estimating the the the gender-specific probability of admission in every department. Yeah? This is the right uh the right estimator uh for this question. Does that make sense? Um people always ask me, uh do you have to do the full interaction? And the answer is no, but I leave that as an exercise for the student. You could have uh an alpha um uh stratified by gender, and then a separate set of parameters for the average admission rate in each department, and you're going to get very similar uh estimates in this data set. Um uh but the full interaction is the default, right? That's what the DAG assumes. DAGs and all the logic of do-calculus assume full interactions. Uh there's a tradition, in particular in psychology and parts of biology, to think of interaction effects as really exceptional things that you have to justify with I don't think that at all. I think the default is the full interaction model and you should argue your way away from it. Uh but statisticians disagree about these things. Um but remember I've tried to convince you the model preexists the data set, right? So you've got a generative model and an estimate and that determines the estimator, the structure of the statistical model. The fact that your data set is small should not change the estimator. Yeah, that's not a that's a flaw with your sample, not with the model. That's what I keep trying to convince you. So taking out interaction effects because you can't estimate them, I think is usually the wrong thing to do. Yeah, you you need to estimate them because of the design of your study. Leading them from the model is not progress. Am I convincing you of this? I know it's it's like the opposite advice that you've probably received other times where people have told you you can't estimate this, don't put it in the model. I this is backwards thinking. It's completely backwards thinking. I'm not saying you will be able to estimate it. But I'm saying you shouldn't make claims about the estimate you've already stated if you can't estimate the model that's required. Yeah, does that make sense? Yeah, I will try this sermon again on you. I promise. Okay. Uh so you'll see in the Ulam code here I declare a matrix with margins G and D. What that means is it counts up the number of unique values in G and D and makes a matrix of those dimensions. Here I think it's 2 by 6. Yeah. Or 2 by 5. I think it's 6 6 departments. Yeah. Does that make sense? And then in the linear model when we we pull out the particular G and D for each there's a separate alpha that's estimated for every combination of gender and department. Good. Um I'm still teaching you MCMC workflow. So let's do the diagnostics. Uh here's the um effective sample sizes. On the The we have this this scatter plot of sample sizes by R hat for every parameter. Uh This is good. You want all the parameters on the bottom R hat near one. And you want effective sample sizes way above 10% of your total sample size and here they're definitely way above that. This is a chain that's easy to sample from. And then these are these trace rank plots that I like. They look good. This is what healthy trace rank plots should look like. They're all jumbled up like tapestries. Yeah. No no particular color in any of those is always above or below the others. That's what you're That's what you're looking for. Good? Yeah. Okay. We can look at the marginal posteriors of these and also see the diagnostics the R hats and the ESSs. On the left there's just two parameters so there's just two to see. This is a simple enough enough model on the log odds scale. Remember that the pricey output there the rows in that blue table are on the log odds scale. So you've got to you can either use code to transform them or eventually you get so good at thinking in log odds you can do this in your mind. Right? Remember zero is 0.5 probability. Right? So each of these is is less than zero which means the average probability mission is less than fair in for both genders but one of them is lower than the other. Yeah? And gender one here is is women and gender two is men. Right? So in the total data set women are have a lower probability of acceptance than men. Do you see that? Yeah? This is worth transforming onto the probability scale and you should do that and I do it in the book but this is you get practice just like glancing at the table. Yeah? You've also got to transform the intervals so this is the thing you don't want to do in your head. Use the inverse logit but using the posterior samples but you can get a get a sign from it right away. And then on the right this is a mess. Right? This is just a total mess. If you stare at this long enough, maybe you could figure out, but this is deep in type prediction engine land already. So, we're going to process these samples. Um oh wait, I had a joke about this, right? You get good enough you I don't know if anybody has seen The Matrix here. It's an old movie at this point. It's from last century, uh but there's a great scene in it where this uh one of the sub villains, this guy, um has a great line about how he just reads these monitors and he can actually just see uh the lives of people taking place inside The Matrix in them. This is what you get like as an applied statistician with log odds. You can surely just like read the log odds. You just see almost always almost never. It It happens remarkably fast. Okay. So, here's the update. We're up to some estimates. Now, we need to push these out to predictions and inferences because more important than ever, these parameters are on an inconvenient scale and they're going to interact uh to produce predictions. So, we need to um use the generative model to push forward on to simulated interventions and descriptions of the population. So, let's start with the total effect. Uh what we do here, we're going to create a causal contrast. We imagine an intervention uh on reported gender, not an actual gender, but reported gender. You can imagine like changing doing an experiment on the admissions officers and relabeling all the names, right, of the applications. Um and so, we construct first a population where everybody every application is from a woman. That's uh probability of admission {underscore}g1. And we just do the inverse logit on alpha all the alphas, all the samples for alpha one. That's what that means. And then another imaginary population where all the uh reported genders have been manipulated to be so the applications are for men um and we get that uh prediction. And then we we we construct the difference on the probability scale, not on the log odds scale. So, it's you're going to get a different answer if you do the subtraction on the log odds scale, and it's wrong. Right? Cuz what's different is the probabilities, not the log odds. Log odds are gears in the type prediction engine. So, you want to push everything to the outcome scale and then do the contrast, always. Okay? I'll leave it as exercise to the student to prove if you do the order differently, you're going to get a different answer, and it will be wrong. Yeah, so it's important to do do this right. The if you're concerned like how would you ever know, summarize last. Push forward to the outcome distribution for each of your simulations, always, all the way to the end to the outcome distribution, and then do summaries. And those summaries involve calculating means, doing differences, whatever it is, but you always summarize last and you will never make a mistake. Okay? Trust me. I promise you. That's the way to remember this. Yeah? Um And we see here uh this is this uh men have an advantage in the total uh population of applications in this. Um it's it's not very precisely estimated, but you know, somewhere between um um or say that women have a disadvantage somewhere between uh 0.18 and 0.1 units of probability in the total sample. Which is not good, right? That's But we don't know we still have to decompose that into the different paths. Does this make sense so far? Yeah? Um All right. Here's the direct effect. I think I just have time to finish this lecture. This is great. Uh so, the direct effect I'm going to work through this in stages. So, I I realize this is very ugly and complicated, right? It's like, you know, data visualization is my passion, right? Kind of plots. This is just like one of the worst plots I've ever made. It is going to get worse though soon in a couple slides. But I'm building up to something just to show you, try to illustrate these complicated effects. So, the direct effect uh depends upon the department. Right? Remember I did a full interaction? For every department, I've got uh the probability of admission for each gender. Uh so, we can look at the contrast between men and women in ever- every department. And that's what I'm doing on this graph. I've done the contrast on the left, but now by each department. Um and I haven't labeled these departments to avoid shaming the innocent or the guilty, as the case may be. But, you can have your theories. What I want to show you is um there's a diversity of effects. There's this red department on the right where women are advantaged uh by quite a lot. Um and then there are uh a number of departments where men are slightly advantaged. And then there's that yellow one, which is quite fair and right in the middle, and also the cyan one, where there there seems to be almost no difference. There's a diversity of effects across departments. It's probably not too surprising to you, but it means there's really heterogeneous effects, and the total effect depends upon the weight of applications across these departments. Yeah? You with me? Um Uh the the red is engineering, by the way, I think. >> [laughter] >> If you're if you're wondering. So, there weren't a lot of women to apply to it in 1973, but most of them got in. And I'm going to problematize uh the interpretation of that next lecture, I think, for you. Okay. Um Uh so, now we're but we're still in the trying to understand whether what is the average effect now uh across departments when we stratify um uh by department. That is, what is the average direct effect? We stratified by department so that we could isolate this bottom path, remember? The the path that feels the most like discrimination? The taste-based discrimination? Right? You you want to picture the uh the sexist uh reviewer who's just angry that this woman thinks she can be an engineer something like that. I'm joking, but I'm not. Right? this This stuff This stuff happens. And so we we can isolate that effect now if we're willing to assume this is the DAG, but if we were going to say what's the average effect across women applications, what would it be? Well, it depends upon the distribution of applications across departments. So, let's just take Let's start with the empirical distribution in this data set across departments. Right? And in that case, so here I know this is a bit of code, but you can work through it at home. By the way, this script is in the scripts folder. Yeah, all the scripts are in the scripts folder. So, you can reproduce all of my plots and all the examples with the with the R code in the scripts folder. So, we're going to First thing I do is I count up all of the empirical total applications. I construct a fake data set where I'm going to do simulations for where I've got the same number of applications now are going to happen. And and I'm going to do trials on each of them to see if they get accepted or not using the samples from the posterior distribution. And then I average across all the departments here, what are the total fates, and we get this very weird-looking distribution on the right. But you know where this came from, right? That lump on the right is engineering. Yeah. But this is what causal effects can look like in look like in the real world. They can be quite heterogeneous depending. So, the average You can take the expectation of this whole distribution and it's a little bit positive actually, but that would be a weird thing to report given the shape of this distribution, right? Cuz it's bimodal. Yeah? I like this example for this reason, but there's nothing unrealistic about this. This is real data. Yeah? Does this make sense how I've gotten to this? Um We can disaggregate this, right? You can see where it comes from. I should have advanced to this already. You can kind of just see I've smooshed together Smoosh is a mathematical word. Smoosh together the bottom to get the top. Yeah, integrate I think is >> [laughter] >> is a better term here. Uh last thing I want to do is post stratify. We might want to make predictions for some place other than 1973 Berkeley. Right? Uh say you wanted to to use these uh probabilities per department to make predictions for another UC campus. Right? With a different distribution of applications across departments. There are other campuses. So, for example, like Berkeley is a big engineering school. It was back then, it still is. Yeah? Uh big computer science school now. Computer science was not a huge field at the time, but uh '73. But um other UCs are different, right? They don't have as big of engineering schools, so there're going to be fewer applications in this and then less exposure to that situation for applications. Does that make sense? How do you do that calculation? Uh well, I'm here to help you. So, so this is a very powerful technique. It's a routine thing in applied statistics called post stratification. So, we've done the estimation, we have a posterior distribution, we want to make um uh calculate a causal effect as there's a hypothetical intervention, but it might be on a different target population. Or it might be on the total population we took a sample from because the sample is not is not randomly sampled. So, if all survey work does this or should, right? If you do um an internet poll or a telephone poll, no one answers the telephones anymore, so you can't do a telephone poll, but uh some kind of poll, you're going to get responders and they're not responding in proportion to their proportion in the general population you want to make a prediction for. So, all election forecasting is like this, right? You're going to get uh particular you're going to get over representation of old people in your poll and under representation of young people. Happens in almost all election forecasting, all all polling. Then you need to re-weight your sample, right? You're going to construct estimates for each combination of the demographic effects, age and gender and whatever, you know, location uh uh for the people who did respond and then you're going to post stratify that, project it to the actual demography of the population that's going to have an election. Does that make sense? Yeah, question. Okay, but how does it you don't have the like if they were accepted or not, but you have applications, but there's no way you could have done this. Exactly. So, it's like we we So, in the context of, yeah, in this in this case we're going to predict for a different school, we're going to know how many applications there are and we want to make a prediction of the acceptance rates and the total discrimination that we anticipate and then we could design interventions. Like, where would we where would we go? In the election forecasting example, it's that you've taken a very small sample of the general electorate. It is not stratified like the real electorate, the age groups are completely out of balance. Uh, the the income groups are completely out of balance, but you have that data about the electorate separately, external to your model. So, you now you transport the estimates for each of the demographic categories to make a prediction about the election. That's absolutely essential to to realistic election forecasting. Um, and it sometimes it's hard because uh, there will be um, strong partisanship in some demographic subcategories that are really hard to sample uh, in these things. So, that that it's becoming increasingly difficult, I think, to do this, but it's still possible. Okay, there's enough talking, but that's what post stratification is. This is uh, just a standard part of the applied statistics toolkit. Um, it's really important uh, for transporting causal effects and it doesn't take a lot of code, at least in simple examples like this. I'm just going to make up uh, some some numbers of applications for an imaginary UC and that's what I do at the top there, apps per department. I just say um, there are six new departments and I'm going to give the first one five applications and all the others 10. The raw numbers don't matter here cuz I I just keep sampling from it in the posterior distribution to get the expectations. Uh if you make these like multiply them all by 100, it'll just take longer. You'll get the same distribution. Um uh But this is These are data you would collect, right? Or Or they might be part of another model, right? We anticipate from last year's applications this many, and then you can use last year's application counts to make predictions for the new year. Something like that. Does this make sense so far? And then the rest of the code is the same as previously. I've just rigged uh uh fake uh numbers of applications in here. And then it reweights. What I've done is I've downweighted department one, which is the engineering one on the far right, and you know you see it changes the distribution. Yeah, cuz this would be some school some UC where engineering is not as big like UC Davis or something like that. Does this make some sense? This is a really important technique. It comes It's really useful in lots of cases, especially in the human sciences where we're projecting onto some population with a particular age structure, and age is such a very powerful effect in human behavior. Yeah, well it's weird to call age an effect. It's an index of of things that cause effects. Yeah. Does this make some sense? Okay. Um Right. So, where are we? We have reached the end. I'm right on time. This is amazing. This never happens. So, uh where are we so far? Uh is there evidence for discrimination here? I think there is of different sorts. There There real structural effects for sure because the total causal effect in this population is that um an application from a woman had lower odds of acceptance. Uh a lot of that arises in this data set because different departments have radically different probabilities of accepting applications. Some of them are vastly oversubscribed, and women tend to apply more to the ones that are oversubscribed. At least they did in 1973. And that results in this kind of structural discrimination across departments. There's also evidence I think of direct discrimination in some cases. And in other cases, maybe evidence like an engineering of anti- discrimination or positive discrimination, we call it here. Right? Uh uh but in the next lecture, I want to problematize that a bit because we haven't talked about um unmeasured confounding. Uh which can mask Here's this gives away the This is a spoiler alert, but that can mask discrimination. Yeah. Okay. Um with that, uh thank you very much for your attention and tolerating my rants. Um and I will see you next time.