Computable philosophy #RB21

hello everyone today we will discuss computable philosophy a proposal Flanagan Wong and me made out of our book so in this proposal there are three key ideas that and computing judgments and information are more interlinked than people think of them the second proposal is that computing and making judgments by programming and setting rules does not scale so we need to complement them with learning with inference from observations so that is like first axis is computation judgment and information the second axis is learning how to how to infer laws and rules from observations and the third axis is probabilistic thinking which will see is inevitable if we want to do correct inference and robust inference those those three axes are when studied within computer science and and and information science what we believe is that there as we discussed in the video on put on paper is that there are more philosophically aspects to be discussed in particular when it comes to questions like AI a text or or law itself if we look at it from an algorithmic perspective and even the scientific method from an algorithmic perspective there are many illustrations we have in the book or why these three tools could help us discuss aspects like moral philosophy the value alignment problem we want to align the objective function of an AI with human preferences etc the side effects of algorithmic decision making the good heart slows what happens what goes wrong when we maximize a metric and preference and volition learning how to learn not only once people prefer but what they would want to prefer if they had more time and an information social choice theory which was researched by game theory and economics in the past century but that is increasingly important in algorithmic decision-making aggregation of preferences between many users adversarial computing decentralization also isin a stack that means a new toolbox of probabilistic thing in learning and how to look at computation as a form of judgment and finally I like all questions that are relevant for a city like reward hacking credibility etc we will not discuss all of these so we'll discuss just the three key aspects of computation judgment information learning and pervasive thinking and in the end to illustrate how they could be useful just by discussing privacy and furs which are two important questions in algorithmic decision making yep yes so this is I think a big program that we have in mind and I think it's very interesting like it's really an interesting insight that computer computer science can give us into what it means huge provider judgment good judgment the reliable judgment and to command this with molecules of inserts like this and in particular maybe you can just cry a bit by what you mean by computationally in general but one important feature is that we need this idea of of step by step reasoning like very clear steps and also if you explain ahead of time what all this is what the steps are going to be then this can allow them to analyze the step by step procedure of the algorithm that will be used and you can analyze them in in many many respects and arguably also this is something that has been extremely popular in the history of mankind that really has changed a lot of the way we do a lot of things so maybe watch which case like something people don't think of as a moral issue if you just type in a search engine go with 19 boxing so Kane's first as a result of like what games in the first 10 pages of results what to show you in the first 10 pages of results is a decision that is algorithmic algorithmically made with the search engine ranking algorithm but which entails enormous moral consequences because how you would not astray billion people about resisting or accepting vaccines for coated it would have consequences on human life so this is this is a moral question this is the philosophical question and it has a very short deadline half a second like a few milliseconds the search engine should answer it in a few milliseconds and we tend not to think we tend to think of this as a technical question oh yeah we'll just show what's most relevant or we just show what what most people are discussing what we want - for example there is a minority of people initiating a conspiracy against vaccine should we amplify it just because this minority of people are super super active on the platform yeah and also an interesting thing is that when we raise this sort of of dilemmas like I think a very common we are trying to all of this is to say that these are if it's like a very difficult dilemma like we tend to postpone the decision for this we just say that it's a difficult diet in mind we need to be discussing this and like it's a good point that we cannot make a decision right now but what we can do as of right now is try to think of how we're going to come up with a decision just like saying we'll have to discuss it is not really a algorithm that will has the right properties of of coming up with a eventually a decision and good decision impossible when we have this dilemma is important not just to leave the question open like this but to and it's also important not to just take this decision like padam institution like right like now but instead like we should try to think of as a future step by step one algorithm and propose different algorithms that will eventually make a decision because we need to eventually make a decision especially for times like here so results to search to search queries yeah I think a good illustrative example of that is something we discussed previously concerning and medium-bodied problems and the clinical trials so here we are a case where continuing as as as algorithms are right now is an ethical problem because all these decision by algorithms have ethical implications and also the solution we intend to to figure out how to do better will have to be done so we'll have to come up to a satisfactory answer as quickly as possible while certainly making modification to this algorithms so it's very important to correctly decide what procedure we implement to to which this is interestingly historically this has been like for a long time we could you could imagine that this was difficult you to think about these things because I can algorithm has to be described at least and should be explained to one another so okay I guess back in the old days I was more like transferred as transitions from one to another but at some point like mankind invented writing and the invention of writing completely proportion as the game in the sense that people would be able to write down the algorithms to be followed these algorithms by now we call them the text off flow so they are not like up like very rigorous algorithms as the one you would tell your computer to execute like the beginning of this algorithmic approach to decision-making in particular to judgments in the case of the law what by having this algorithm written down you have several nice properties that one of them is that the same row can apply to different settings so you have and look see sometimes called the procedural fairness okay the same algorithm to different people but you also have other properties are concerns now that the role is written like the algorithm is written you can analyze it you can verify it so you can say to the judge well wait a minute you did not judge me according to the algorithm you can also improve upon it I can say oh yeah the current version of the law has this flaw that this makes this decision in this case and like most people maybe think it's not good so we can change the law we can improve the algorithm so these are all features of written laws and algorithms that have been a major breakthrough in the history of judgment so maybe the takeaway message from this part is that if someone tells you they don't want to be judged by an algorithm ask them if they prefer to be judged by law or by the mood of the judge that does not tell them on which basis they judged like the judge tells you like you are guilty because I think you're guilty and if a judge tells you you're guilty because in the law of this country if you put your car in front of the police station for more than three hours and then there is an accident in the police station then you want to go to jail so this is an algorithm if it's then you realize like most people prefer to be judged by an algorithm his transparents stated public that you can know in advance ideally or that you are assumed to know in advance so actually being judged by an algorithm is historically a progress that we made thousands of years ago so just like now maybe the problem is being judged by algorithms that are that you can't read that are too too big or you read and to audit and to assess but that's the read that's the real problem it's not being judged by module it's being judged by an intractable by a non readable very long complex algorithm and we already the case the on the low as well like the tests of low have become like they are transparent in the sense that the text is fully returned somewhere but they're not transparent in a sense that it's very hard to interpret the law correctly and also just because it's long but also that because it's easy and by the way like even we go to the this example we like to give thousands of years ago in all cases you were judged by an algorithm this awkward is either either the if-else it hughes teamed one cow you have to pay the equivalent money of one cow so this is an this is an algorithm and then there is the other algorithm which we call the mood mood of the crowd the moves on the judge it's not transparent it is a mugwort it is a decision making process it's very chaotic you can't predict it you can't anticipate it so you always prefer the short clear transparent algorithm if this then this this then this then country there is nothing new we always preferred transparent here states and algorithms to obscure chaotic non transparent algorithms yeah yeah and also reject sometimes our foreign algorithms you can prove properties so for instance if you take it the game shape layer algorithm that's used to judge whether student will go to this university or this or the university then we know like my fans have been studying this algorithm and we know that finds and it has some some nice properties constant is it leads to two so-called a stable matching so I won't go into the details I guess but essentially it has these nice properties like incentive compatibility as well it's finding the right way and this can only be done if you have known algorithm that it's very hard to predict the properties like this ability of the mood of the crowd of a single human yeah now having said this there's one limitation to the law which is the fact that it has to be written by by humans and we humans are like all very smart and all but we also are limited in your connection and we we often have trouble imagining cases that have never occurred before in the past and also like the world is getting more and more complex so it's getting harder and harder to to design the right algorithm to judge in case in no societies and that's where instead of just writing things down we often rely rather on the brain of a judge and on the than on the law as it is written and this has an explanation again in terms of computer science particle in terms of what is known as as the Solomonov complexity also known as the Kolmogorov complexity which is is defined as the shortest algorithm in terms of like the description of the algorithm so the photos text upload that is able to do what he wants to be doing and there are strong arguments like from chewing in particular that there are many things that cannot be shortened to like they are probably a good text or blow does not fit in a book of two or three hundred pages and maybe does not even fit in one thousand bucks of of one thousand pages because the world is just really complex in a very meaningful sense in this case we cannot have written laws that contain everything we have to do something else and this was proposed by chewing in 1950 and is the idea of doing learning so instead of writing everything is down you're going to learn through experience what ought to be done and and this also occurs in in the case of the law and is known as jurisprudence and by the way this occurred also in science and it's known as the scientific method like now or before the revolutionary idea that we call the scientific method started to be of course it existed for at least ten centuries but it took off it took off with the Galilean revolution Kepler Newton and so on and then we started inferring rules of nature from observations and this is where were where human knowledge took off because it just scales more then having someone sitting down like a wise philosopher and then stating the rules of the universe yeah yeah yeah the case of science is interesting because like you have like two texts all blue it takes two of the rows of the nature of nature that have been written down more and more and we have to improve these algorithms but then we also went further we ask ourselves how should the text of the laws of nature be returned like how should we come up with the right laws of nature and this led to some sort of meta algorithm or the austere algorithms but they are learning algorithms algorithms about how to find out about the laws of nature and this is also known as epistemology so that's why there's also a natural link here which in computer science and and special learning theory and and epistemology philosophy learning is a what we took off the bottleneck of learning was really to have a lot of data because you to do learning but you also need some a lot of continuous from power to do the computations of learning algorithms and also you need like like allows memories and subsidizing machines to do these things but once you have all of these things it turns out that learning is much more efficient than while you can do this in human brain as well but that groaning is more much more efficient than just writing down the text or glows because watching down by humans is just too hard but it also comes with some disadvantages though this is not a keep you concur to to the empty set sometimes there's no algorithm that humans have written that's able to recognize a cat with 99% accuracy for instance so when you're saying that the algorithm to recognize cats has some flaws for instance it's not as transparent is not very transparent well you actually comparing this to the empty set in a sense so but still it could say now we have these algorithms or these are judges that learned from past occurrences what they should have they should judge in the future and these algorithms are now too complex to be studied using the mathematical tools that we usually use for for small algorithms and so it's harder like the all more black box easier or harder to to understand this is our limitations but it's a limitation that inherent you're learning and that's like you cannot do without learning I guess for some tasks and like this creates just new challenges that needs to be to be faced the verification of learning algorithms of algorithms that have learned is much harder than of algorithms that we designed to be analyzed look again here Adama Lee wants to say something about the learn learning you go ahead [Music] which wanted to conclude this part which like there again a simple takeaway message if you hear just like first the first time exposed to these ideas that like maybe the takeaway you should keep here is that hand programming rooms does not scale like that's that maybe the green key inside of chewing is that we could not sit down and start writing who's like if this do that if this do that if you speed up and produce ass marks set of rules the smart algorithm I've written is a set of rules so Turing realizes that if we want to speed up the programming of an intelligent algorithm we need if are willing to be adaptable so it's has like if this or this or this and this like those conditions are could be tweaked it could be modified depending on what observations have been made modifying the conditions of the if and else with respect to the observed like I observed that when I when I do this four times I get this results but then I had a new experience where I only need to do it three times so maybe this for the parameter for could be moved a bit down and then I realize no no 3.5 is on average better so I the algorithm needs to have some parameters that can be modified depending on the observations or the experiments and Turing argued that this this is faster more efficient and this is realistic we can have a program within our lifetime that becomes intelligent in some in some sense not contagious like realizing an objective if we let it learn from data and if we have wants to hand program it which will take us a very very very very long time of writing moves and this is more or less what happens like you take a task like image recognition for four decades or five decades people were trying to come up with handwritten rules okay there is a polygon like this and then you shake look at the shape and the nose etc then this isn't many in pink the polygon is like that and the nose like clears and the mouth and I don't know the ratio between atoms just like Mickey not then it is probably a woman and this did not work but if we feed an algorithm many data points and let the algorithm like however what we call a learning algorithm so that it can change the parameters now we achieved of course we are not yet there but we have for example algorithms need Facebook the turpentine spaces and they recognize that this is me and this is Katrina and this is etc and those algorithms clearly we could not have obtained them by ran handwriting if this is with that but then you just let them learn from data and and and this idea is very old it's from 1950 it was by Alan Turing and it is the key idea behind learning learning the scales better than programming so if we want to write algorithms we want to write laws we need to compliment programming by learning and sometimes we sometimes we mainly need learning and argue me in the context of law it happened all so we call this dirichlet jurisprudence I think in English all soldiers can also French we're like you observed cases and you make up rules based on cases that please everyone some so to say we observed it when when we punish a killer with this punishments there is no riots they like drone is happy with this punishment for almost everyone when we punish the killer with this punishment people are not satisfied the family of the victim are not satisfied clearly this low need to change so this is the learning process writing law itself is a learning process and this is again another point where law and Al bridge mix means just like they met initially thousands of years ago another point I'm thinking of would correspond much with the the first section of the podcast but is that no I'll give you two ways to think about the way we write laws this is a discussion that I've had with Gilda Wieck who told me this very interesting like century you can think of the law as either an algorithm another way that people sometimes feel like rising the role is as specifications like this must happen this must happen this must happen this is not an algorithm like this is just the things that you want your decision to satisfy and the the the annoying thing with specifications whether the good thing is that is arguably easier to write specifications like you can just say oh yeah like this but the trouble with specifications is that well sometimes the the set of specifications describe an empty set meaning that all the specifications want to put you you want the law to satisfy our means that there's no such decision that can satisfy or all this decision and that's why I think it's it's at least interesting to not just stop at specifications which arguably is a lot of what people are doing when they discuss guidelines for for a is equal to a I can say that a good a I need to satisfy these these these these and I think is you but can only be seen as a first step because like eventually I think we need an algorithm to know what should be decided and not just like what are the specifications are it's something an interesting also like if you have an algorithm you can also analyze that you although things like computation time because we know from like enjoying help during the halting problem that just determining if there is a solution to a set of specifications if there is a exist decision X and such that this ends and this is astigmatism it's a conjecture and we know from chewing that determining if this contract is true or not or has a proof or not is a is undecidable in general's mean that there's no algorithm that achieves this all the time so that's not the argument for why we should think in terms of algorithms rather than just a specification the third discussed is a probabilistic thinking which is clearly critical in the case of the court of law even though it's been forbidden in the UK after some idea the singing departments that a lot of people are including myself very hard have a very hard time thinking probably sticky it is just very very hard but I'll give it's also very critical so the way sometimes things are phrased in the context of law and of science is people will talk about truth and if you think about this well proofs are only well-defined in mathematics but in the context of of science or in a context of law what we have is more evidence like we have data essentially and based on this data we can infer we do the learning from this data we infer what is more likely to have occurred or not but you never get to do any point of certainty because it's always possible that there's some like explanation that we have not thought about it's much more complicated and actually these more complicated or unforeseen explanations or arguably quite frequent in the case of the law so you need to take into account this uncertainty and you need to reason with uncertainty to to come up with decision so instead of saying if the person is guilty then we should do this and I'm saying if the person decimals and then we should do this which sounds very good but in practice you never get to this state like you should think in terms of like well even how likely it is that he has done this and this what should be decided for this person this would be much more probabilistic thinking and you might think is very weird in the text in context or blow but yeah sometimes you just don't have enough data and becomes even more critical in many times find sense that involve a lot of uncertainty for instance for the current situation so what should be answered when you when someone is searching career vaccine good night vaccine on Google for instance this is a very very complicated question because also we don't know so far like how long it's going to take to have a vaccine how data is the vaccines are going to be how are they going to be a producer board at scale there are lots of open questions and what you're going to reply today to these questions is very important to prepare the the population for what's coming next and so you need to make a decision right now despite the huge instant ante on what's going to come around like in the next month yeah one example for this world or was the one of the legatus study on the hit rock synchro rocking that was retracted a few weeks after so because there is a possibility that when you see a study it it was actually not the high quality information that you expected but sometime quite often it is actually the high quality information that you expect but raising a decision based on disk on this kind of evidence which as they say is not a clear proof that is that that will tell you 100% what is the area to adopt so you should treat this as an evidence knowing the possibility that it was actually a there was actually mistakes on the on the process of creation of these evidence and that's why also much stronger evidence that we should look at is things like meta analysis or due to global context in which the whole science is produced and without this Provost ik thinking in mind then we get into a mistake either being absolutely convinced that authors of the study are trying to manipulate the the result due to conflict of interest or being on the other side foon convinced that the key toxic work in treatment is absolutely so that there should not be any any we should not be at weenie and puppies extremes like we should consider every piece of evidence as something that moves slightly our probability estimators of what our right decisions to take a given the situation yeah yeah so there's a lot of work and it's very hard to be cuppa tea we need to improve this like it's really critical for better decision making to improve in terms of priorities probabilistic thinking in particular estimating more correctly the probabilities of different events and then there's this other side of of kava stick thinking which is now that you have this and society what should you what should you do and one thing that is very hard but you really should really be done is to not reflect only in terms of the most likely scenario it's very tempting to say well I believe this and you forget that you doesn't mean that you fully believe it and it may be like a five percent chance that the alternative scenario occurs and this is particular critical in the cases in the case of pandemics for instance because if you were back in February or January I'd say 2020 for those who watch this announcing the future then there were different scenarios and maybe you could imagine that the more likely scenario for the covenanting outbreak but then not yet the pandemic was that it would not be a pandemic and maybe right now you could say that maybe in 20 2021 there's like probably the most likely scenario is that there's not going to be a pandemic of another virus of another disease that's much worse than the curve in 19 that's the most likely scenario but you should not think in terms only of the most likely scenario and we should prepare for the possibility that things go bad and particularly we should prepare for this if the probability of this thing going very bad is not too small if it's one person I'll give a of something extremely bad it's already huge but if it's like ten to the minus twenty while it's negligible and there's a big difference between 10 to the minus 20 and 1% but it's very hard for us humans Act which make this distinction because you tend to to confuse like you to consider that these two are just unlikely scenarios yeah maybe to illustrate the difference if something that has one percent chance to happen every year out of 1,000 years it will nearly happen for sure but something that has 10 to the minus 20 chance to happen if you out of thousand years it really not happen for sure yeah yeah and so the decision-making has to take into account and sometimes the safety mindset they trying to make sure that you compute it to the probabilities are very very bad scenarios and if this probability is not that smooth then you should at least plan for if it occurs and maybe then plans to to reduce this probability maybe also so one thing about some holistic thinking that is really including your people who work in Polynesia the same thing we keep saying about us people working in computer science we neglect how how it is technologically concepts we have in computer science can be and can be applied outside computer science I'll recommend the book on Brian Christian and hungry kids algorithms to live by in which illustrates this fact actually for probabilistic thinking to live by there is the 200 years old book written by small as decimal a test which is called a sh t loser fixed or equality philosophical essay on probabilities and it fits like in some chapters you could you could see the tremont like the preliminary version or for example a lot of the work that has been done in the 20th century about coca-cola Tobias's for example and it is laplace cold like today we call them cognitive biases Laplace calls them illusions in estimating probabilities and illustrate that with lighting it's for example being biased towards what is common and what is familiar to him and what has been told to him in his childhood and trying to see it's in phenomenon that has nothing to do with the like he likes like me this once rose to the Chinese emperor the Chinese ever liked maths and I'm trying to convince him of Christianity using a phenomenon in a series sums of series by telling him look at like you can have one out of zeros and this is creation just and then and then la classe goes on more brilliantly that what I just have said I'm only not reflecting how near the statement of La Paz was just like showing like how much when you are like used to something and exposed to something during our childhood or during our life we tend to be biased for it for confirming it and seeing it everywhere we look he shouldn't was like and then and also like he's giving examples like for example is the given the example of slavery and castes in India as something that people normalized because it's common and then he goes on to expose why frequency commonality are not valid epistemic arguments so it's something it's frequent or if something is common but doesn't mean it is okay either morally or it is like it's like commonality is not valid a piston here has an epistemic argument or as a moral argument and of course he also does a lot of connections with moral philosophy and unfortunately this work is really overlooked like by by people who work in colleges who's never taught this course discovered it's ten years after my more than ten years not twelve years after my undergraduate studies yeah again I'm not going to make a lot of friends by saying this but I think this is the best book includes of you ever written it's really fantastic like have really highly highly recommend it and I just want to say to quote well to two sentences I was going to say one but I mean you could too 13:27 good right the first one is the theory of probabilities is basically just good sense reduced to computation well I think it's fantastic quality it's like it's very border claim if you think it liked it like here like more good judgment than good sense of common sense yeah yeah you straight it into common sense I can't include sensation but I think this quote is like really really it is really really bold and I think it's very very good I think it's like very a lot of food for thought it's really also aligned with what we've been saying like it's a competition can reduce things to computation you've done like 99 connection and because I saw a job in a sense what speed are - you see - to effective competition yes but it's really really important like this book and the theory of probabilities and he's inside and like just like the other quote I wanted you to give is that there is no science but other than the science of probabilities more worthy of our meditations and whose results are more useful it says better what I may be just to conclude will illustrate all of us have been said now with the of course like we'll be superficially discussing fairness and privacy those are very very very complex problems like and unfortunately some researchers tend to feed them unlike them just like we can just tackle them with some solution and then they're so they're like privacy and fairness are not as chargeable as I don't know let's say convict proving convexity of a loss function like sometimes those are very very complex topics so obviously or not or not broadly discussing them or just like superficially discussing them just like to a narrow angle which is how probabilistic thinking helped us in the past decade improve the way we think about privacy and fairness so maybe you can start by differential privacy which is maybe more natural now ten years although more than ten years like 12 years old at least and then we can move to Turnus which is even more even younger but it's built upon some of the reasoning that has been doing it been made in different for privacy so I don't know if you want to go with that you know only that's the idea of differential privacy is that when some of your data is always going to be like you instead of raising your data you're losing some noisy version of the data such that a new observer would be like unable to infer with high probability what your what your true data were well it's not exactly that but it's more it has more to do with how much he can change his beliefs like how how can how much it can update his beliefs by having seen the data that you were releasing so that that would be like the probability or interpretation of the concept called differential privacy which has been the leading one of the leading concepts for privacy over the last 15 years the other one being this concept what the other big line of thoughts in terms of privacy being this complexity really like you you you cipher your message such that no observer that has a limited computational power can this I can can learn anything from your formula your message so what's in column to see here is the how actually probabilities come up into a corrected if this this very interesting definition of privacy and not only as we were discussing earlier the binary definition of private or not private but something that is as much private as possible by by showing as few as possible bits of information about yourself it's um information meaning how does someone change what it thinks of me based on the data he has coming from me and the less bits of information unless someone is able to update the probabilities yeah yeah it's also interesting because you can then think in terms of trade-offs there's a lot of research about like okay if all data remains private in different short privacy terms then what you cannot learn anything from from the profession and this can be a problem the concerns in the case of the pandemic you do want you to know like things like what is the the fraction of the population currently has the coven 19 this is really important information to have to reach you know whether I should do lockdown or whatever but this information for individuals or every individual can be also cause of concern for this individual so you you want to learn but not too much about your cooperation in a sense and differential privacy gives you a way to do to to write down this trade-off and to compute the depending on how much you care on controlling the pandemic and avoiding deaths and how much you care about privacy and surveillance right yeah you have a natural promote you to do the to choose your trade-off yeah it is a an additional argument against simply having list of specifications for other items and so found that if you have to specification like the algorithm should avoid death and the algorithm should remain private then what happened when some specific decisions can avoid death but by being intruding at all as privacy should we take these kind of decisions that don't fit one specification but are imposed by the by the first specification and and the right answer to this I believe is to to think of it in terms of trade-offs so somehow having an estimation of how how important it is to to avoid death how important it is to to about privacy and it's great to have a measure of how much actually are we inviting privacy with this differential privacy definition to to allowed to take decision in one facing skeletons so what's interesting also in terms of finesse is that so what did the basic idea openness is typically you would want them to to to go to to guarantee that for two different subpopulations like some consensus and rate of of job offers is done so the probability of Ghia of having a job of receiving a job offer constants given that you're from this population or given that you calm disorder population maybe should not be true to different so this is called group fairness comparing fairness between two groups and there's another idea of fairness which you can think of which is individual fairness which essentially means that you are treated given audio data so you every feature of you is rightfully taken into account and so this would be more about your policy of getting your job given your your what what is known about you it is publicly known about differences and it turns out that and you may say yeah yeah every individuals would be judged based on on his competence for instance and you say we also like there should be no disconnection between the different group so you want may what you may want both individual fellows and goodfellas so there would be a specification approach and it turns out that you can prove mathematically that in many cases or in both cases these two are incompatible so the so the specification approach to finesse would be well they cannot have all all versions of an S at least simultaneously so so then you need to specify this more and and again like you to compute the trade offs and what we really mean by fairness the language of poverty has out to be very very very early I had like few things to add but I think like just like maybe to take away from this part is that like again we're like very superficially tackled three fairness because it's a people are just realizing that it is a scientific question hey so of course is a socially very important question but also that it highly scientific question that could be tackled with the scientific method maybe just like a side note sometimes we still like in some communities the topic is not as highly regarded and as I don't know proving some conversion speed of spoke a steep gradient descent on a convex function and I believe this is this is not this is this is something that that is not okay for example in the machine learning community - like this regard research in earnest as non-technical actually it's it first of all it's a very highly technical question and if we go back to the beginning of the broadcast algorithms like the researcher that gave us the name algorithms was actually trying to improve lo he was a lawyer and he was trying to make law more rigorous more transparent and this is how we invented algebra and algorithms by trying to by trying to improve law so working on fairness is extremely relevant for computer science and it's extremely it's an extremely interesting research topic so just like if they're like grad students watching this podcast please don't disregard this stop it like you shouldn't and I can it's not like just like about like please don't do it like if you are doing it you're wrong like if you are doing it you are really disregarding a research question this is highly technical highly subtool highly complex and unfortunately seem like very respected researchers disregard the topic and that that's not that's a bit sad and unfortunate I hope like now it's just a generational problem yeah personally I didn't experience that with your younger researchers mostly the old generation of computer scientists it means the question didn't look at these questions as maybe what's happening today so maybe it's just a generational problem that is going to improve itself with time there is a very good book written by Aaron Ross and Michael Kern's my memories good iron rod was a PhD student of cinch at work since yet work is the researcher to which we owe differential privacy so our roof also low 25 for privacy and armored attack she she she she did the sentient work a lot of very relevant research neck she has a very broad portfolio of questions that she tackles short in distributed computing initially they were typically computer science that she also gave us the formalism on differential privacy and now with the researchers like iron rod and others there is no growing community around fairness accountability and transparency conference by ASEAN we could list we could go on with the biggest of research like in just like the conference and look at the proceedings I don't want to name a few and not name the others but this this is this is obviously like this is a question where probabilistic thinking is is very helpful if you look at the statement of differential privacy it is a holistic statements so I think we have a video on at least there is a data byte video on on differential privacy but you could look it up and the same now applies for fairness it's it's not something you define the binary way or in a formal like this like it's not it's not something you can do with first-order logic it's something where probabilities are not a luxury or unnecessary so with that Sonya we can just good so this here next time