Privacy Preserving AI (Andrew Trask) | MIT Deep Learning Series

today we were happy very happy to have Andrew Trask he's a brilliant writer researcher tweeter that's a word in the world of machine learning and artificial intelligence he is the author of grokking deep learning the book that I highly recommended in the lecturer on Monday he's the leader in creator of open mind which is an open source community that strives to make our algorithms our data and our world in general more privacy-preserving he is coming to us by way of Oxford but without that rich complex beautiful sophisticated British accent unfortunately he is one of the best educators and truly one of the nicest people I know so please give him a warm welcome thanks those very a generous introduction so yeah today we're going to be talking about privacy preserving AI this talks can kind of come in in two parts so the first it's going to be looking at sort of privacy tools from the context of a data scientist or a researcher like how their actual UX might change because I think that's sort of the best way to communicate some of the new technologies that are that are coming about in that context and then we're going to zoom out and look at under the assumption that these kinds of technologies become mature what is that going to do to kind of society like what sort of consequences or side effects could could these kind of tools have both positive and they give so first let's ask the question is it possible to answer questions using data that we cannot see this is going to be the key question that we look at today and let's let's start with an example so first if we wanted to answer the question what do tumors look like in humans well this is pretty complex question you know tumors are pretty complicated things so we might train a classifier if we wanted to do that we would first need to download a data set of tumor related images right so we build sophistic we start these and be able to recognize what tumors look like in in humans but this kind of data is not very easy to come by right so it's it's very rarely that it's collected it's kind of difficult to move around highly regulated and so we're probably going to buy it from a relatively small number of sources that are able to actually and managed this kind of information the scarcity and in sort of constraints around this likely to make this a relatively expensive purchase and if it's going to be an expensive purchase for us to answer this question well then we're going to find someone to sort of finance our project and if we need someone to finance a project we have to we have to come up with a way of how we're going to pay them back I'm ready create a business plan and have to find a business partner I'm gonna find a business partner we have to span all our classmates in LinkedIn you're looking for someone to start a business with us right now is because we wanted to answer the question what do tumors look like in humans what if we want to answer a different question what if we wanted to answer the question what do handwritten digits look like well this would be a totally different story right we download the data set we download a state-of-the-art training script from github we'd run it and a few minutes later we have you know a ability to classify handwritten digits with potentially superhuman ability right if such a thing exists and why is this so different between these two questions the reason is it getting access to private data data about people was really really hard and as a result we spend most of our time working on problems and tasks like this so imagine a and s if R 10 anybody's trained a classifier on M this before raise your hand I expect pretty much everybody instead of working on problems like this does anyone trying to cause fire to predict dementia diabetes Alzheimer's like is she going depression anxiety no one so why is it that we spend all our time on tests like this when these tasks these represent you know our our friends loved ones and problems in society that really really matter not to say that there are people working on this it's absolutely you know there are there whole fields dedicated to it but but sort of the machine learning community at large these tasks are pretty inaccessible in fact in order to work on one of these just getting access to the data you'd have to dedicate like a portion of your life just to getting access to it whether it's you know doing a start-up or or you know joining a hospital or or what-have-you whereas for other kinds of data they're just simply readily accessible this brings us back to our question is it possible to answer questions using data that we cannot see so in this talk we're gonna walk through a few different techniques and if the answer to this question is yes the combination of these techniques so we try to make it so that we can actually pip install access to data sets like these in the same way that we Pittman still access to other deep learning tools and the idea here is to lower the barrier to entry to increase the accessibility to some of the most important problems that we would like to address so as as Lex mentioned I lead a community called open mind which is an open source community of a little over six thousand people who are focused on sort of lowering the barrier to entry to privacy preserving AI machine learning specifically one of the tools they're working on we're talking about today is called PI seft pi sift extends the major deep learning frameworks with the ability to do privacy-preserving machine learning so specifically today we're gonna be looking at the extensions into pi torch so if pi torch people will turn on a torch yeah quite a few users and it's my hope that by walking through a few these tools it'll become sort of clear how we can start to be able to do sort of data science the act of sort of answering questions using data using data that we don't actually have direct access to right and then on the second half of the talk we're going to generalize this to answering questions even if you're not not necessarily a data scientist so first first tool is remote execution okay so let's just uh walk walk me through this so we're a jump into code for a minute but hopefully this is sort of line by line and relatively simple and even if you are from there at PI torch I think it's relatively intuitive looking at like lists of numbers and these kinds of things so up at the top we import torch as a deep learning framework sift extends towards with this thing called torch hook all it's doing is just iterating through the library and basically monkey patching in lots of new functionality and most deep learning frameworks are built around one core primitive and that core primitive is the tensor right so you know and for those of you are don't know what tensors are just think of them as nested lists of numbers for now and and that'll be good enough for this this talk but for us we introduced a second core primitive which is the worker right and a worker is a location upon a within which computation is going to be occurring alright so in this case we have a virtualized worker that is that is pointing to say a hospital data center right and the assumption that we have is that this worker will allow us to run computation inside of the data center without us actually having direct access to that worker itself right it gives us a limited sort of whitelisted set of methods that we can use on this on this remote machine so just to give you example so there's that Corp I'm talked about a minute ago we have the torch tensor so one two one three four five and the first method that we added is called just dot scent right and it does exactly what you might expect it takes the tensor serializes it sends it into the hospital data center and returns back to me a pointer as pointer is really really special and for those of you actually familiar with deep learning frameworks I hope that this will just really resonate with you because it has the full PI torch API as a part of it but whenever you execute something using this pointer instead of it running locally even though it looks like and feels like it's running locally it actually executes on the remote machine and returns back to you another pointer to the result right the idea here being that I can now coordinate remote executions remote computations without but not necessarily having to have direct access to to the machine and of course I can get a get request and will see that this is actually really really important so getting permissions around when you can do get requests and actually ask for data from a remote machine back to you so just remember that cool so this is just this is where we start so in the kind of like the Pareto principle you know 80% for 20% this is like the the first big cut right so pros Dayna remains on a remote machine we can now in theory do data science on a machine that we don't have access to that we don't know right but the problem is the first first column we want to address is how can we actually do good data science without physically seeing the data all right so it's all well and good to say oh I'm gonna train a deep Loden classifier but but the process of answering questions is inherently iterative right it's inherently sort of sort of give-and-take and I learn a little bit and I ask a little bit I learn a little bit and I ask a little bit right this brings me the second tool so search an example data again we're starting really simple it will get more complex here in a minute so in this case let's say we have what's called a grid so PI grid if PI sift is a library at PI agree is sort of the platform version so it's sort of again this is all open source Apache to stuff this is we have what's called a grid client so this is this could be a interface to a large number of data sets inside of a big hospital right and so let's say I wanted to train a classifier to do something with diabetes right so it's mean to predict diabetes or predict certain kind diabetes or certain attributed diabetes right I should be able to perform remote search I get back pointers to throw the remote information I can get back sort of detailed descriptions of what the information is without me actually looking at it right so how it was collected what the rows and columns are what the types of different information is what the various ranges of the values can take on things that allow me to do sort of remote normalization these kinds of things and then in some cases even look at samples of this data so this these samples could be sort of human curated they could be generated from again they could be they could be actually you know short snippets from the actual data set and maybe it's okay to release small amounts but not large amounts and and the reason that I highlight this this isn't like crazy complex stuff so prior to going back to school I used to work for a company called digital reasoning we did sort of on-prem data science right so we did delivered sort of AI services to corporations behind the firewall so we did you know classified information we worked with investment banks you know helping prevent insider trading and and doing data science on data that like your home team you know back in Nashville and in our case it's not able to see is really really challenging but there are some things that that can give you sort of the first big jump before you jump into kind of the more complex tools to handle some of the more more challenging use cases cool so so basic Roman execution so remote PC recalls basic sort private search and the ability to kind of look at sample data gives us enough sort of general context to be able to just start doing sort of things like feature engineering and evaluating quality okay so now the data remains the remote machine we can do some basic feature engineering and here's where things get a little more complicated okay so if you remember in the very first slide where I show you some code at the bottom I call dot get on the tensor right and what that did was it took the pointer to promote information and said hey send that information to me that is an incredibly important bottleneck right and unfortunately despite the fact that I'm doing on my remote execution if that's just naively implemented well I can just steal all the data that I want to right I just called get him whatever pointers I want and I can and there's the sort of no additional added real security so what are we gonna do about this Springs it's a tool number three called differential privacy differential privacy little higher okay cool awesome good so I'm gonna do a quick high-level overview of the intuition of differential privacy and I'm gonna jump into how it could can can and is being is looking sort of in the code and I will give you resources for kind of deeper dive and difference for privacy at the end of the talk should you be interested so differential privacy loosely stated is a field that it allows you to do statistical analysis without compromising the privacy of the data set right so it more specifically it allows you to query a database right while making certain guarantees about the privacy of the other records contained within the database so let me show you what I mean let's say we have an example database and so this is kind of the canonical DB if you look in the literature for differential privacy it'll have sort of one row for person one more row per person and one column of zeros and ones which corresponds to true and false we don't actually really care what those zeros and ones are indicating you know it could be presence of a disease could be male-female could be it's just some some sensitive attributes something that's that's worth protecting right now what we're going to do is we're going to our goal is to ensure as physical analysis doesn't compromise privacy what we're going to do is query this database right so we're gonna run some function over the entire database and we're going to look at the result and we're gonna ask a very important question we're going to ask if I were to remove someone from this database say John with the output of my function change okay and if the answer to that is no then intuitively we can we can we can say that well this this output is not conditioned on John's private information now if we could say that about everyone the Dave in the data day base right well then okay we would be a perfectly privacy-preserving query right but it might not be that useful but this intuitive definition I think is quite powerful right the notion of how can we construct queries that are invariant to removing someone or replacing them with someone else okay and the notion of the maximal amount that the output of a function can change as a result of removing or replacing one of the individuals is known as the sensitivity okay so important so if you're reading the literature you look you finds come across sensitivity that's been talking about so what do we do when we have a really sensitive function we're gonna take a bit of a sidestep for a minute I have a sister a twin sister who's finishing a PhD in political science and political science often they need to answer questions about very taboo behavior okay something that people are likely to lie about so let's say I wanted to survey everyone in this room and I wanted to answer the question what percentage of you are you know secretly serial killers right and not because like yeah not because I think any moment one of you are but because I genuinely want to understand this trend right I'm not trying to arrest people I'm not trying to sort of sort of be an instrument of the criminal justice system I'm trying to be you know sociologists or political scientist and understand this this actual trend the problem is if I sit down with each one of you in a private room and I say I promise I promise I promise I won't tell anybody right I'm still going to get a skewed distribution right make me some people are just gonna be like why would I risk telling you this is this private information and so what what sociologists can do is this this technique called randomized Response where I should about a coin you take a coin and you give it to each person before you survey them right and you've asked them to flip it twice somewhere that you cannot see so I would ask each one of you to flip a coin twice somewhere that I cannot see and then I would instruct you to if the first coin flip is a heads answer honestly but if the first coin flip is a tails answer yes or no based on the second coin flip okay so roughly half the time you'll be honest and the other half the time you'll be a you'll be giving me a perfect 50/50 coin flip and the cool thing is that what this is actually doing is taking whatever the true mean of the distribution is and averaging it with a 50/50 coin flip right so if say 55 percent of you answered yes that that you are a serial killer then I know that the true center of the distribution is actually 60% because it was 60% average with a 50/50 coin flip does that make sense however despite the fact that I can recover the center of the distribution right given enough samples each individual person has plausible deniability if you said yes it could have been because you actually are or it could have been because you just happen to flip a certain sequence of coin flips okay now this concept of adding noise to data to give plausible deniability is whether the secret weapon of differential privacy right and and the field itself is a set of mathematical proofs for trying to do this as efficiently as possible to give sort of the smallest amount of noise to get the most accurate results right with the best possible privacy protections right there is a meaningful sort of base trade-off that you you you you know you can escape there's kind of a Pareto trade-off right and we're trying to push that push that trade-off down but so the the the the field of research that is differential privacy is looking at how to add noise to data and and resulting queries to give plaza deniability to the entrance to the members of it of a database or a training dataset does that make sense now a few terms you should be familiar with so there's local and there's global differential privacy so local differential privacy adds noise to data before it's sent to the statistician so in this case when with the coin flip this was local difference or privacy it afford you the best amount of protection because you never actually reveal sort of in the clear your information to sup to someone okay and then there's global differential privacy which says okay we're to put everything in the database perform a query and then before the output of the query gets published we're gonna add a little bit of noise to the output of the query okay this tends to have a much better privacy trade-off but you have to trust the database owner to not compromise the results okay and we'll see there's some other things we can do there but with me so far this is a good good point for questions if you had any questions got it so the question is is this verifiable they get any of this this process would under privacy verifiable so that is a fantastic question and one that actually absolutely comes up in practice so first local difference or privacy the nice thing is everyone's doing it for themself right so in that sense if you're flipping your own coins and answering your own questions that's not your verification right you're kind of trusting yourself for global differential privacy stay tuned for the next tool and we'll come back to that all right so what does this look like in code so first we have a pointer to remote private data set we call dot git whoa we get big fat error right you just asked to sort of see the raw value of some private data point which you cannot do right instead pass and get epsilon to add the appropriate ment of noise so one thing I haven't mentioned yet differential privacy so I mentioned sensitivity right so sensitivity was related to the type of query the type of function that wanted to do and it's invariance to removing or replacing individual entries in the database so epsilon is a measure what we call our privacy budget all right and what our privacy budget is is saying okay what's the what's the amount of statistical uniqueness that I'm going to sort of limit what's the upper bound for the amount of systick --kw neatness that I'm going to allow to come out of this out of this database and actually I'm going to take one more size sidetrack here because I think it's really worth mentioning data anonymization anyone familiar with data anonymization come across this term before taking a document like redacting the the social security numbers and like all's kind of stuff by and large it does not work you don't remember anything else from this talk is very dangerous to do just data set anonymization okay and differential privacy in some respects is is the formal version of data automation we're instead of instead of just saying okay I'm just gonna redact out these pieces and then I'll be fine this is saying okay that we can do a lot better so for example Netflix prize Netflix machine-learning prize if you remember this a big million-dollar prize maybe some people in here competed in it so in this prize right Netflix published an anonymized data set right and that was movies and users right and they took all the movies and replaced them with numbers and it took all the users and replaced them with numbers and then we just had sparsely-populated movie ratings in this matrix right seemingly anonymous right there's no names of any kind but the problem is is that each row is statistically unique meaning it kind of is its own fingerprint and so two months after the data set with published some researchers at UT Austin I think it was I think it's UT Austin were able to go and scrape IMDB and basically create the same matrix and IMDB and then just compare the two and it turns out people that were in the movie rating we're in the movie rating and and and we're watching movies at similar times and similar similar patterns and similar tastes right and they will de anonymize this first dataset with high degree of accuracy happened again with there's a famous case of like medical records for like I think I'm I didn't bid a Massachusetts senator I think it was someone north-east being dean Onam eyes through very similar techniques so someone person goes and buys a anonymize medical they said over here that has you know birth date and zip code and this one does zip code and and gender and this one does zip code gender and whether or not you have cancer right and and when you get all these together you can start to sort of use the uniqueness and each one to relink it all back together i mean i this is so doable today to the extreme that i unfortunately no of companies whose business model is to buy anonymize datasets d anonymize them and sell market intelligence to insurance companies ooh right but it can be done okay and and the reason it can be done is that just because the data set that you are publishing and one that you are physically looking at doesn't seem like it has you know Social Security numbers stuff in it does that mean that there's enough unique statistical signal for it to be linked to something else and so when I say maximum out of epsilon epsilon is an upper bound on the statistical uniqueness that you're publishing in a data set right and so what what this tool represents is saying okay apply however much noise you need to given whatever computational graph led back to private data for this tensor right to ensure that you know to put an upper bound on the potential for link tax right now if you said epsilon0 okay then that's that's saying effectively like there's the I'm only going to allow patterns that have occurred at least twice okay so meaning meaning two different people had this pattern and thus it's not unique to either one yes so what happens if you perform the query twice so the random noise would be reran demised and sent again and you're absolutely absolutely correct so this epsilon this is how much I'm spending with this query so if I ran this three times I would spend epsilon of 0.3 so it makes sense so this is a point 1 query if I did this multiple times the absalons put some and so for any given data science project right I should I we're advocating is that you're given an epsilon budget that you're not allowed to exceed right no matter how many queries that you you could say now there's that there's another sort of subfield of difference or privacy that's looking at sort of single query approaches which is all around synthetic data sets so how can I perform sort of one query against the whole data set and create a synthetic data set that has certain invariances that are desirable right so I can do good statistics on it but then I can query this as many times as I want there basically you can't yeah anyway but we don't see it at now does that answer your question cool awesome so now you might think okay this is like a lossless cause like how can we be answering questions while protecting while while keeping cystal signal gone but like it's the difference between it's the difference between if I have a data set and I want to know what causes cancer right I could query data set and learn that smoking causes cancer without learning that individuals are are are not smokers does that make sense all right and the reason for that is is that I'm specifically looking for patterns that are occurring multiple times across different people and this actually happens to really closely mirror the type of generalization that we want in machine learning assistants anyways does that make sense like as machine learning petitioners we're actually not really interested in the one offs right I mean sometimes our models memorize things this this happens right but we're actually more interested in the things that are the things that are not specific to you I want I want the things that are gonna work you know that the heart treatments they're gonna work for everyone in this room not just I mean night you know obviously if you need a heart treatment I'd be happy that'd be cool for you to have one but like what we're T FLE interested in are things that generalize right which is why this is realistic and why with with continued effort on both tooling and and the theory side we can we can have a much better reality today cool so pros just review so first remote execution allows this allows data to remain the remote machine search and sampling we can feature engineer using toy data difference or privacy we have a formal rigorous privacy budgeting mechanism right now shoot how is the privacy budget set is it defined by the user or is it defined by the data set owner or someone else this is a really really interesting question actually so first it's definitely not set by the data scientist because that would be a bit of a conflict of interest and up at first you might say it should be the data owner okay so the hospital right it's trying to cover their butt right and make sure that their assets are protected both legally and and torchy right so they're they're trying to make money off this so there's there's there's sort of proper incentives there but the interesting thing and this gets back to your question is what happens if I have say a radiology skin in two different hospitals right and they both spend 1 epsilon worth of my privacy in each of these hospitals right that means that actually two epsilon if my private information is out there right and it just means that one person has to be clever enough to go to both places to get to join this is actually the exact same mechanism we were talking about a second ago when someone went from Netflix time TB right and so the true answer of who should be setting epsilon budgets although logistical II it's gonna be challenging we're talking about a little bit in part two of the talk but I'm going a little bit slow but okay is it should be us it should be people in it should be people around their own information right you should be setting your personal epsilon budget that makes sense that's an aspirational goal we've got a long way before we can get to that level of infrastructure around these kinds of things I'm gonna talk about that and we can definitely answer session as well but I think it theory in theory that's what we want okay the two cons we still a suit two weaknesses of this approach that we still have lack are someone asked this question he was you yeah yeah you asked the question so first the data is safe but the model is put at risk and what if we need to do a join actually actually yours is a third one which I should totally add to the slide so so first if I'm sending my computations I model into the hospital to learn how to be a better cancer classifier right my models put at risk it's kind of a bummer if like you know this is a ten million dollar healthcare model I'm just sending it to a thousand different hospitals to get learn to learn so that's potentially risky suck it what if I need to do a joint computation across multiple different data owners who don't trust each other right who sends whose data to whom right and thirdly as you pointed out how do I trust how these computations are actually happening the way that I am telling the remote machine that they should happen this brings me to my absolute favorite tool secure multi-party computation come across this before raise them high ok cool a little bit above average most machine learning people have not heard about this yet and I absolutely is this is the coolest this is the coolest thing I've learned about since learning about like AI machine learning this is there is a really really cool technique in cryptic computations you how about homework encryption you come across homework encryption okay a few more yeah this is related to that so first the kind of textbook definition is like this so if you went on Wikipedia you'd see security PC allows multiple people to combine their private inputs to compute a function without revealing their inputs to each other okay but in the context of machine learning the implication of this is multiple different individuals can share ownership of a number okay share ownership of a number show you what I mean so let's say I have the number five my happy smiling face and I split this into two shares a two and a three okay I've got two friends Mary Ann and Bobby and I give them these shares they are now the shareholders of this number okay now I'm gonna go away and this number is shared between them okay and this this gives us several desirable properties first its encrypted from the standpoint that neither Bob nor Mary Ann can tell what number is encrypted between them by looking at their own share by itself now I've for those of you who are familiar with kind of cryptographic math I'm hand waving over this a little bit this would typically be so in incre decryption would be adding the shares together modulus a large prime so these are typically look like sort of large pseudo-random numbers right but for the sake of making it sort of intuitive I've picked pseudo-random numbers that are convenient to the eyes so first these two values are encrypted and second we get shared governance meaning that we cannot decrypt these numbers or do anything with these numbers unless all of the shareholders agree okay but the truly extraordinary part is that while this number is encrypted between as individuals we can actually perform computation right so in this case let's say we wanted to multiply these shares times a encrypted number times two each person can multiply their share times two and now they have an encrypted number ten right and there's a whole variety of protocols allowing you to do different functions such as the functions needed for machine learning wild numbers are in this encrypted state okay and I'll give some more resources for you if you're interested in kind of learning more about this at the end as well now the big tiya models and data sets are just large collections of numbers which we can individually encrypt which we can individually share governance over now specifically to reference your question there's two configurations of screen PC active and passive security in the active security model you can tell if anyone does computation that you did not sort of independently authorize which is great so what does this look like in practice when you go back to the code so in this case we don't need just one worker it's not just one Hospital because we're looking to have shared governance shared ownership amongst multiple individuals so let's say we have Bob Alice and Te'o and encrypt provider which we won't go into now I can take a tensor instead of calling dot send and sending that tensor to someone else now I call dot share and that splits each value into multiple different shares and distributes those amongst the shareholders right so in this case Bob Allison tayo however in the frameworks that were working on you still get kind of the same PI torch like interface and all the cryptographic protocol happens under the hood and the idea here is to make it so that we can sort of do encrypted machine learning without you necessarily having to be a cryptographer right and vice versa cryptographers can improve the algorithms and machine then people can automatically inherit them all right so kind of classic sort of open source machine learning library making complex intelligence more accessible to people if that makes sense and what we can do on tensors we can also do in models so we can do encrypted training and encrypted prediction and we're going to get into what kind of awesome use cases this opens up in a bit and this is a nice set of features right in my opinion this is this is sort of the MVP of doing privacy preserving data science right the idea being that I could have remote access to a remote data set I can learn high-level latent patterns like like you know what causes cancer without learning whether individuals have cancer I can pull back just just that sort of high-level information with for mathematical guarantees over over you know what sort of the filter that's that's coming back through here right and I can work with datasets from multiple different data owners while making sure that each each individual data owners are protected now what's the catch okay so first is computational complexity right so encrypted computation secure NPC this this involves sending lots of information over over the network I think this is the state of the art for training or for deep learning prediction is that this is a 13 X slowdown over plain text which is inconvenient but not deadly right but you do have to understand that that assumes like it's like two AWS machines or like talking to each other you know they're relatively fast but we also haven't had any like hardware optimization to the extent that that you know Nvidia did a lot for deep learning like that there'll be you know probably like some sort of Cisco Player and it's similar for for doing kind of encrypt a or securing PC base deep learning right let's see so this brings back to kind of the fundamental question is it possible to answer questions using data we cannot see the theory is absolutely there I think that's that's something that I feel reasonably confident saying like like that sort of a theoretical frameworks that we have and actually the other thing that's really worth mentioning here is that these come from totally different fields which is why they kind of haven't been necessarily combined that much yet I'll get I'll get more into that in a second but it's my hope that that by sort of by considering what these tools can do that'll open up your eyes to the potential that in general we can have this new ability to answer questions using information that we don't actually own ourselves because from a sociological standpoint that's net new for like us as a species that makes sense if ever previously we had to have we had to have like a trusted third party who would then take all the information in themselves and make some sort of neutral decision right so we'll come to that in a second and so one of the big sort of long-term goals of our community is to make infrastructure for this secure enough and robust enough and of course in like a free Apache to open-source license kind of way that you know information on the world's most important problems will be this accessible right and we can spend sort of less time working on tasks like that and more time tasks like this so this is gonna be kind of the breaking point between sort of part 1 and part 2 part 2 will be a bit shorter but if you're interested in sort of diving deeper on the technicals of this here's a six or seven hour course that I taught just on these concepts from the tools it's free on your Nazi feel free to check it out so the question was he's asking about how I that a model can be encrypted during training is that same as homework encryption that's somewhat something else so a couple years ago there was a big burst in literature around training on encrypted data where you would homomorphic encryption data set and it turned out that some of the statistical regularities homework encryption allowed you to actually train on that data set without without decrypting it so this is similar to that except the one downside to that is that in order to use that model in the future you have to still be able to encrypt data with the same key which often is sort of constraining in practice and also there's a pretty big hit to privacy because your your training on data that inherently has a lot of noise added to it what I'm advocating for here is instead we actually encrypt both the model and the data set during training but inside the encryption inside the box right it's actually performing the same computations that it would be doing in plaintext so you don't get any degradation in accuracy and you don't get tied to one particular public/private key pair yeah yeah so specifically so the question was kind of comment on federated learning specifically Google's implementation so I think Google's implementation is is great so obviously the the fact that they've shown that this can be done hundreds of millions of users is incredibly powerful I mean even inventing the term and creating momentum in that direction I think that there's one thing that's worth mentioning is that there are two forms of federated learning one is sort of the one where your model is a federated learning sorry who got to talk about what that is okay yes I'll do that quickly so a federated learning is basically the first thing I talked about so remote execution so if everyone has a smartphone when you plug your phone in at night if you've got you know Android or iOS you plug your own up phone at night and touch the Wi-Fi you know when you text in it recommends the next word next prediction that model is trained using federated learning meaning that it learns on your device to do that better and then that model gets uploaded to the cloud as opposed to uploading all of your tweets to the cloud and training one global model does that make sense so so if all your phone a night model comes down trains locally goes like it's federated right that's that's that's basically federal earning is a nutshell and and it was pioneered by the cork team at Google and and they're there do you really fantastic work they've they've paid down a lot of the technical debt a lot of the the risk or technical risk around it and they publish really great papers outlining sort of how they do it which is fantastic what I outlined here is actually a slightly different style of federate learning because there there's federated learning with like a fixed data set and a fixed model and lots of users where the data is very ephemeral like phones are constantly logging in and logging off you know you're you're you're plugging your phone in an eye and then you're taking it out right this is sort of the the one style of federated learning that's it's really useful for like product development right so it's useful for like if you want to do a smartphone app that has a piece of intelligence in it but train that intelligence is going to be prohibitively difficult for you to get access to the data for or you want to just have a value prop of protecting privacy right that's what federated learning that South Area learning is good for what I've outlined here is a bit more exploratory federated learning where it's saying okay instead of instead of the model being hosted in the cloud and data owners showing up and making it a bit smarter every once in a while now the data is going to be hosted at a variety of different private clouds right and data scientists are gonna show up and say mmm I want to do something with that with diabetes today mmm I will do something with with studying dementia today something like that right this is much more difficult because the attack vectors for this are much larger right I'm trying to be able to answer arbitrary questions about arbitrary data sets in a protected environment right so I think yeah that's that's kind of my general thoughts does federated learning leaking information so federated learning by itself is not a secure protocol right to the extent that and that's why I sort of this ensemble of techniques that I've so the question was does federated learning leak information so it is perfectly possible for a federated learning model to simply memorize data set and then spit that back out later you have to combine it with something like differential privacy in order to be able to prevent that from happening does that make sense so just because the training is happening on a device does not mean it's not memorizing my data does that do that make sense okay so now I want to zoom out and go a little less from the kind of a data science practitioner perspective and now it take more the perspective of like a economist or scientist or someone looking kind of globally at like okay what if this becomes mature what happens alright and this is where I gets really exciting anyone entrepreneurial anyone everyone I know okay cool well this is this is the this is the part for you so the big difference is this ability to answer questions using data you can't see because as it turns out most people spend a great deal of their life just answering questions and a lot of it is involving sort of personal data I mean whether it's my new things like you know where's my water where are my keys or you know what movie should i watch tonight or or you know what kind of diet should I have to be able to sleep well right I mean a wide variety of different questions right and and we're limited and are answering ability to the information that we have right so this ability to answer question using data we don't have sociological II I think is quite quite important and there's four different areas that I want to highlight as like big groups of use cases for this kind of technology to help kind of inspire you to see where this infrastructure can go and actually before I before I jump into that has anyone been to Edinburgh Umbra cool I just see tour like the castle and stuff like that so my wife and I my wife we wouldn't say Edinburgh for the first time six months ago September September and we did the underground was it the we did a ghost to her yeah yeah we did the ghost to her and it was really cool it was something that took away from it there was this point we were standing we just walked out of the tunnels and she was pointing up some of the architecture and then she started talking about basically the cobblestone streets and why the cobblestone streets were there cobblestone streets one of the main purposes of them was to sort of lift you out of the muck and the reason there was muck was there is that they didn't have any internal plumbing and so the sewage just poured out into the street right if you live in a big city and this was the norm everywhere right and actually I think she even sort of implied to like the invention or popularization of the umbrella had less to do with actual rain a bit more with you with buckets of stuff coming down from on high which is it's a whole different world like when you think about what that is but the reason that I bring this up is that you know however many hundred years ago people were were walking through you know like sludge sewage was just everywhere right it was all over the place and people were walking through it everywhere they go and they were wondering why they got sick right and in many cases and it wasn't because they wanted it to be that way it's just because it was a natural consequence of the technology they had at the time right this is not malice this is not anyone being good or bad or or evil or whatever it's just it's just the way things were and I think that there's a strong analogy to be made with with kind of how our data is handled as a society at the moment right we've just sort of walked into a society we've had new inventions come up and new things that are practical new uses for it and now everywhere we go we're constantly spreading and spewing our data all over the place right I mean every every camera that sees me walking down the street you know goodness there's a there's a company that takes a whole of the earth by satellite every day like how the hell am I supposed to do anything without without you know everyone follow me around all the time right and I imagine that whoever it was I'm not a historian so I don't really know but whoever it was that said what if what if we ran plumbing from every single apartment Business School maybe even some public toilets underground under our city all to one location and then processed it used chemical treatments and then turn that into usable drinking water like how laughable with that event would have been just the most massive logistical infrastructure problem ever to take a working city dig up the whole thing to take already already constructed buildings and run pipes through all of them I mean so so Oxford gosh I there's a building there that's so old they don't have showers because they didn't want to run the plumbing for the head you have to ladle water over yourself it's in the Merton College it's quite quite famous right I mean the infrastructure anyway the infrastructure challenge is it just must have seen absolutely massive and so as I'm about to walk through kind of like four broad areas where things could be different theoretically based on this technology and I think it's probably going to hit you like whoa that's a lot of code or like whoa that's that's a lot of change but but I think that the need is sufficiently great I think that that I mean if you view our lives it's just one long process of answering important questions whether it's where we're going to get food or what causes cancer like making sure that the right people can answer questions without without you know data just getting spewed everywhere so that the wrong people can answer their questions right is important and yeah anyway so I know this is gonna sound like there's a certain ridiculousness to maybe what some of this will be but I hope that that you at least see that that theoretically like that the basic blocks are there and and that really what stands between us and a world that's fundamentally different is is adoption maturing of the technology and good engineering because I think you know once they know that Sir Thomas Crapper invented the toilet right I do remember that one at that point that the basics were there right and and what stood between them was was implementation adoption in engineering right and I think that's that's where we are and the best part is we have you know companies like Google that have already already paved the way with some very very large rollouts of of the early piece of this technology all right cool so what about what are the big categories when I've already talked about open data for science ok so this one is a really big deal and the reason it's a really big deal is mostly because everyone gets excited about making AI progress right everyone gets super excited about superhuman ability in X Y or Z when I started my PhD at Oxford I work for my professors name is Phil Blount some the first thing he told me when I sat my butt down on his office on my first day is this dude he said Andrew everyone's going to work on models but if you look historically the biggest jumps in progress have happened when we had new big datasets or the ability to process new big datasets and just to give a few anecdotes imagenet right imagenet GPUs allowing us to process large datasets even even things like alphago this is synthetically generated infinite datasets or or or if you don't know did you guys anyone watch the the alpha star livestream on YouTube I talked about how it had trained on like 200 years of like of StarCraft right well if you look at Watson the playing playing jeopardy right this was on the heels of a new large structured data set based on Wikipedia or if you look at Garry Kasparov and IBM's deep blue this was on the heels of the largest open data set of chess matches haven't been published online right there's this there's this echo we're like big new data set big big new breakthrough big new data set big new breakthrough right and what we're talking about here is is potentially several orders of magnitude more data relatively quickly and the reason for that is that that we're not I'm not saying we're gonna invent a new machine and that if she's gonna collect this and then it's gonna go online I'm saying there's thousands and thousands of enterprises millions of smartphones there's there's an hundreds of governments that are all already have this data sitting inside of data warehouses right largely untapped for two reasons one legal risk and two commercial viability right if I give you a data set all of a sudden I just doubled the supply right what does that do to my billing ability and there's the legal risk that you might do something bad with it that comes back to hurt me with this category I know it's like just one phrase but but this is like imagenet but for every data task that's already been established right this is us like I mean I were working with a professor at Oxford a psychology department who wants to study dementia right he is the problem with dementia is is every hospital has like five cases right it's not like a very centralized disease it's not like all the all the cancer patients go to you know one big Center and like where all the technology it's like dementia it's it's it's sprinkled everywhere and so the big thing that's blocking him as a dementia researcher is access to data and so he's investing in private data science platforms and I didn't persuade him to I found him after he was he was already looking to do that but but pick pick any challenge that where data is already being collected and this can unlock not larger amounts of data that exists but larger amounts of data that can be can be used together that makes sense this is like a thousand startups right here whereas instead of going out and trying to buy as many days as you can which is a really hard and really expensive task talk to anyone who's in Silicon Valley right now trying to do a data science startup right instead you go to each individual person that has a data set and you say hey let me create a gateway between you and the rest of the world is going to keep your data safe and allow people to leverage it right that's like repeatable business model take a use case right BB the radiology Network gatekeeper right okay so enough on that one but like does it make sense how like on a huge variety of tasks just the ability to have a day of box silo you can do data science against it's going to increase the accuracy of a huge variety of models really really really quickly cool all right second one single yes accountability this one's a little bit tricky get to the airport and you get bag checked right everyone's familiar with this process I see what happens someone's sitting at a monitor and they see all the objects in your back so that occasionally they can spot objects that are dangerous or illicit right there's a lot of extra information leakage over to the fact that they have do they have to sit and look at thousands of of all the objects you know basically searching every single person's bag totally and completely just so that occasionally they can find that one answer that if you have the question they actually want to answer is is there anything dangerous in this bag but in order to answer it they have to basically acquire access to the whole back right so let's let's let's think about this same approach of answering questions using data we can't see the best example of this in the analog world is a sniffing dog smear with like sniffing dogs so give your bag a whiff at the airport right these are actually a really privacy preserving thing because dogs don't speak English or any other language and so that the benefit is dog cuz buy no everything's fine moves on the dog has the ability to only reveal one bit of information without you having to search every single back okay that is what I mean when I say a single-use accountability system it means I am looking at some data stream because I'm holding someone accountable right and we want to make it so that I can only answer the question that I claim to be looking into so if this is a video feed right for example right instead of getting access to the raw video feed and and under you know the millions of bits of information every single person in the frame of you walking around doing whatever which I could use for you know even if I'm a good person I technically could use for for other purposes but instead build a system where I build say a machine learning classifier right that is an auditable piece of technology that looks for whatever I'm supposed to be looking for right and I only see frames you know I only open up bags that actually have to okay this does two things one it makes all of our Callaway systems more privacy-preserving which is great mitigates any potential dual or multi use right and two it means that some things that were simply too off limits for us to properly hold people accountable might be possible right well one of the things that was really challenging so we used to do email surveillance does the reasoning right and and it was basically help investment banks find insider traders right because they want to help enforce the loss they you know get fined billion dollar fines if anyone caused an infraction but one of the things that was really difficult about developing these kinds of systems was that it's so sensitive right we're talking about you know hundreds of millions of emails at some massive investment bank is so much private information in there that say none of our data scientists barely any of them we're able to actually work with the data and try to make it better right and and and this this yeah this makes it really really difficult anyway cool so if on that third one and this is the one I think it's just incredibly exciting and encrypted services what's up everyone's me with what's up telegram any of these these are messaging apps right where a message is encrypted on your phone and it's sent directly to someone else's phone and only that person's phone can decrypt it right which means that someone can provide a service you know messaging without the service provider seeing any of the information that they're actually providing the service over right very powerful idea what if the intuition here is that with a combination of machine learning encrypted computation and differential privacy that we could do the same thing for entire services so imagine going to the doctor okay so you go to the doctor this is really a computation between two different data sets on the one hand you have data set that the doctor has which is their you know medical background their knowledge of different procedures and diseases and tests and all this kind of stuff and then you have your data set which is your symptoms your your medical history you know your recent things that you've eaten your your genes your genetic predisposition and your heritage those kinds of things right and you're bringing these two data sets together to compute a function and that function is what what what treatment should you have if any okay and the idea here is that so there's this new new field called structured transparency probably mention I'm not sure maybe sure you can call it a new field yet because it's not in the literature but it's been bouncing around a few different circles and the and it's it's uh x/y never give a chalk sorry and then this is Z okay so this to you for people providing their data together computing a function and an output so so difference for privacy protects the output encrypted computation so like MPC which we talked about earlier protects the input right so allows them to compute f of X of Y right without revealing their inputs remember this so basically encrypt Y encrypt X compute the function while it's encrypted do we remember we remember this right and so there's there's three processes here right there's input privacy which is the NPC there's logic and then there's output privacy and this is what you need to be able to do end-to-end encrypted services okay so imagine imagine so there are there are machine learning models that can now do skin cancer prediction all right so I can take a picture of my of my arm and say that their machine a machine learning model and they'll predict whether or not I have melanoma on my arm right ok so in this case machine learning model perhaps owned by a hospital or startup image of my arm okay encrypt both the logic is done by the machine learning model the prediction if it's gonna be publish to the output to the rest of the world use difference or privacy but in this case the prediction can come back to me and only I see the decrypted result okay the implication being that the doctor role facilitated by machine learning can classify whether or not I have cancer can provide this service without anyone seeing my medical information I can go to the doctor and get a prognosis without ever revealing my medical records to anyone including the doctor right does that make sense and if you believe if you believe that sort of the services that are repeatable that we do for millions and millions of people right can create a training data set that we can then train a classifier to do then we should be able to upgrade it to be end and encrypted does that make sense so again it's kind of it's kind of big it assumes that that AI is smart enough to do it there's lots of questions around quality and my quality assurance and all those kinds of things that have to be addressed there's very likely to be different institutions that we need but I hope that at least these three sort of big category this isn't by no means comprehensive but helped at least these three big categories will be sort of sufficient for helping sort of lay the groundwork for how sort of each person could be empowered with sole control over the only copies of their information while still receiving the same goods and services they've become accustomed to cool thanks questions let's do it first please give Andrew a big hand [Music] and it was fascinating really fascinating amazing amazing set of ideas and hopefully this can really get rid of the sewage off of data under on the this vision of end-to-end encrypted services if I understand correctly the algorithm would also run on two or more services and the skin image would go to them ooh and then you would get the diagnosis but the diagnosis itself is not private though if that's the output of that it is is being revealed to the service provider so we could it could optionally be revealed to the service spider so in this case oh yeah something I didn't I didn't say first securing PC from cryptic computation except for some with with some exceptions but for securing PC when you perform computation between two encrypted numbers the result is encrypted between the same shareholders if that makes sense meaning that by default Z is still encrypted with the same keys as x and y and then it's up to the key holders to decide who they want to decrypt it for so they could decrypt it for the general public in which case they should apply differential privacy they could decrypt it for the the input owner in which case the input owners not gonna hurt anybody else by him knowing whether or not whether he has a certain diagnosis or it could be decrypted for the model owner perhaps allow them to do more training or some other arbitrary use case right but so it can be but not not not as a strict requirement just to be sure I mean if if Z is being computed by say two parties to send you back to back to Y in this case the machine knows what Z is so in that sense even if you encrypt C with the with the key of Y there's no way to protect the output itself hmm I haven't described this correctly so when we perform the encrypting computation we split this into shares so we'll say you know y y1 and y2 right y2 goes up here and then this populates what actually happens is this creates Z 1 and Z 2 at the end right which is still which is still owned by you know person Y so we'll say this is Alice and this is Bob right so we have Bob share and Alice's share what gets populated is is shares of Z so if Alice if Bob sends his share of Z down to Alice only Alice can decrypt the result so does that make that make more sense okay even the unser is it's even the answer is protected yes and you would only need to use differential privacy in the case you're planning to decrypt the result for some some unknown audience to be able to see some models have biased based on real data biases and society tries to make unbiased models like on gender race and so on how does it work with privacy especially when everything is encrypted so how can you unbiased models when you do not see biases in a data and so on that's a great question so the the first the first gimme for that is that people don't ever really D buy us a model by physically reading the weights right so so the fact that the weights are encrypted doesn't help it hurt you so really what it's about is just making sure that you provision enough of your privacy budget to allow you to do the introspection that you need to be able to measure and adjust for for bias so I think that's that's a does that is that sufficient cool awesome great question then how far away do you think we are from organizations like the FDA requiring differential privacy to be used in regulating medical so I think the best answer I can give to that so if one I don't know and even laws in the UK regarding privacy GDP are not prescriptive about things like like differential privacy but I think the best and most relevant data point I have for you on that is that the US census this year is going to be protecting the census data 2020 census data using differential privacy and like some of the leading work on actually applying difference or privacy in the world is going on at the US Census and I'm sure they'd be interested in more helpers if anyone was interested in joining them so I guess her question was kind of my one of my questions but it was more just like how much buy and are you getting in terms of adoption for open mind or any of like do you have like any hospitals that are like participating or yeah so actually there's a few things I probably should have mentioned so the open mine is about two and a half years old in the very beginning we had very little buy-in because it was just so early it was kind of like who cares about privacy no one's ever gonna sort of really really care about that post GDP our total total change right everyone's scrambling to protect the data but the truth is it's not just privacy it's also commercial usability right now if you're selling data every time you sell it you lower the price because you increase the supply and you increase the number of people that are that are also selling it so I think that there's there's people also waking up to kind of the commercial commercial reasons for protecting their own data sets and protecting the kind of the unique statistical signal but that they have it's also worth mentioning so the pie charts team recently sponsored $250,000 in open source grants to fund people to work on our pie safe library which is really good and we're hoping to announce sort of more grants of similar size later in the year so if you guys like working on open source code and and like to get paid to do so so that so that accident that's sort of a big vote and buy and as far as our community is concerned so this year is when I hope to see kind of the first pilots rolling out I am there are some that are sort of in the works but I I can't they're not public yet but okay yeah so I think basically this is the year for like pilots okay I think it's best about as far as we are and then I have another question that's kind on the opposite end of the spectrum that's a little more technical weeds go um so when you do the separate when you're doing the encryption where you separate everything into each of the different owners how does that work for nonlinear functions because I would just you need it's a lot of linearity to like add it back in for it to total maintain the totally so the nonlinear functions are the most complicated the most performance intensive until they said they they do get the biggest performance hit when you have to do them the for deep learning specifically there's kind of two trends so one line of research is around using polynomial approximations and then the other line is around doing sort of discrete comparison functions so which is good for a lose and it's good for lopping off the ends of your polynomials so that your unstable tails can be flat and i would say that's about that and then like the science of kind of like trying to relax your security assumptions strategically here and there to get more performance is about where we're at but as far as the the one thing is this worth mentioning though is that there's there are multi cut of what I described was securing PC sort of on you know integers and and and fixed fixed precision numbers you can also do it on on sort of binary but in that sense it's you get a huge performance every doing of it binary but you also get people to do things sort more classically with computing encrypted computation is sort of like doing computing in the 70s like you get you get a lot of the same kind of kind of constraints thank you very much for your talk Andrew yeah I'm wondering about your objective to ultimately allow every individual to assign a privacy budget you think you also mention you mentioned that it would take a lot of work to provide the infrastructure for that to be possible so do you have an idea for what kind of infrastructure is necessary and also when people are reluctant you know even perhaps lazy and you know they don't really care and they don't want their data to be protected yeah you know I guess it takes some training but yet what are your thoughts building of that infrastructure I think it's gonna come in waves it's the kind of thing we're like people don't usually invest money and time and resources into things that aren't like a straight shot to value so I think this probably gonna be multiple individual this Griet jumps the first one is going to be just enterprise adoption enterprises are the ones that already have all the data sort of the ones you're most natural to sort of start adopting privacy-preserving technologies and I think that that adoption is gonna be driven primarily by commercial reasons for commercial reasons meaning my data is inherently more valuable if I can keep it scarce while allowing people to answer questions with it if that makes sense so it's more profitable for me to not send copies of my data to people if I can actually have them bring their question answering mechanisms to me and just get their questions answered that's that make sense that's not a privacy narrative but I think that that narrative is going to mature privacy technology quite quickly post sort of enterprise adoption I think that that's when that's and encrypted services are still really hard at this point and the reason for that is that they require lots of compute and lots of network overhead which means that you probably want to have something in in the cloud right so some sort of machine that you can hold and control in the cloud or have the internet get a lot faster so there's that but there's this question of like how how do we actually get to a world where for each individual person sort of knows or as notional control over their their own personal privacy budget right and let's just say you had perfect enterprise adoption right and everyone's tracking their stuff for difference for privacy the piece that you're actually missing here is is just some sort of communication between all the different enterprises that are joining up and making it you just just make a counting mechanism all right it's it's it's it's a lot like so like the IRS right it's it's like it's it's just someone to be there to make sure that that you're not double spending in different different places right that you're your epsilon budget it's over here versus over here versus over here is all come from the same place it's not totally clear who this actor would be right maybe maybe there's an app that just does it for you maybe there has to be an institution around it maybe maybe it won't happen at all maybe it'll just be decentralized but you know whatever another option is that they'll actually be data banks price has been some literature in the last couple years around saying okay maybe maybe institutions that they currently handle you know your money might also be the bank where all of your information lives right and and that becomes the gateway to to your data or something like that so there's there's different things that are exact I would obviously make the accounting much easier and also that would give you kind of that cloud-to-cloud sort of performance increase so so it's I think it's clear we wouldn't go to data banks or these kinds of centralized accounting registries directly because you kind of have to have the initial adoption first but if I had to guess it's something it's something like that right we won't see that for awhile and it's not even clear what what that would look like but but I think it is possible we just have to get through sort of non-trivial adoption first yeah so it's kind of hazy but it's picking the future so I guess that's how that goes are you wondering I was just wondering if you can comment briefly on what you think is the biggest mistake being made with respect to recommendation systems transparency and what if you can comment briefly on what you think might be the most or the best solution so all of this is a mistake I would say the biggest opportunity for recommendation systems is that they have the potential to be more holistic so for example if you recommended a movie to me based on whether or not it's most likely didn't keep me engaged but I keep me watching movies it's not really a holistic recommendation it's nice saying hey you should do this because it's gonna make your life more fulfilling more satisfied whatever it's just going to glue me to my television more right so I think the best the biggest opportunity and particularly with privacy-preserving machine learning is that if a recommender system could have the ability to access private data without actually seeing it right and answer the question you know how do I give the best recommendation so that this person gets a good night's sleep or has more meaningful friendships or whatever like these of these these attributes that are actually particularly sensitive but there are things that we actually want to optimize for that we could have vastly more beneficial recommendation systems than we do now just by virtue of having better infrastructure for dealing with private data so I think actually what the as far as like the biggest limitation of our major systems right now it's just that they don't have access to enough information to have good targets that doesn't make sense we would like them to have better targets but in order to do that they have ax to information about those targets and I think that's what sort of privacy-preserving technologists could bring to bear on recommendation systems yeah thanks yeah great question by the way one more time please give Van gerbig an thank you so much thank you you