Real World Crypto conference 2020: session 13

all right we'll get it started the first stalk of the session is gonna be given by Felix from tech Technical University of Munich and we'll just go ahead all right thanks okay hello everyone so this certainly ended up my Twitter equate recently and it nicely describes the problem I want to address in this talk so it says I'm a programmer looking for a solution on Stack Overflow to paste into my project so as you can see it received a lot of love from the community lots of likes and lots of retweets and indeed Stack Overflow is the most popular question and answer web site for programmers it relies on community moderation to bubble up the best answer and weed out on my advice to any programming question so most of the time answers come as a code snippet and that makes it incredibly easy to just copy that code straight into your software and think no more of it apparently as the survey indicates this is common behavior and part of most developers workflows know it nowadays so is this speaks for the high usability and utility of Stack Overflow it unfortunately comes with a major risk for application security so and that's when it comes to security related questions and usability issues around cryptographic api's so this stack overflow question for example shows one of the biggest issues in android how can you accept a certificate during TLS handshake that is not part of the default android trust store so this question showed showed up quite a lot and we observed that millions of developers have looked it up so it seems that it's a super important use case for Android developers but the crypto API just didn't support it and popular answer on Stack Overflow for this question were unsaved workarounds where the crypto API was simply overridden they contained this null verifier here or some kind of variation of it that renders the TLS handshake vulnerable to man-in-the-middle attacks so here certificate verification is simply turned off however it technically solved the initial problem as it now accepts any certificate and developers seem to be super happy with it they got rid of their pesky certificate error and that's probably why they happily uploaded those answers until they became the most popular and accepted answer on Stack Overflow so in our Auckland paper in 2017 we've shown that those kind of code snippets that were insecure due to crypto misuse were indeed reused in almost two hundred thousand Android applications available on Google Play and those included high profile apps with an install base of over five billion uses and apps from security sensitive categories like business finance and health and social media another paper from one of our co-authors demonstrated how to attack apps and steal credentials credit card numbers and other private data based on these insecure code snippets from Stack Overflow in our Isis ePaper last year we've shown that this that stack overflows content indicators such as the community given score and the view counts for of given answers all point towards the wrong direction security works and therefore inadvertently promoted crypto misuse so usability issues with crypto API Slee to vulnerable code which additionally gets promoted and distributed by an Alexa top-50 website almost all developers used to get help so there's to tackle this huge problem different forms of security advice have been improved tested and compared with Stack Overflow for instance books informative documentation static code analysis tools and simplified cryptographic api's that were specifically designed keeping usability in mind so even though all of these approaches help in improving code security developers really struggled with getting a running code out of it they were less productive under given time constraints then developers that were allowed to use deck overflow a very surprising and disappointing example were the simplified cryptographic api's as they performed the worst in terms of productivity some developers even had to look up the source code of the API to figure out what it actually does and that's the complete opposite you want to achieve with an interface they were oversimplified and therefore only supported a very small range of use cases so the most important thing for us to learn from the studies was that whenever developers encountered a usability issue with one of these approaches they turned to the web and went for coach helping on Stack Overflow once again so they went back to their default behavior so when that goes in line with a famous quote of Richard Taylor the founder of the net Siri which is a concept from behavioral science and economics so when it says first never underestimate the power of inertia so again as we've seen when a whenever developers encounter usability issues with critical api's they ask Stack Overflow for help so when the quote continues with second that power can be harnessed so don't even try to change the default behavior as it's too powerful but rather try to harness it in a way that it improves the outcome so when that inspired us for for our main idea so let's try to harness code shopping on Stack Overflow to help developers get cryptography right okay but well how to do that so one of our most important findings to answer this question was that on Stack Overflow similar and secure code examples are available for almost all of the insecure code snippets so for any insecure code example there is a pretty high chance to get an alternative code example that practically does the same thing but in a secure way so in the end we see getting cryptography right as a decision-making problem and that's where the nudge theory comes into play so the basic idea is to nudge people towards better decisions without restricting their options or requiring them to change their incentives so you rather changed the choice architecture to steer people into a particular direction here on the in the example on the right people have two options for getting up stairs however using the stairs is the better option in terms of health but people prefer to use the escalator but with the new choice architecture where the stairs now look like a piano keyboard that makes sounds when you walk up stairs people tend to favor this option so and that's the the key aspect of the nudge theory it does not try to restrict or require to change incentives it reason redesigns the choice architecture in a way that identified behavior leads to better outcomes so our goal was to design a new choice architecture on stackoverflow and it not just people towards reusing code examples that provide secure and strong cryptography and it must not interfere with the usability and the utility of Stack Overflow such that developers can keep their high productivity level using the website and are not drawn away from it so but to be able to do that we first had to find these better alternatives on Stack Overflow so we had to solve free technical problems we needed to be able to predict the similarity of crypto API usage patterns they use cases and their security of course so we combined supervised and unsupervised deep learning to learn these things directly from code available on Stack Overflow so in the first step we learn how to predict the similarity of crypto API usage patterns everybody we had to consider a problem that's very specific to Stack Overflow and we're deeper learning helped so code examples on Stack Overflow are oftentimes incomplete and erroneous programs that means their representation the code graphs used to determine similarity may be unsound so you may end up with different code graphs for the same pattern for since if one comes from a complete program and the other one does not deep learning however it doesn't really mind it learns its own representation that's optimized for the problem it tries to solve so the network tries it's best to determine those features that allow it to predict similarity even though inputs are unsound so is the first step we learned a new representation for crypto API usage patterns and we did that by the embedding their code graphs into a vector space using structure to Veck so in these embedding so large such that similar patterns are closer together and this new patterns are more for away from each other in the embedding space so this way we can simply use a distance function to determine whether patterns are similar or not so this is the architecture of our embedding Network it's a Siamese architecture that uses two networks to generate embeddings either for two similar or dissimilar patterns so in during training we calculate the distance of both embeddings generated by the network and if they are too close or too far away from each other we back propagate the loss to update and improve the network in generating new emetics so in the second step we wanted to predict to learn how to predict the use case of a pattern for instance initializing a cipher or verifying a certificate so since we already trained a model for pattern similarity we consider very beneficial knowledge for predicting use cases we transfer this knowledge from the similarity domain into the use case domain by applying transfer learning and in a lot of step lost the most important step we trained to the security model and it predicts whether your cipher initialization or a certificate verification is secure or not and here we basically do the same thing again we applied transfer learning to reuse the similarity information encoded in the embeddings to train a new model for predicting security so what we did was we just edit another hidden layer highlighted in red on top of the embedding network highlighted in blue so with a blue layer why the blue layer already encodes the similarity information the red layer will learn the use case or the security information of patterns and based on that information the classification layer on the right will then be able to predict what use case it is and whether it's insecure or not so we apply different techniques to train the classification network with one technique called transfer of learning we input the pattern graph into the fixed pre trained and Benning network then we only update the weights of the use case or a security layer based on the classification loss with another technique called warm starting we also updated the weights of the pre trained and Benning network based on the classification loss so this way we update the pattern embedding as well in a way that it helps deciding whether a pattern is insecure or not okay so since code similarity does not necessarily need to be not learned from code that applies crypto we were able to compile an arbitrarily large dataset so for instance theoretically all public Java repositories and on github however the dataset with a cryptic code snippets we obtained from Stack Overflow was relatively small so in warm starting and transfer learning helped us and in tackling this challenge transferring knowledge we obtained from large datasets helped us in learning from small datasets more effectively so here on the left side you can now see the results of the similarity model so it shows all crypto API patterns we extracted from Stack Overflow highlighted by the use case so each point relates to a pattern so for several use cases you can already see that the similarity model creates dense clusters for instance the cypher cluster in blue or the TLS cluster in orange but it creates fast clusters for some of the other use cases for instance for key generation or IVs as you can see on the right the use case model is able to correct this it moves patterns closer together that belong to the same use case but that are not necessarily very similar here in the middle of this slide you can see the results of the similarity model again so the distance still represents similarity but the color now indicates security so rat is insecure and blue is secure so in this cipher cluster down here nicely indicates that our main idea of matching people away from insecurity secure is actually technically feasible so the cipher cluster here has a security boundary that means that patterns that are close to this boundary actually provide very useful alternatives they do the same thing but one is secure and the other one is not so in this shows a cherry-picked example of it so on the left side you see a warning for an insecure pattern as we show it on stackoverflow at the bottom you can see the list of recommendations which is ordered by similarity in use cases when you click on the first link you will enter on on this secure stackoverflow post shown on the right and as you can see it is basically the same code it only differs in the statement that renard the whole code snippet insecure before however developers might ignore basically everything I just showed and copy the insecure code anyway whenever we detect an insecure copy attempt on Stack Overflow we trigger a reminder notch and that shows the warning and recommendations again in order to make the user to pay attention all right so now that we have had everything together we wanted a test or a system design within a developer study and we had two treatments the nuts in the control group and both had to solve two programming tasks symmetric encryption and certificate verification we had two metrics functional correctness which allowed us to measure the productivity of developers and security of course which told us whether a secure solution was secure a nod so once again functional correctness was very important for us as their notches must not interfere with a great user experience of Stack Overflow so developers should be able to happily continue to copy and paste stuff and stay as productive as they have been with original mistake overflow and yes our nudges did not have a significant effect on functional correctness so both treatment snatched and control achieved a very high level of functional correct solutions within the given time constraints that means it doesn't matter whether you were not or not stack overflow remains it remains a very effective and efficient in solving programming tasks so if we now get secure solutions on top we exactly achieved what the nudge theory had promised us and indeed the nut treatment achieved significantly more secure solutions than the control and interestingly being a professional or security knowledge didn't have any effect on security so solely the nudges made a difference here so to wrap up the talk I'd like to show one of the most surprising results of our studies and that was that it helped technically nala verify us quite well so a short reminder we found out that 91% of apps with code from stack overflow contained in Albury file so certificate verification was basically turned off based on those code examples from Stack Overflow one of our co-authors was able to attack them only 0.2% of apps that reused code from Stack Overflow got certificate verification right and in another study where participants were only allowed to use simplified cryptographic api's therein none of them got it right however much participants achieved 77% secure solutions while the control group was again quite behind with 67% in secure solutions so that's cool because this was the problem with the highest risk for application security and it was also the use case on Stack Overflow that had the fuse secure code examples we found over a thousand examples of null verifies and only 50 to 60 examples that provided security best practices however if you implement the right choice architecture on stackoverflow those few example is seemed to be already enough all right what's so what's next we recently applied for the academic partnership program with stackoverflow and that would allow us to further test and improve our approach within a larger field study with realistic overflow and realistic overflow uses of course we'd be happy to test our approach with any company or institution so feel free to contact us so we'd love to see Stack Overflow considering some of our ideas since almost all developers use this website we should make sure that they stay on a safe path we believe that it could have a huge positive effect on how cryptography is used in the real world all right thank you very much and on now I'm happy to take questions [Applause] hi do you have any plans on trying to integrate this into text editors so currently not but sure why not given that we have some time could you go into a little more detail about how these nudges actually appear on the stack overflow page yeah sure sorry okay so here so this is one of the notches so here we have the security warning so the text and the icons is basically inspired by the text and icons used in Chrome for security warnings so yeah we show this warning we show also annotations like below the statement that is that that causes the snippet to be insecure and yeah below the warning we have the recommendation so that list displays different Stack Overflow posts and it's ordered by the similarity of the code to the code snippet you see above and a warning so that's the the first the first metric we use to order it and the second one is the use case so if the similarity is not enough we usually we at least show something that applies the same use case okay and yeah and then you can just click on one of those and then we show positive security indicators as shown on the right that indicate that we didn't find any anything that causes a problem and we had another nudge called the default match where we basically reordered the search results on the webpage so if you search for something we really ordered the results based on security so that you can that you get the the posts that are secured first right hi how do you make sure that the results of your user study are not biased I mean you're working with developers who know how Stack Overflow looks like how it operates so if you introduce something new why they're just following this new feature then so we didn't do any priming like we tried not to mention security or something like that and and we also tested for systematic differences based on the demographics and we didn't find any okay so looking at this example it seems like there's it's this sort of the this example seems very simple to find you're looking for just allow all hosts and verify is this model able to capture more subtle bugs or is it yeah sure how's it compared to say a trivial we're gonna look for these certain indicators and flag them yeah sure so this yeah this is a very simple example but we are also able to detect box-like based on the for instance like if initializing a key or initializing an IV or like yeah or most sophisticated vulnerabilities this is just an example and it shows a very simple case where you can where you basically have to find yeah like this this this Java field at this Java field is on one point would this technique be also be useful for finding seek other security but non cryptographic Allah much as buffer overflow probably yes I mean we so that the the whole thing is based on on code crafts and program dependency graphs and we feed these graphs into the neural network and there are other approaches that kind of use the same representation of code to actually find buffer overflows and so on so yeah [Applause] Shoebill from facebook is gonna talk about Krypton frame thank you for giving me the opportunity to be present on the behalf of the Krypton team Krypton is a very young framework it's essentially a machine learning framework based on secure NPC right now but hopefully other techniques in the near future the open sourced last October so giving us this opportunity to present means a lot to us I should also say that I'm not a cryptographer this is only my second time at real-world crypto and that might reflect in some of the design decisions we have taken in in the framework itself so a lot of the talk is going to be on how and why we design Krypton the way it is every design has a set of trade-offs Krypton does too but hopefully a different set of trade-offs than what we have seen in frameworks and libraries in this space especially in a space of secure computing I'm gonna use secure computing as a very broad term I don't know if formal definition exists but for me it is a computing technique where you're competing on data that is encrypted in some way to kind of showcase a forty thousand foot view of what these design decisions are there is a piece of code on the right hand side which may be foreign to a lot of people here but if you show this to somebody in the machine learning community they're gonna say oh it looks just like by torch and by torch is a leading machine learning framework along with tensorflow which also looks very similar and that in some sense captures what we are trying to do with Krypton our primary goal is to expose the machine learning community to various secure computing techniques and the trade-offs that they're kind of in my view two broad aspects to this the first aspect is that of the choice of models in the machine learning community now for example let's take an example of computer vision for example the models that are prevalent are deep residual networks there are actually not very convenient to work with in secure computing mainly because they have these nonlinearities that are hard to approximate they're also deep in the sense there's a lot of multiplies and sequence that that are problematic so making these trade-offs explicit to the community would hopefully help the community think in in different ways build models that are much friendlier maybe shallower models another concrete example is number encoding currently in machine learning we use float32 there is a special 16-bit float type called B of 16 neither of which are very convenient to work with and in cryptography or in secure computing which likes to work with integers usually so this is another thing that we want exposed to the community as there are there is a space to think about different number and codings when you're training models or doing inference with models the other aspect is to expose the community to a new feature a feature that allows them to encrypt data and into computing with data and the hope is when you expose this feature the community is going to think about more applications where the data is sensitive and cannot be trained on in the clear I should at this point step back and say that Krypton is very much a research framework nobody should take this framework and start training on data that actually needs to be secured we are not at that point yet but this is our first step and the hope is like when these applications come to be there are people from this remark from this community can help it build in the necessary security needed for the data and so we look at this framework as very much a conversation starter between these two communities because let's face it machine learning isn't going away and privacy is becoming and security are becoming more and more important so with that in mind let's look at what our main design goals are in krypton so the first thing that first major design goal is we wanted to present a machine learning centric interface interface based around tensors and computation graphs which I'm going to get a little bit into more detail later on that is how machine learning is done these days with neural networks the second one is explainable performance by that I mean having a design that is very modular because modularity helps in figuring out performance it's easier to work with performance problems in a modular setup then when things are one monolithic piece of code so this is something we are in mind because if things are not fast nobody's gonna use it the third one is debug ability krypton's a very young framework and the field is very new so he wanted a mode where users who use this can figure out what has gone wrong so right now we only have a secure NPC set up for example so we have a mode where you can have multiple parties on one computer for example which helps you debug when models are not training and I know that debug ability and security are kind of at odds of with each other with security you are trying to hide stuff while the debug ability you are trying to reveal information and I'm not really sure what the right design call here is and the last one is interoperability and by that I mean you know people have been doing machine learning for a while so there are model formats models that are people have trained or they have trained more or have model specified in other frameworks we need to be able to import these models and continue training or do inference so we have a compatibility layer which is actually open source from Facebook called onyx which is what we use to load models so not only can we load models from by torch we can load models for other frameworks like tensor products for example which is from Google all of these design goals leads to one trade-off which is the threat model so we are in the in the current version of crypt and we are in the honest but curious mode and I should also say that this is not something that is set in stone this is kind of the first setting on the dial and there's obviously gonna be trade-offs between things on the right and the things on the left so we want to pursue other address other threat models as well but we want to be very deliberate about what these trade-offs are what do you gain from getting a better threat model and what do you lose from on the usability usability side we want to make these trade-offs very very explicitly before I go into the details of crypt and I wanted to give a brief kind of overview on what machine learning frameworks look like what the ingredients are of what we have seen in the last five years or so so what makes an ml framework so the first component of an ml framework is this object called a tensor which is really a fancy name for a multi-dimensional matrix usually of six dimensions and lower so anybody who has worked with MATLAB this would be very familiar so it's a in frameworks tensors are first-class objects the next thing you need is something that you need to up that will do something with these tensors so we have different operators that are basically functions that take tensor says your input and 10 sources produce 10 sources output and these are chained together by directed acyclic graph which is called a computational graph you can think of these as operators that are chained together they they take some tensors as input and produce inputs outputs that go to other operators and this computation graph is a special structure so it usually has one sync node and then that sync node is used to propagate what is called gradients so in machine learning we are trying to optimize a function so we do a forward pass and then we do a backward pass through this computation graph for the gradients I'm going to show you a very very simple computation graph because picture is a thousand words and it'll help put some diagram behind what I've talked about so I'm going to start with these square boxes that are that are tensors that go into an operator which is a multiply in this case elementwise multiply of two tensors of two matrices that produces another another cancer so this is kind of what a forward graph is and to calculate the gradient we kind of invert the graph we kind of flip it around what I'm going to show it as a graph on the right hand side so for every operator on the left hand side you're gonna get a backward operator on the right hand side so for the mall we have a be mall operator and the black arrows are the gradients flying back and then we have two special operators accumulated gradient which essentially what they do is they accumulate gradients into the tensors from which the input came from so every machine learning framework in existence has something like this underneath it obviously the graphs get fairly complicated so let's see what among these components that we've seen exist in in Krypton itself we have tried to be as much one is to want as possible so we have what is called a krip tensor object which is encrypted tensor which you can think of it as an abstract base class people who are coming from the c++ world which promises some functionality that these tensors will have it doesn't actually it's kind of agnostic to what computing you're going to use to ensure security at this point right now we have we use as I said a secure multi-party computer there is a MPC tensor that sits underneath it what we want to do going forward is our goal is to have other cancers at this level so maybe we have a 10 that is backed by a holomorphic encryption for example at this level as well obviously not every tensor will implement every operation that might differ between what technique we use so underneath the NPC tensor we have two kinds of sharing so we have an arithmetic share tensor and we also have XOR a binary share tensor and we can go back and forth between the two to two of these dancers so what what I have said so far is only about interfaces and an api's there has to be a tensor which actually stores the data and and that is done by this thing called a long tensor a long tensor is basically a tensor within 64 type and this is where we kind of cross over from Krypton land into by torch land and this was a and PI torch as I said is a is a leading machine learning framework now in in the neural network deep learning space and this decision we took very very deliberately there are a couple of things happen when you have this hard link to an existing machine learning framework one of the things that happens is the interface filters up so whatever functionality and the API the long tensor has it filters up the API all the way up to the crypt answer you may or may not choose to implement all of these api's but it does does filter up and that gives a very natural interface to the people in the machine learning community the other thing that happens is performance gets linked in the sense when things in by torch gets fast Crypton gets fast and that's also a very deliberate choice and the other thing also happens for example nobody had well there really wasn't no long tensor in photographer crypt and showed up so Krypton also interfere influences by torch designs it's a nice give-and-take going on between these two frameworks and a third one may be a little bit non-obvious is the communication libraries so machine learning is a very distributed operation currently we can train models on you know thousands of processors so there is a communication library that does this it turns out that a lot of the communications that we communication patterns that we see in multi-party compute map very well to the communication libraries that we use for distributed machine learning so this was a very lucky find in some sense what this design choice also leads to is it kind of decouples where protocol specific optimizations need to happen and where non protocols specific optimizations say to happen so anything in the in the protocol specific stuff can happen in the cripton layer above the dotted line and then anything to do that is not to do with the protocol for example making the communication libraries fast or making some math operations fast can happen completely in the PI torch layer so this this makes life a lot easier because then we can make minimal code changes to Krypton or pi torch depending on what we're doing so now that we have seen seen tensors let's look at what operations does machine learning training needs and this is a obviously a restricted set of operations then what you would do in a general-purpose program but there are quite a few challenges so I'll go from the simplest of the hardest so its simplest from the view of secure NPC so the first of two simplest ones are matrix multiply or a dense matrix multiply which is a it's essentially is the component behind a fully connected layer fully connected layer is essentially a matrix multiply followed by adding a vector called a bias and this is easy to do because it's all additions and multiplies additions come for free in additive sharing multiplies come with Bevo triples and convolution the way it's done in machine learning is usually done using matrix multiply because a spatial extent of this convolution filters is very very small so if you can do matrix multiply fast we can do convolution as fast as well and a beaver triple formula doesn't not only holds for scalars it holds for as well so you can do it with tensors the next up is logs and Exponential's the logs we these are all done through various series approximations so logs we do through householder iteration exponential we do a variant of repeated squaring we do division using newton-raphson then we do power power and square root using essentially exponential for power and and log for square root and then finally we have operations that are very hard to polynomial e approximate so we have value which is a it's a weird-sounding function but all it does is if I if a value is negative it sets it to zero and if a value is positive it just lets the value through so you can think of it as an if condition basically it's people have tried doing polynomial approximations of this but it never actually works so these are the last the last line of rail you and Max and Arg max are done using circuits and this is what we have to go from arithmetic sharing to binary sharing and back and Max is usually used in layers called as a max pooling layer which essentially looks at a filter and finds the maximum value within the bit and the filter and art max is also used in the same layer so now that we have seen both the tensors and the operators we need something above krypton to make things work and the stuff that we need but need above krypton is to essentially some some metadata and data to accumulate gradients so we have a separate tensor which is called an auto grad krypton sir which is used in the backward graph so we now we have these tensor separate but it's quite likely we are going to fuse these tensors into one tensor which we can use for both forward in the background graph and we need one more object called a module module is essentially a kind of a convenience object so in the in the computation graph you saw these operators as nodes it's turns that it's good to have some state along with these operators and it's good to have a consistent interface for this operator so there's a forward function and a backward function so what the module does it's takes all these operators and adds this kind of a standard API so that graph can be traversed very easily module can also contain other modules so you can take a sub graph and express it as a module so there are some layers of the neural network that are better expressed as sub graphs so so that's where the module comes in handy as well what this module allows us to do is to essentially use what I said Onix so it gives us compatibility to noodle other models that have been written out using my torch or tensorflow or any other framework and and then we can read it through Onix as a sequence of modules essentially what this allows us to do is if you have pre trained models that I've been trained not encrypted but you want to do inference on encrypted data you can use those pre trained models you don't have to retrain anything if you have models that is specified in in a PI torch like fashion you can use Onix to read this non Train model and you can train it from scratch also using Krypton so if kind of seen all of the components that major components that Krypton has I'm going to spend a little bit of time on on communication as well in the next slide this term might not mean anything to most people here so already use is a communication pattern where you have a bunch of parties or peers who have a value and they need to exchange these values with each other some all their values and then broadcast it back to everybody so this is essentially open to all in in NPC speak and already you so on on everything and I was shown on the left is something that already exists for a distributed machine learning distributed machine learning need needs this communication operators as well the next part is reduced which is the same thing where you don't do the broadcast back so you send data to one party that party sums up all the data and you reduce to that party and that's open to one you can also do a broadcast you can one party can send different values to different parties so this you can think of as communicating from trusted dealer to multiple parties so these communication patterns already exist for doing distributed machine learning I also realize that this is constrained because this is fine when you have an out of n shares and parties and n shares it won't work for any kind of threshold scheme but it turns out that another form of distributed machine learning which is called model parallelism needs more flexible communication libraries and those communication patterns used for model parallelism are also can be used for doing more flexible communication that are used needed in in in threshold sharing for example so those are coming as well one thing I should also say is these communication patterns have been around in in at least in computer science for a long time they actually come from this very old communication library called MPI I don't know if people have used MPI here message passing interface it's used in machine scientific computing a whole lot because in scientific computing you have bunch of compute going on in parallel and then you need to broadcast results every once in a while so we found that quite interesting that it looks very much NPC ish from from that perspective so I wanted to give some examples of code I've shown one example at the very beginning but I wanted to show how similar code and Krypton looks like to code in Python so I'm gonna start with some code on the left in Pytor these are very small examples so this is something that takes two tensors and adds them and the code on the right is script and doing the same thing there is not a lot of difference you import a new library and everything has a in front of it in some sense and then you do have some in it so this makes it very intuitive for somebody who are used not just white or even even tensorflow looks very similar to this if you wanted to gradients so here is some code on the left and right which which does gradients so on the Left we have a tensor there's a cross entropy is what is called a loss function on the right we have the same variant of code in Krypton it looks very similar and what we have done also is we have machinery in place to run Krypton completely in a browser so a jupiter notebook is a very common tool in the space so we can load up for example Krypton an interpreter notebook to kind of get started very easily and this has proved very very beneficial for people who are starting on this starting starting to kind of play around with this and in fact over examples in our source code is actually they have examples in Jupiter as well so people can intake and get a taste I also wanted to show you what loading a real model and a real dataset looks like and this is a very small inference example with imagenet which is a very popular data set in computer vision for example so on the right I'm going to show you code so what I'm doing is here is importing so library initializing Krypton then there is a miniaturized again image transform which crops the image for example I load the data set which is loading it from a folder image dead folder this is the data set about 1.7 million images or so and then I load a pre train model address that eighteen is a full-scale model it's not not a toy model and then we encrypt the model we encrypt the image we get an encrypted output and you can reveal the encrypted output and you're going to get the same result as you would have done they have everything in in plain text so it doesn't look all that different from what you would do if you not encrypting things so this is what we have now so where do we go from here and by we I mean not just people at Facebook I would be would love to have participation from both people here and in the machine learning community as well because this is a long road so the first thing on our mind is improving performance we work in as I said and in 64 space there is very little optimized libraries in 64 space so we are writing some of like more optimized version of in 64 matrix-multiply in 64 convolutions Facebook has a library called FB gem for matrix multiply so rewriting vectorized code it's usually using avx-512 for faster matrix multiplies we aren't done yet so we have done some initial implementation we have to do more here on the hardware side my wish list would be support for wider data types 128 would be fantastic wider vector simply lengths would also be fantastic in this space the next one that we are working on is a trusted third party so our current restrict third party the trust of dealer which i think is fine as a first cut but we want to explore other options of doing of generating beaver triples one idea may be using something like an intellect Sorek encryption schemes in the space do I sometimes wonder why we don't have a service for generating beaver triples that will make our life a whole lot easier these are some of the near-term things that are working on going forward longer-term would be support for other secure computing techniques as alluded to should we have more pic encryption based answers should we have some sort of Enclave based answers we don't have the right answer and we don't know where what people would like to use so in some sense we are looking for feedback as well and also other research like things for example privacy and security are are not the same thing in many ways you can do secure computing but at some point you have to open the result to actually take an action so you want you know maybe your model and data is encrypted but at the end of the day you want to know what the classic what the classifier I told you and that might leak information about the model and the data so how can you quantify how much information you're leaking and this goes to like maybe marrying things like differential privacy techniques when we open from a secure computing domain and how much noise to add is one research direction that we want to pursue but one thing that we want to keep in mind is no matter what we do as a as a community in this space we should work in models and datasets that are actually practical there's a lot of research that works in the space using say M nest which is a very small data set or C 410 which is also a very small data set these are not very useful they're useful in some in proof of concept but it's very hard to take something that works and just say M nest and see far and then extrapolate it to something that is really practical today so with that in mind I want to set forth the challenge of some sort for everybody here and also in machine learning in some sense but a little bit of history so machine learning has had this challenge called the ILS PRC challenge I think it has run since 2010 I believe I don't think it runs anymore so what this challenge said about was the first it created a large data set a million images each image came with a tag of what object was there in the image and the idea was to train a model on the image and then classify with I and on a test set classified with very high accuracy so this is solved in some sense we can do this with very very high accuracy and not only that we can train a model on a 1.7 million image data set in minutes I think the fast in the record is 2 minutes 43 seconds or so but with some degradation and accuracy but within 15 minutes we can train you know very very good and this has completely changed machine learning as we know it in the last 5 years in some sense the popularity of neural networks now is because neural networks was shown to be the best way of doing this so the same vein I have a question to say we want to train on a million encrypted images say from image net classify with high accuracy we can cut ourselves some slack obviously we are working around cryptid domain we are not going to get everything we want so maybe we want and accuracy of say a relative 20% what we can do now in clear-text and then instead of being done in a minute say we give ourself a week I don't think anybody has done it to my knowledge but if we can do it I think this would be a step function change in the community it'll be the same step function change as how neural networks were first able to do the ILS VRC challenge I believe in 22 2012 with I forget the number but it was a massive increase in accuracy it'll be that that kind of a change so with that I wanted to introduce you to the cripton team at Facebook we are a very very small team alphabetically we are counting Brian Laurence mark Shobha myself Vinny and Shane where we are very open to collaboration we are a research group so and everything we do is an open source we will love for a love for people here to be interested what we do and collaborate so in the examples that you showed I didn't understand you you can actually specify a multi-party machine learning situation like if you want to turn the water using data from multiple parties how do you declare the parties yeah so I what I didn't show you is a tensor constructor that you saw how the source argument so I'm basically showing one party here so you can have one party you have the data and another party have the model and you specify the sources so MPI has this notion of a rank so you can say if Frank equals 0 which is the first party then you have the data if rank equals 1 you have the model and we can do multi-party computing any number of parties yeah so we have it in the examples but for convenience I didn't try it here Thanks hi thanks for a nice talk so when you spoke about the performance coupling between sort of my torch and the lower layer and then the cripton at the upper layer spoke about it very positively but it can also have a negative side to it because I assumed developers of pi torch are interested in optimizing my torch for the regular case and that sometimes might make it slower for NPC like if they reduce the amount of operations greatly but at the cost of increasing the amount of multiplications say right that would be faster on up in a regular execution sense but an NPC will be slower so do you do you have any thoughts on that yeah I mean that's a good point and that I think battle will exist just the fact that I mean I know the PI torch team very well so I could it's easy for us to influence let's say those those decisions and for example being able to do you know convolution in the n64 space if you went to PI torch team and said I want to do convolutions in the in 64 space be like you're crazy but here we are but yeah we're very cognizant of this and we are trying to push to have more of these features in pi torch so other than that community in pie tart is huge so the so the motivation there is if we can do these changes in pi touch core then everybody benefits in some sense but yeah absolutely that that will exist thanks you mentioned that you have to do some transformations potentially to do like the max type of operator do you implement specializations for a particular number of parties such as those in like a py 3 or Sakura and okay that was an explicit design goal not to have party specific optimizations and I don't know how far we can go with that at some point we may have to have party specific optimizations but we wanted to have that was actually a problem for us I think our first implementation we were using two parties and then we spent we went to three and everything broke hello Sahara is one from George Mason University thank you for a great talk I really enjoyed it so my question is about so the framework that you just described is another framework tied to how to do Shining istrico computation and so as we saw yesterday there are a couple of other like frameworks out there that does similar functionality provide similar functionality so you have a kind of like do you have any idea of how your functionality is compared to others in terms of because everything's goes down to how you approximate those functions like for example lock that you just mentioned or max pooling or some other functionalities so do you know have a look do you have a benchmark in mind that how do you compare with others how the approximations differs in terms of like the performance the leakage that you just touched by Sun and some other like features yes our max is actually not approximate or max is done using a circuit so it's exact max actually we are doing this now for our own sake the approximations we we do are valid in number ranges that we see in training so we want to do it I should also say the frameworks that were discussed yesterday there are much more general framework so you can write a code in in that specific framework with particular annotations obviously we are not doing that we are everything we do is very very specific to machine learning and this is a decision that we took very very consciously so our goal was to avoid compiling as much as possible because what we have seen as languages take a long time to get traction in a community just going from Python to to Python 3 took 10 years and that just one language so we wanted to stick to an interface that is familiar to community and start from there but ok yeah we are starting to look at comparing with how good our approximations are or fast they are actually our slowest operation is a division operator which is kind of weird Newton Rapson so the other thing with like using Newton Rapson is you cannot insecure even in a secure space you cannot say when did I converge because that reveals information so we have like a fixed number of steps of newton raphson that we do and yeah and that works for some number ranges but you have to see what we can do probably thank you what you say thank you so much so in your time at the n64 for deep learning your dynamic range I mean is there a decimal point floating over to the left somewhere because otherwise fixed point right now so which is represented as in 64 essentially so log is to look up in a table of 64 numbers log is not a lookup we actually do a series approximation for log so log you can expand it a series right right but that's not I mean if you're taking a log and you're rounding it down the integer around n 64 it's the map there's very few outputs there what there are very few outputs as in the precision or the met the biggest flaw you have 64 yeah and then the smallest all you can have is 0 yeah and so why do we need a new iteration for this so you want like because the in 64 we basically it's a fixed point representation 64 so it's a decimal point that is fixed Oh over over in the over in the left yeah thanks you again the next speaker Flavio you're from IBM research thank you good afternoon everyone this talk is towards a homomorphic machine learning Big Data pipeline for the financial services sector so everybody's talking about machine learning today and I'm glad a lot of people talked before me so I can cut a lot of the parts that I was going to say this talk is about a collaboration that we did with one of the banks with bankable disco in Brazil in the first in the second quarter of last year I would like to thank my co-authors in this work and also my collaborators shy and a victor that were with us there and also the reviewers of this conference because since I couldn't reveal the name of the institution and do today it was kind of a very dry abstracted I had to submit but hopefully are going to be able to do so present a lot more today so just to put things in context banco bradesco in Brazil Latin America not everybody heard of it but it's the second largest private bank in Brazil it's in terms of brand is the most valuable brand in the country and this is important because if you consider data leaks that acceleration and all sorts of things that can happen that damage your brand so security is paramount for them the number of clients individual current account holders is 72 million they do 70,000 tasks this is not only the transaction this is the task that involved the transaction that commits on a database and everything else that happens on the back-end per second and you're gonna say well why they are looking at homomorphic something to protect the data is because they are the embrace advance the technology very early so they want to be ready for when technology is available so what was the challenge that they came to us is sharing data amongst the different business units it's an interesting thing because there are in particularly in regulated industries like financial services because there are not only privacy laws but antitrust and a lot of other things that don't allow people to see data from different departments of your own company altogether right so they can be breaking some regulation somehow the order is this year the equivalent of GDP are it starts to be enforced in Brazil so they wanted to be prepared to how they're going to move this forward which then led to the end of 2017 beginning of and discussions in 2018 for them to start looking at what sort of technologies could be applied to keep the data encrypt all the time so they approached us because of form of encryption and the other important challenge is how do we do this in a hybrid cloud environment right and there is a lot in so when you do this sort of work what are the people that you're gonna get together right so just to give an example of the kind of the breath that we had to discuss there we had our sponsors entertained which are which were there are indeed team and the CTO but then you need the systems infrastructure people involved because you need to understand what the impact of this new technology is going to cause and everything else that people have there you need the data governance people because you need to show that what you are doing is secure to some extent the security people will say well that what the security people doing there well home or freak encryption and some of the advanced encrypted stuff that we talk here is not mainstream yet and these people don't understand necessarily what it is so there was a whole education process to show them what security can be achieved with it and the important people the data analysts because they are the ones that are going to consume whatever we did and obviously our team with very frequent technical meetings and exchanges along the process so just to put things in context homework encryption allows us to process data without giving access to it technically achieved by computer on encrypted data without ever decrypting the data which basically means that it's not encrypted in the registers or anything of the machine it addresses that problem that I mentioned before which is how I can share data when data can only be shared on a need-to-know basis and there are regulations and so on so how can we our address that aspect and it's important to consider what threat model we are addressing here for this scenario is the honest but curious which basically means that the entity perform your computation is a legitimate entity to perform that computation but it wants to learn from what you're doing and fhe is based on lattice cryptography the four quantum resistant to the best of our knowledge today so let's have a look at the problem banks and financial institutions use machine learning something what people called traditional machine learning which basically regression based machine learning for a variety of things and you can see about why not to the fancy neural network stuff because they are regulated industries it needs to be you need to be able to explain easily why a given prediction was done in a given way so if you have many hidden layers there and you can't say why a decision was taken that can be tricky so bank sees it is a lot for certain tasks like marketing loans approval and so on so the data set that we used where comprises real financial data over a window is a sliding window of 24 months of a measurement that they make about every one of us so basically the bank measure meet in five hundred and forty six individual explanatory features which is a mix of quantitative categorical and binary features and the other important thing is the amplitude of the values take take just one thing let's say your current balance your current balance might be minus few hundred dollars or plus few million dollars right so when you're trying to do machine learning this type of thing things get tricky because of how you do how you manipulate to the precision you want to do so this is what they do right and with that group that we put together we had to figure out what is case and what we're gonna do so we looked at okay so let's put a fully homomorphic encryption so it's action data store and see if we can do predictions if we can do machine learning with that so we took the marketing scenario the marketing scenario that they work with is one that can they predict whether someone who is going to need a loan within the next three months this is in this is an important task they do because they can upsell loans but look at the second bullet it's a rare event in that data set is around 1% which basically means that if I didn't do any machine learning just said no I'm gonna be right 99% with that but the Golden Nugget is in that 1% if you can find that 1% in your transactions that's where you make money so that's why the importance for this and the data is very sparse so what was the success criteria for us to do prediction homomorphic lee the first one was if i have an existing model an existing data can i encrypt the model in the data run a predictions with the same accuracy as the predictions done without encryption right and this is an important aspect because you don't go to a bank and say well you know this 10 years of modeling that you have threw it all away because I'm gonna start too low again no so you have to be able to do that with what they have the second is to perform a task in machine learning which is very important and quite often overlooked which is variable selection right so remember I said 546 explanatory features but the models will have tens of features because although we have a lot of features how many of those features are highly correlated so you have to get rid of those features and find the ones that are the most relevant for the condition that you try to to predict and the question is can we do this variable selection homomorphic lis with the same accuracy that's done without encryption and so those were the two main success criteria and obviously with some acceptable overhead because if you're doing everything as we saw from the last talk things can happen in seconds or a week or so so how do they do it today and this is an important aspect is everything is done on premise because this date is private it's confidential is sensitive so they don't put this information in the cloud and to prevent acceleration of data the environment where the data analysts work is a secure environment so you can't take your cell phone that sort of stuff in there right so when you can't take your laptop and come back with the data either so the data and the machines stay wearable which is very costly for organizations to do it that way and some of the organizations because of their regulations the data has to be physically separated and you can't use you can't even use a multi-tenant environment system right so you can see that the cost of infrastructure if you have to do this in-house is very high so we came along and said okay let's do this in the cloud right so if we only premises which is secure we take our transactions we encrypt our transactions and we send them encrypted to somewhere in the cloud where we can do predictions if I've already got a model or I can run some machine learning to derive new models I have encrypted predictions so the cloud cannot see anything it's honest environment but curious and I bring back the results and then I can decrypt so that was the premise that we did so when and we did that right our paper shows all the mats in how we organize the data how we encode the data how we try to optimize everything for assembly like computation but what I want to show you is more of the results that we got so our experimental platform a mainframe you're gonna say what mainframe why a mainframe right so why can't you do this elsewhere because most of the transaction data is on the mainframe out of the financial institutions they are also currently evaluating how they use the mainframe in an integral part of a hybrid cloud strategy so you have the elastic two environments that you can consume in the cloud and how this can work there our library hla-b runs in the cloud sorry runs both in the cloud and on the mainframe and it's open source so these were some of the characteristics that were appealing to them too to come to us and hope in the requirement of the mainframe so results how does this look like prediction we took it an existing model this is a 16 variables existing model we took the data we encrypted data include the model we run through an encrypted persistent regression based prediction model and the accuracy was pretty good right because we are using here an approximate number a scheme for profit encryption and the accuracy was very good so that part number one so when we prove that we could do predictions with the same accuracy the next step was well can we do very much selection can we do the training can we retrain that model with new data but now encrypted and we did this too so what we are showing here is the log loss of the variable selection based on how many steps we do in the training versus the sigmoid approximation that we used right so that was nation before because you cannot stop in your you cannot check how good you are and you cannot stop your computation you have just do so we did that and we show that for 5 in 60 steps with the sigmoid sigmoid approximation of 3rd degree of 7 degree was pretty much the same when compared to this yellow curve is doing it in the clear without encryption the next question you are going to ask me is how long does it take so this is the computational overhead computation overhead in terms of depending on the security level for 256-bit security was fifty times and you gonna say oh dude takes 50 times longer to compute now this is pretty good when you're talking on most encryption right in 2019 would be a few hundred times the memory of her head for the prediction wasn't bad either is about 20 times for 256 bit security so once we have this how do we put this together right remember this what's wrong with this chart the wrong with what's wrong with this chart is that this is the kind of a more research academic way of looking at it where I have machine learning thing and I have encryption and so on but I'm missing this part which is how do I deploy how do i generate my keys how do I store my kids how I manage my keys how I make sure that the whole system works and there is the coordination that I require right which basically means that I still need the secure environment because at that side I have data in the clear become encrypted to be deployed in an unsecure environment I can I have to retrieve my keys from my key store and remember these are homework keys they are very very large compared to everything else that we have been using so far and when I decrypt I mean they decrypt I need my secret key to decrypt so again I need this trusted environment otherwise that can leak my secret key and then everything it's gone away right so this is this is the environment that we have been working recently and how we integrate everything in a framework that can actually be consumed for what are you do mission learning whether you do searches or some of the others okay and I have one minute for questions [Applause] do you see homomorphic encryption as a way of reducing consumers exposure to like maybe data leaks and stuff well data leaks is still gonna happen but if it's if it happens in an encrypted form with a strong encryption then there is no damage on the data being licked a lot of the data has already been leaked just by going on the internet and it has been captured in an encrypted form but people are trying to decrypt it so if we get one computer somewhere or someday that could be vulnerable with lattice based encryption then I'll be curious you know you know you know based on your experience and all the work that you guys have done how far away do you think we are from you know fully homomorphic encryption to be at a point where it is sort of feasible for most of us to be able to use it so any commercial setting or you know a lot more than what we're able to do now which is not much because of the performance cost associated with it well actually for it's a use case dependent and we tend to say that right now we are at that inflection point whether the performance is adequate for certain use cases most of what I shown you runs another batch system the predictions they are not some second predictions they yeah it's an overnight task so if it takes an hour or ten hours to run but with security and I can outsource that to the cloud instead of having to to do everything in-house that makes a lot of sense thank you great that's thanks a speaker again thank you it's breaks now