The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

Yeah, this will be pretty cool. So, my name is David. If you're here yesterday if that in the other hall, um yeah, I'm kind of doing AI. I've been doing it for a while. It's very fascinating. I love doing it. Um I think there's so many cool and interesting things that you can do with it and um yeah, I kind of have a weird background, kind of all over the place, storage, um working on virtualization. Did a lot of open source, Kubernetes, and then like 5 years ago got into the AI and ML thing. Um but yeah, that's kind of a little bit about me. So, kind of we're going to talk about acoustic side channel attacks and the agenda we're going to kind of cover this weird moral duality with AI and ML. And I kind of want to just start with that because it's kind of important, especially in this day and age, especially after the keynote, which was amazing. Um That's And then we'll talk about literally how to build an acoustic keystroke logger. And the slides and scary enough, the code to do this will be in my GitHub repo. Um so and if you don't want to go through the training to train the model, the the already trained model will be made available for you. Um and so we're going to talk about the three very technical slides of like what how to build this thing, what the architecture looks like. Um and then we'll do a lot of demos. I think there's five of them. And then we're going to talk about real world attack vectors, what you could use this thing for. Um and then we're going to talk about what to do about it, like how to protect yourself and your privacy and uh possibly even the how to disrupt this kind of stuff. And then we'll have some Q&A at the end. So, let's start off with the moral duality of AI and ML. So, here's a tweet from Elon Musk. You know, he's got this super extreme positive outlook. You know, a lot of people, especially in the news you see, you know, everyone's kind of very fearful of AI and and machine learning. And you know, you have Elon Musk here says, "Oh, don't worry. Like it won't be uh you know, basic income." He says, "Things will get so cheap with AI that you know, everyone will have universal high income." And I would love to believe that. That would be awesome. But you know, kind of just look looking at history, that's probably not going to be like that, right? Otherwise, we'd all be sharing and everything would be all great. And if you look at the flip side of it, you see like this MIT study that says AI can already do 11.7% of the jobs that are out there. And another scary statistic is that you know, you have the the number of programmers have hit the same level as it back in the 1980s, which is scary. And it's you know, there's this extremely hopeful outlook on AI, and then there's this extremely negative. I think it's going to be end up somewhere in between, but we you know, kind of my advice or my suggestion for everyone is is like you know, it's here. It's not going to go away. I would really encourage you to take a look at it, you know, um use like Chat GPT or Claude, you know, whatever it is, you know, whatever AI or ML system, just kind of learn how it can uh help you out and help you make your life easier, um because unfortunately, it's probably not going to go away. Now, some of the very interesting and very cool things that AI can do, here is a uh it's a an uh uh yeah, it's an an I N coming up with new ways to cure disease. And so here's like an article about an an NIH article saying, "Hey, AI is pretty cool." I mean, you know, if you kind of take this to the end, it's going to, you know, who knows? We might be able to cure cancer and maybe there won't be any more disease, right? So, it has extremely good applications. >> [snorts] >> And it also has very extreme negative consequences. So, a lot of people don't know this and you know, everyone's talking about inflation and like what's happening, like why is everything's cost so much? Um, there's literally companies out there that are trying to figure out how to maximally extract how much you're willing to pay for things. And you know, like I would definitely click this link down here, this video. Um, and just take a look, just open your eyes up that, you know, some of the inflation stuff is greed for sure. Like I've probably a lot of it, most of it, if I had to guess. So, this is like the extreme negative of machine learning. And I kind of want to bring the light because we're going to get into a weird topic where I'm going to literally tell you how to build a keystroke logger from acoustics. >> That's happening in real time. >> Yeah, yeah, that's what those digital tags are for. For Thank you for saying that. Yeah, those digital tags, so based on the time of day, they will update the price because that's when most people hit the grocery stores. And when certain times when it's like lower or whatever, their algorithm says, they'll drop the price so that the price is cheaper. And if you use apps, like you're going to get a different price than what's in the store, too. So, just be aware of all this stuff that's happening, right? So, this session is literally based on this archive uh research paper. It's a practical deep learning uh based on acoustic side channel attacks on keyboards. And they basically created a machine learning model to when you push a button on the keyboard, just on the acoustic signature alone to determine and figure out what key you're pressing on the keyboard. And this is where we're going to start kind of this discussion in this in this like this session. And I don't know if it's good or bad. I I improved on it a little bit. So, um yeah, we're going to I'm going to I'm but I'm going to describe it. I'm going to tell you exactly how and and also hopefully how to like defend against it. And one reason I'll kind of get to at the end like why I'm giving you the code to do this. Um in order to disrupt, you need to learn. And so, that's one reason. So, let's go ahead and start. So, building an acoustical keystroke logger. So, it sounds wild and it sounds crazy, right? That you can literally build sound classifiers to classify sound like different types of sounds. But in reality, this is happening for this has been happening for years. And many people are doing it. And so, kind of like the process is you have sound and you need to record it. And you need to have a lot of recordings and a lot of people you you know, you record it. Most of the time it's a wave file. Could be MP3, but typically people use linear 16 wave. And you convert that wave file into images. And so, in the upper right-hand corner is what a wave file looks like in pictorial form. You can see the frequency and intensity, which is the frequencies on the Y axis, and then over time is the X axis. And so, this is what a pictorial representation of a of a sound is. And if you look at the picture just below that, there are many different types of sounds in each one of these squares. But what you can immediately see just looking at it that there are certain patterns that are there, right? And so, those certain patterns are saying like um like the one right here and the one next to it, you can probably just looking at it say, "Hey, these two sounds might be very similar. They might be of the same thing." Whereas the one next to it, you it's obviously very different. So, it might be a different sound. So, people have been actually doing this for quite some time. Starting simple, you know, dog barking, cat meowing, you can be able to tell the difference between the two. Even environmental sounds, I know that there the last one is there's an actually an application of this for um recognizing gunshots in urban settings, and being able to distinguish the difference between a gunshot versus a car backfiring, versus a firecracker. I live in Long Beach. Every day there's a firecracker. It's ridiculous. But being able to you know, make the distinction between those, so maybe you can signal to the police saying, "Hey, this is actually a gunshot, and we need to send someone in this area quickly, right?" So, this has been happening for quite some time now. And just to give you other applications, so I last year I gave a session in at AI Dev in uh Amsterdam, and this was actually recorded. The GitHub repo is still there, but I actually used created a multimodal model to do sarcasm detection. So, you can actually find out whether someone It's not 100% perfect. It was actually the confidence level was approaching 80%. Well, using Well, so using the data sets that were there, which we basically used I used an existing data set that was for like Big Bang Theory, Friends, and and stuff. But, you could just like you're saying, um there are definitely It's not ever Nothing's ever 100% perfect. And I actually I ended the session saying like if I took something that Aubrey Plaza, who's very deadpan, and ran it through, it would come back she was serious every single time. And so, you know, there's a lot of flavor and uniqueness that we as individuals all bring. And so, um you know, it sometimes it will work and sometimes it will won't, but it's a very still a very interesting application of using acoustics and also facial uh like seeing eyebrows, you know, when you make gestures and, you know, you as actually using machine learning to do the voice and also the facial affect changes, like your eyebrows raising and all that stuff. Uh but, it was recorded. I would go ahead and take a look at that. It's pretty cool. We're not going to watch that though today. >> [snorts] >> So, if you saw my session yesterday, it always starts with the data set. And we're going to use two data sets, in fact, actually because we're going to solve two different problems. So, the first problem is uh it's a data set keystroke noiseless final. Apparently, it was used in some uh academic potentially like, you know, uh contest where they were trying to do this exact same thing. And it was using a single keyboard, so we didn't vary the keyboards. It was actually one keyboard that was fixed, stationary, used for the uh training data, the validation data, and the test data. You don't need to worry about it. It was just a single keyboard that we use. We're trying to detect uh sounds based on a single keyboard. Then the second data set, which is the one right below it, and these links work so you can actually go download the data sets yourself if you want. I also provided links in my GitHub. But, the second one is multi keyboards. So, it was like five or six keyboards. I think six keyboards um of all various different types of manufacturers, like I think um Lenovo, MacBook, um some other ones. This might sound frightening. Uh Lenovo over Zoom, uh Lenovo over uh Messenger, I think was the other one. Um so, different keyboard sounds whether you are physically there or on the other side of a different laptop. And you know, I I pointed to both of these research papers so you can like kind of like look at yourself to see like kind of like what their kind of conclusions were. Um obviously, the multi keyboard problem is far more difficult um to to crack, right? Because different sounds for different keyboards. And so, now you there in lies the problem for that one. But, two different problems, single keyboard, multi keyboard. So, we kind of already talked about a little bit actually about the the data processing pipeline, how we start. So, we had need a lot of recording, and those links provide wave files for the first one was I don't I can't remember what keyboard it was, but it was multiple clicks of every key. I think it was like six or seven times for every key on the keyboard. And then, the multi keyboard data set was multiple keyboards, multi clicks for every key on every keyboard, right? And we need to take those wave files, turn them into that pictorial spectrographic image, frequency over time, with intensity, and the colors all matter. And we need to create classifiers, so that when you hit a button, we classify what letter is being pushed when you hear the sound. Yes. We'll We'll get into that. You're jumping way ahead. >> [laughter] >> No worries. For sure. So, if we're trying to classify all these sounds, we need labels for them. So, right? A through Z, 0 through 9, space, enter, and we're only going to look at 38, and it was just for train time training for the training time for all this stuff. But, we're going to do frequency over time. We have three different channels because it's a color picture, RGB. And we're going to fix the size of the picture, so that every picture is the same. We're going to normalize some stuff because maybe the key press isn't exactly at the start of the wave file, so we're going to like shift things over and do all sorts of fun stuff. Basically, we're going to try to make it so that it we can train with adding variability, so that we can hopefully classify these key different keystrokes. >> [clears throat] >> So, here's the Actually, these are the three slides that are most nerdy and very technical, and then we'll get move away from it. So, the model training. So, we use a Coats Net style architecture. It's just a neural network that does depth-wise convolution and self attention. Remember the word attention from yesterday? So, we can go back over what was previously important to like what is relevant as we're processing the sound. Um but ultimately, we're kind of like using local patterns and global context. So, local as the frequency starts to change with as we progress through time. And then take in the final picture for the entire image as a whole and try to find the patterns both in time and as a whole. And when we're training this, we're going to do something super and I didn't invent this, by the way. It I'm using literally the stuff that was used in that research paper in the beginning. So, I didn't like do any crazy hand-wavy stuff. Um so, they what they did was they trained it for an unusually, very unusually high amount of epochs. And an epoch means you have all of your data and you're going to do one epoch is one pass of the data. And you're going to we're going to do a 1,200 passes of the data at a very small learning rate to find out what little things make this thing very, very special to make this like the sound for the letter A. And so, that's kind of like the very high-level version of like what we're doing here. There's this other fun stuff, optimizers and whatever. We'll just skip over. So, the first problem. Acoustic key detection, single keyboard. So, our we have training data, which is we need to learn like what the sounds sound like for every letter on this one keyboard. And then we have validation data, which basically says when we for this letter A, we have validation data, so another sound for a different instance of a key press and we're comparing the two and we're basically trying to get these two to converge. And then we have test data that says we can test it to see how close are we to getting these two things to converge to 100%. So, the goal is you hit a button, you want it to be 100% certain that that letter that button press is the letter A. But we're only using a keyboard, which will drastically reduces the scope of the problem. The very fancy words, single keyboard, we just want to make sure we hit a key many keys on a keyboard and we want to make sure that we get the right uh acoustic signature and the right classification for A through Z, 0 through 9, space or enter. >> [snorts] >> So, instead of just talking about it, we're going to demo it. Cuz talking about it's kind of boring, right? Um so, So, in my GitHub, which you'll have, and don't you don't need to take pictures or anything, the slides are also in that GitHub, too. So, you'll have the links to click and everything. Um so, the first thing we need to do is we have a bunch of wave files. Should go data. We'll just look at the training data for one. And I've obviously reduced it. We're only going to look at the letter A. Because otherwise this would take forever to do. And so, if we look at doing LS of training data, the training data and the letter A, you can see that we have roughly eight sounds, eight different wave files for the key for different instances for the letter A. You pressing the letter A, record it. Press the letter A, record it. Over and over again. So, we need to convert these wave files to spectrographic images. Right? Those pictures. So, let's go ahead and do that. And because I can never remember these commands, I have this cool little thing that will just and copy and paste. Oops. It's just a lot. I have too many Oops. So, pretty quick. And if you receive, it's in the images folder, right? So, we converted those wave files to images and we do an LS, there's now an image directory. Actually, let's just go ahead and look at one of them. To prove that I'm In fact, we did training, the letter A. We do an LS, we have PNG files. If we open one of them, super small. This is the letter A. And that's just one instance, right? This is the letter A. >> [snorts] >> Different, but different instance of it, yeah. Yeah, it and it's intentionally like that, so it's chopped off so that every wave file is exactly the same length. In machine learning, when you need to compare things, they all need to be of the same length, same dimensions, same like whatever it is. Like whether they're the pictures or or if you're chunking text, like 2K or 512K, right? Um I can't remember off the top of my head. You have access to the code. Um definitely you know, you can take a look at it later. Um I I just don't remember off the top of my head, sorry. Um so, oops. So, we now we have wave files to PNG files. So, let's No, fixed. Yep. Yep. >> [snorts] >> So, now we've converted Well, we've done the sample set, but we've converted all of them to PNG files. So, if we actually look at our data, we go >> [snorts] >> Oops. Okay, yeah. Sorry, I've already divided it out. Um so, if we move not org, so, train test, Uh I'm just going to look at training data. All of the different keys, I'm just [snorts] going to pick a random letter, letter N PNG files, right? So, every letter we have a multiple PNG files for every letter so that we can train our machine learning model. This would obviously take far too much time. But, you to train this model, Python one one {underscore} train and it will build it. I would advise you using an H200. It will take roughly about 14 hours. Um but it is there. So, you can train this and build it. And if you don't want to do that and burn your own money, you can download the models. I provided the link that's already built. So, you can do this on your laptop at home. And we have one wave file here, S {hyphen} 09.png, which represents the key press S. And we're going to now you take our model, we're going to take our image, run inference on it, and see what happens. And again, this is a single keyboard. We'll we'll get into it. Yeah, yeah. You heard it. It's the same You're not jumping ahead. You guys are smart. >> [laughter] >> Um so, take the the keyboard, run it or take the picture, run it through. 100% certainty. I know this is the letter S. Single keyboard. So, if you're using a single keyboard, we can actually pass it through this machine learning model. It will guess 100% certainty. Cool. Or I guess not cool, depending on your take of this. >> [laughter] >> So, single keyboard, we can actually, believe it or not, with 100% certainty be able to classify what keys you're pressing. Now, let's look at Uh can we save the questions to the end because I'm going to run out of time for sure. Um so, a multi-keyboard. So, we have multiple keyboards. Um Lenovo, Apple, HP, MSI, over Zoom. I can't remember which one it was. And then over Messenger, I can't remember which one it was. Just goes to show you, as you're typing, don't have your microphone unmuted. So, now we need to do the same thing. The previous one was a single keyboard. Now, we have multiple keyboards, so the problem's far more difficult. So, we're going to divide the all of the sounds for all of the keyboards, training, validation, test, right? And we're going to train our machine learning model to hopefully guess multiple keyboards. So, let's go ahead and give that a try and see what that looks like. Um ch- So, um just like before, Python one {underscore} train. This will build the machine learning model from scratch. Um and I will provide the download for the completed one so you can just run inference at home. And before everyone gets freaked out, keep in mind that they're these keyboard sounds, right? Not like just generically anyone's keyboard. Okay. So, we're going to run inference on it. And for this one, oops. We're going to use the Lenovo five PNG H, so the letter H. And we're going to take that image file and just pick one randomly. And keep Keep mind that this this machine learning model has been trained to detect uh six different keyboard sounds, two of which are over a remote meeting session like Zoom or something like that. And we're going to pass that keyboard in or that sound for the letter H for Lenovo and see what we get. >> [snorts] >> Now, notice that it's vastly different. Right? The other one was 100% certainty every single time I can figure out that this key button is that one. Now, what's very interesting is if you look for that image, still the number one slot is the letter H with 40 uh 8% confidence. And I set the top K equals to five, so we can see what the top five guesses for our machine learning model are. The next one's G at 24% confidence. The next one's J at 16% confidence. Um the next one's K, so on and so forth. So, keep this in mind because it will be important a little bit in a second here. So, we can see that the confidence score drops dramatically once we it start to introduce multiple keyboards. And it's although this one happened to guess it exactly right. Um not exactly right, but I mean with the 40% confident 48.6% confidence, it can guess the letter H. So, but we have these little nuances because now we've introduced multiple keyboards. So, let's see. Um What am I doing? Hang on. Sorry. Yeah, I just forgot which one. Okay. Yeah, it is this one. So, now if you wanted to simulate what typing is, right? You you typing multiple letters um with the multiple keyboards data set, we're going to spell out a little message and we're going to use enlist the help of these sounds to simulate me typing something on a keyboard. I just picked one, the Lenovo keyboard. And you can see the different images that we're going to use and pass through our model. The letter A, C, D, E, H, and so on. And we're going to spell out a message using these images, which would correspond so it'd be effectively I were typing on a keyboard. I'm hitting a button, the sound can be picked up. We'll pass it through the machine learning model. It'll tell us what like what letter it thinks it is with a confidence score, hit the next one, so on and so forth. And we're going to try to spell out a message and see what we can what we can do with this. So, you can probably figure out what I'm trying to say right here on this side right here. So, hello scale. So, we can see that the first letter is H, 48% confidence, E, uh 69% confidence, L, L, O, space, D, B, S, L, E. So, it wasn't able to get all of it right. But there is a reason why I and this is actually where the paper I not failed, but maybe lack of imagination or whatever, they were trying to get the thing to give you the letter as the the you know, this is the most confident letter to be selected. And there was a reason why I picked a top K of two because you can actually get S, C, A, L, e Right? So, I increase the top K to two. This is basically saying, "Okay, D is not correct, but S is." So, how can I use this to my advantage to figure out what someone is saying? And so, I'm going to do a live demo. It's going to hit the internet. So, there's something very interesting here. Like, most of the keys on your keyboard are all kind of the same, right? Like, they're you know, the same shape, right? What is one key that is very different from all of the others? Got it. So, I don't need to test every permutation to figure out what this message is spelling out. I can delimit it by space, chunk the word, so that the word is here, space is a hard delimiter, look at the next word, and now run just the permutations on just five letters versus the entire string. And so, that's what we're actually going to do. And we're going to do this I'm going to hit the internet. I'm going to take every combination for the first five letters. We have a space delimiter, then do every combination for the next five letters, and I'm going to run it through a spell check to give me actual words. Right? And what we're going to do is we're going to do spell check. And hopefully the internet will work. I'm hardwired this year. So, hopefully it will. But we're going to run through every permutation to give us my firewall. And every complete word, we're going to output it to the screen and also save it to the file. Hello. But we already knew that, right? If we look up, we can see that the first letter is h e l l o, right? Top confidence score. Right? And so, it's now going through 32 different combinations, five letters, right? Five to the power uh two to the power of five is 32, right? 32 different combinations. It's going to run through, find every potential combination, hit the space bar. Now we're going to take the next five letters, scale, what's supposed to be scale, and we're going to run those 32 combinations, same number of letters, five, and we're going to run that through a spell checker, too. This is going to take a little time. I know this is probably anticlimactic here. Um keep going. Okay, cool. And I'm actually going to output like some of the potential words. But we already know that hello is one of them, right? Because we can see complete hello. We know that's a word. We just threw it out there. Plus, um it's also the topmost word. That's uh the confidence score on that's from the spell checking API. I forgot which service I used, but it was free, so I just ran it through. Now we're looking at the second word. We're using the space as the delimiter because it has a very unique sound, because it's the the key is very different, and then we have scale, right? Complete word. And you know, and it's going to go ahead and it's going to chunk every other or chug through every other uh key to try to figure out what other words there are. And I could have made this a lot more intelligent. Like, obviously able is a word, but it's only a four-letter word, so I could just junk that, right? Because it's we know that there are five keys. And we got a four-letter word, that's obviously not it, throw it away. And then if we do a cat on our, which we already know, completed words.text, which is dynamically throwing it in there, hello scale. It doesn't need to be perfect. It needs just to be close enough to get to where you need to go. And then employ little things to say, "Hey, think outside of the box. How can I get that to get closer and closer and closer? And so that's whether it's good or bad, that's the improvement that I kind of realized and made is don't just look at top k equals one, maybe top k equals two or three. And then try that in different ways, right? So. Oh, So then it occurred to me if you're typing away and you're listening to all the sounds and you're trying to use brute force to figure out what these words are that might not be 100% perfect. Like it's going to probably screw up every once in a while, right? And it's going to be like give you a garbage word or a word that's not a word or maybe it doesn't even make sense. So if you saw my session yesterday, I created a masked language model, which basically says, "Hey, I think this word is garbage. I'm going to omit the word in the middle of the sentence and I'm going to use AI to help me figure out what that word the most probable word to fit in that location is." So basically the idea is I want to be able to figure out what this person is typing without actually intervening at all and have AI do all of it. And so the purpose of a masked language model is uh if you've done Mad Libs before, you have a sentence, you have a space, and then you kind of want to be able the point of Mad Libs is you put in a funny word and it's you're all supposed to have a laugh. But we're going to instead of making a funny word, we're going to find the most probable word to fit in that location. And so um this is also in GitHub, too, to train this model and build this model from scratch. If you were in my session yesterday, we're just going to run inference on it. And we're going to pass in two sentences. So, they're just example sentences, right? That says, um, "The capital of France is blank." This is the same demo I did yesterday. And the blank, the most probable word is Paris, right? The capital of Paris or the capital of France is Paris. Distilled models are the blank of of large language models. They are the smallest of large language models. So, if you came across a word that might seem like garbage or not correct, you could put in a blank or a mask for like what that word is and then try to have AI fill in the blank for like what maybe the most probable word is, so that you don't have to intervene into trying to figure out what someone is typing on their keyboard. So, let's summarize all this. On a single keyboard with 100% accuracy, like you can figure it out, right? So, if I collected the sounds on my MacBook here and only had the MacBook and trained a machine learning model 100% of the time. Multiple keyboards, I ran it through, you know, I used all the test data, ran it through, and it was only 46% accurate. 46% is actually, if you think about it, sounds low, but it's actually pretty dang high because if you did a normal distribution, if you guessed randomly what the key press is, it would be one out of 38, which is 3%. So, almost half of the time the the most probable answer would be the right key. And maybe the second one it maybe it's not the first one, maybe it's the second one. In the worst case, maybe it's the third one. But, that's still pretty crazy. So, what we did was we basically said, "Hey, let's consider the second most probable or the third most probable and try to like work backwards from there." And then use AI in that last demo to fill in the blank for words that we know were garbage. Now, what we did not test and this is where you can sleep better at night. So, we did not take we did not train a machine learning model and we did not introduce a new keyboard. And I don't need to do that test because I know that would fall apart very, very quickly and the the results will drop dramatically, like very, very dramatically. And if you want to look in more detail about that, the sarcasm uh video that's recorded, I actually did the this independent data set test, which is I introduced a whole new uh comedy TV show to detect whether it's sarcasm or not. It was actually still good, but it starts to drop off dramatically and I didn't need to do it here because I know it would break way more drop off than, you know, before. So, I didn't do this one. I didn't do the next one, either, but I think this would be very interesting. What if I took a MacBook or a keyboard manufactured in the same lot and I'm basically banking on the fact that the manufacturing process to build these keyboards are so identical that I might be able to use maybe this keyboard to capture the sounds, but in the same lot number be able to see if I can guess what number that the key press is on that. And to your question, I actually have this picture right here on the the right-hand side. This is the frequency in which you would normally press keys just from normal typing. And it's actually a well-known thing that you hit certain keys more than other ones. And so, what I'm saying here is, like the letter E is the most important like most pressed key. So, what I'm actually saying here is because we know the frequency of how often we hit keys, maybe the wear and tear on everyone's keyboard is all the same. I didn't do this, but it's a very interesting thought experiment, right? So, So, some of the improvements that we could do, which is getting to I forgot who's which is your question, the things that we could do to potentially maybe even improve this even more, not that I'm going to get in trouble by the NSA or something already. Um when you type naturally, you flow from one letter to the next letter and by changing the variability of your hands, maybe that has some sort of change in the way you might hit the next key. So, what you might actually need to do is you have a understanding what you hit before to better predict what you might hit next. And um I think so what the other thing was, you know, maybe the person could change based on like I think it was like uh man versus woman or you know, different people hit keys at different strengths or whatever and maybe that has something to do with it. But, you're still hitting the same key, so maybe you just need to do even more training with a larger sample set to try to even figure that out. And then you have the idea of like multiple users. I know most of us was was taught like the type on home row. I don't do that. I weirdly memorize the pattern of what the keys are on the keyboard and I like peck at it like really fast, but I don't have home row. So, I'm sure that would affect it, too. But, kind of the point is is you need to record all of that stuff to try to teach the model to recognize all that stuff, right? And to to filter out the stuff that is not important like that variability to find and hone in on the unique thing about that key to make that key press recognizable or classifiable. And then there's also noise, right? Background noise. That could add into it. So, real world attack vectors really quickly. So, this isn't a new thing. This is actually really weird and fascinating. So, there was a toy that appeared in Captain Crunch as a whistle. And this whistle, when you blew into it next to a payphone, it would actually give you free calls. This is a side channel acoustic attack back in the '80s, '70s, '80s, probably '90s still worked, I'm guessing, because they were sold at home base. So, it's not a new thing. It's just a different application of the same thing, right? So, not that I'm trying to like do bad things here, but if we were to manufacture a an attack using a single keyboard. So, let's just say we need to do something evil and you need to have a single keyboard to try to make an attack on that particular keyboard. Like, what kind of keyboard would you attack? This is the first thing I thought of. Everyone uses that same keyboard and you could like walk up to the ATM every day, hit the one a couple times, record the sound each time. Next day, do two, three, four, whatever. Again, I'm not trying to like come up with something crazy. Like, the credit card readers that you put the covers to get the PIN thing, so that when you slide your card in, it also gets reads the magnetic strip, you know, like people that are trying to steal your credit card information. Put a microphone in it. Right? Record the sound. The The insert of the card triggers the recording. The recording listens to the keys you're pushing. Boom, you can tie the two together. Frightening. Multiple keyboards. So, like I already said, you can't introduce a new keyboard because we need to actually find out what the unique sound for that particular keyboard is. So, this one's a little more difficult, but you know, if you have your keyboards out and someone has the patience or maybe there's that manufacturing similarity. Maybe they can get all of those sounds, capture them, and then, you know, listen over Zoom or whatever the heck it is, and they can listen to what you're typing even even over Zoom. And this is just kind of like a wake-up call, right? Just be very aware of like what's happening in your surroundings. Check I'm I test the ATM thing every time, even before this. >> [clears throat] [laughter] >> Um so, just be aware, right? So, now that I've frightened the crap out of everyone, um protecting yourself, privacy and security, right? Like what do you do about this? Two-factor authentication. Please, do it. >> [clears throat] >> This is the wake-up call, right? Um I like this is a Yubico YubiKey. Whatever you like to use, use it. Um I would not use SMS. It's still better than nothing, but um because there are ways of doing femtocells and anyhow. Use two-factor authentication, whether it's the Google Authenticator thingy, you know, the little QR code where you have to type in the number every, you know, 30 seconds or whatever. Use something. Use passwords you don't know, right? If you don't know the password, right? There's no way anyone can socially engineer what your password could be based on your kids' birthdays or what your interests are, who your favorite football team or whatever is, right? They can't socially engineer it if you have no idea what the hell your password is, right? So, use a something I would recommend using something offline because things can get hacked and things get leaked. Make sure every single password for every single service is different. Do not use like one password for all your credit card companies because then one breach of one now cascades into all the other stuff in your you know, whatever you use. Change your password frequently. I I know everyone hates the you're you need to change your password every 90 days from your company and you look at it and you're like, "Oh, I got to something else." Change it frequently, right? Now, this is going to sound weird. And this is actually kind of like the point of maybe this session or whatever. I kind of like live by this motto anyways. Be intentionally difficult. >> [laughter] >> Um >> [snorts] >> I I not to the point of like a harassing or whatever, but be be difficult, right? I mean, there are some things that I implore like that I do every day to you know, sometimes it's not even on purpose. Like I type as I'm typing in my password, I screw up and I backspace or I you know, but even weirdly introducing uh errors on purpose could be enough to throw off some things, right? And so, I'm just saying be intentionally difficult. The whole point of machine learning is to figure out patterns. Be the anti-pattern, right? Yeah. >> [laughter] >> So, this seems cool. Apple acquires a new startup using wearables to interpret facial skin micro movements and there's a patent, US patent for it. And you know, the stuff that's coming out of tech these days is crazy, especially with AI. But when I saw this, I thought exactly this. If you're talking about micro-facial movements, you're literally talking about the ability to lip read. Right? You don't even need audio in order to figure out what someone's saying. And especially with all these new wearables that are coming out where, you know, like the necklace thing, I'm sure Open AI think I think they have a necklace thing. I think Apple wants to put in cameras in their earbuds now. Be aware of your surroundings and like what's happening in the world, right? So that brings us to resources. So everything code all of it is on my GitHub. And why I'm giving the code away. Um it's like I said, it's not something new. It's not something I invented. But I think it's important. I hope that maybe some of you will take the code figure out ways to disrupt this. Right? Figure out ways to like put a monkey wrench in all this nonsense and make it difficult for people to do these things. Um there are maybe creative ways, there are maybe technical ways. Um yeah. So everything you saw to be able to do everything I did today and all the demos is right there in my GitHub including the slides. Um the original research paper, the sounds, um the data sets for the single keyboard, the multiple keyboard. >> [snorts] >> If the I included the link for the sarcastically speaking, so being able to detect sarcasm. Um so you can click the video, watch that. It's an hour long. It's pretty cool. Um also I'm going to throw in this one here. I did a session at We Are Developers. It was called uh confuse obfuscate disrupt. It was basically being able to confuse AI enough to get it to do things that it's not supposed to do. Um, and there are definitely ways of going about doing it. Some of them are super simple, like misspell a word intentionally, so that it can't recognize that it's a certain word. You get your intention across, but it might be able to disrupt the AI to have its guardrails work correctly or whatever, right? Um, I can tell you I would definitely go ahead and give that a a watch. Not that like trying to sell my session or anything like that, but I mean, honestly, when all these AI agents start popping up, I'm getting a discount on stuff. >> [laughter] >> Right? >> [snorts] >> For people that are building these AI systems and they're not kind of like building in the governance and the security, yeah. I mean, deal with what you want, right? Against these some of these corporations. And that's it. Thank you.