ToorCamp8 2024 - THE DL ON LLM CODE ANALYSIS – Richard Johnson

[Music] the down low onm code analysis okay thank you so yeah in my day job I work for eclips and do Platform security and they're also uh paying me to do llm stuff and I run my own training company on fuzzing also uh I am the organizer of the CTF uh and I want to just put a shout out to some collaborators this year um specifically superat who I've collaborated with on part of the CTF when you get your CTF badge activated this afternoon uh scan it look at it deeply and you will be onboarded to the CTF also there is some broadcasts that we are doing that are part of the CTF uh called the Olstead broadcast I'd like to um thank Jason and Bliss for contributing to that there's um multimedia experience involved there and then there's the unpon pable from uh 2022 that I brought to Camp that has not been solved yet so there's one reverse engineering and exploitation oriented Pon but uh so I have a bunch of slides and I'm going to skim through some of these early ones to get to the extra meat that I've added to this talk but um today's talk is basically about how do we apply deep learning and llms to code analysis and what are the um risks in deploying such um infrastructure but before we get there um over the last year and a half we've seen a huge expansion of developments in the llm space and this kind of breaks down from the uh GitHub uh active repositories and the timeline of you know how much activity has been going on in GitHub and these are split across four kind of strata of software you have the end user application layer this would be the things like chat GPT that you might use on a website um but below that there are things like agent systems and um libraries that allow you to chain together behaviors um that get llms to do the application layer stuff and below that is how do we fine-tune and train llms to do um task specific um goals and and fine-tune them beyond their base knowledge of all the information gathered off the internet but give it a specific set of data to focus it on um to become highly performing and then past that past that um there is the ultimate uh how do we create models from scratch in the first place and so um these are different kind of software packages and libraries and infrastructure that has been developed over the over the last year or so and we can see this um kind of you know uh expansive growth over time so what are the things you need to know to uh get involved in going Beyond using chat GPT yourself you want to perhaps write some applications that interact with uh proprietary data for your business or whatever it might be and so um we can break down llms or or neural networks that are in use today into a few different categories we have the vision category which are the things that are doing the self-driving vehicles and uh image recognition and image generation and things like that and these are called convolutional neural networks CNN and what this really means is that it's passing a window of observation over a data set so in the in the case of imagery there is a two-dimensional array of pixels that are going to be what you see on your screen and they have a 3X3 or 4x4 5x5 square that they pass through those pixels to do things like Edge detection and whatnot now you can apply a similar approach to doing uh things in code like graph analysis if you convert a graph into a graph neural network and it does a windwing approach and passes over it and you can do detection of patterns within graphs okay so that's kind of where things really started to to ignite and on the side there was the language natural language processing um vertical which really started off with um recurrent neural networks and the way the reason that rnns are important is because they started to have a way of reading information that came from the history of what they have already emitted so they have a forward and backwards kind of looking um pattern of doing the neural network propagation and um through that there was some improvements called lstm which is long-term or long short-term memory and um that was a way to improve the amount of context that it could understand and reason about and then ultimately we now have Transformers that have added the attention mechanism and in 2017 there was a paper called attention is all you need and it's what allowed uh neural networks to actually have context and and reasoning capabilities because it uh stores an intermediate state of its processing that all the layers ultimate ultimately flow through as a filter and so it allows it to achieve um better connectivity and um more relational information that you're trying to extract from what is ultimately a compression algorithm like so neural networks at the end of the day compress data and use heris much like a Huffman table to determine what are the most likely symbols that are to be expanded um from this kind of compression now we have uh in the Transformer world we have two different different things we have the Bert and the GPT line and Bert is typically used for classification um clustering grouping it is more or less what the traditional um machine learning models were doing um but at a higher level of capability and then there's the generative side which is the gpts um that are the things that you're probably more familiar with and talking in a chat model type thing okay so how do we get information into a model or how do we you know go from plain textt into talking to a model so first you're going to start with uh based upon the language that you're interested in you're going to create a dictionary that are basically engrams of that language so what are the most common letter combinations in that language there there's going to be you know consonant vowel combinations that are common there's going to be common uh endings of words such as Ed oring and so on and so forth so these become your tokens within a plain textt dictionary and the so there's a simple you know integer number assigned to these character consonant um combinations and there's several different ways to do this bite pair and coding is one of the more common ones but at the end of the day you end up with a 32k or 64k or 128k long list of a dictionary that are just numbers associated with letters now what you want to do is take those numbers and put them into a graph that are it's a multi-dimensional uh highly like highly dimensional uh Matrix essentially so imagine a cube but with uh you know a thousand sides and so what we need to do is find a path like you're solving a Rubik's Cube by um figuring out which layer and connectivity through each of those layers of this multi-dimensional Matrix that ultimately decides what the P um the distance is in this graph it determines what the next token that's going to Output so so first you take all the language available on the internet every plain text website out out there and they've aggregated those into you know multi multi-terabyte databases and you convert those into those tokens those integer numbers and you pass them through a graph and you know what the end state is you say I'm going to pass through this text and I want you to autocomplete it so I know what the second half of the sentence is supposed to look like and um the distance from what the randomized let's say initial uh weights on the graph are and what you expect the output to be there's essentially a cosine uh angular uh distance metric that you can calculate upon that so so these are how models learn okay so but at the end of the day your text gets converted into an embedding which is a path through this Matrix and the uh the weights that are saved in the model itself have been calculated billions and trillions of times um to kind of move things in a weird high dimensional space that's all you really need to know now to apply this in the real world you're going to uh let's say you have proprietary data and you want to get L talking about it well you're going to need to find which embedding model to use and there are commercial ones uh well first of all so this is a benchmark it's called the massive text embedding Benchmark and it determines across a series of categories of classification clustering and these um kind of Boolean operations whether or not um in retrieval and summarization things like that um it it categorizes them on those capabilities now what's important here is that you're going to see that most of these models are in the um billions of Weights which means gigabytes of space and this and this is video RAM that you need to occupy typically now if you look at the most capable ones they're taking 14 GB or 93 GB of of video RAM which is not practical for running uh locally and so you really need to go down to about number six or seven here and you see there's a 700 megabyte model that you can run to do embeddings and classification now when you do this the embedding Dimensions these this is how many sides of that Cube there are there's 1024 in this particular case and it can read up to 512 tokens of text at a time so you can at least double that so A th characters or so in your token dictionary are converted into this path through the model and it can do things like classification and clustering and and whatnot um so the point to this is uh for your later reference if you do decide to write an application using um a rag retrieval augmented generation approach you do need to kind of like analyze what's available you can also pay for a service if you want the larger scale ones now the real limitation here is that you can see these top four or five have 32k of text that they can classify versus the 1024 and that is a limitation um or sorry yeah 32k versus 512 but uh but those are things you have to work within if you want to run them efficiently now um yeah so I just said that so over the summer last year I was dealing um trying to learn about all this stuff and there's an API called text Ai txt aai and they have a really nice um API for developing rag applications so the first thing I did is I wrote a Discord bot that uh would essentially generate questions from GPT and the answers and then I would look the answers up on Wikipedia and bring in context of from the abstracts of Wikipedia for that topic now if I did this before adding that knowledge you can see like a 1 billion weight a 3 billion weight a 7 billion weight uh rnns here we're getting somewhere between 60 to 70% accuracy and when I added or I'm sorry the it was 20% accuracy and got up to somewhere 90% accuracy we got to keep moving so then I did a one that uh essentially you could do semantic search of Wikipedia so rather than have a keyword search you could actually ask a natural language question such as what's the name of the first woman appointed to the US Supreme Court and it would go out and it would determine that oh well that's Sandra de o Conor through uh semantic search and a and a custom embeddings model and then it would go grab the full text of the Wiki page and then it would generate its own summary and compare it to the human generated summary and so later on you can go look at that and the summary is um quite good uh compared to a human written one and then I made one last December so there was the whole lk99 uh superconductor thing in the news and we're all wondering like okay is this the new the new hotness and of course these models were trained before any of that was in the news so if I asked it about lk99 it says uh you know I'd be happy to tell you about it there's no info blah blah blah and then I turned on what's how do we augment our knowledge well we use a search engine right you don't need to make a custom database you can just hook it up to duck ducko and so I had it go search for the top three results for that and then grab the text from the web pages summarize those web pages and put that in the context and then after that uh it shows the summaries of all three of those web pages and at the end it says lk99 appears to be a hypothetical superconductor that's been making headlines recently yada yada yada so you can do this in a weekend yourself okay now how do we apply that to security well the first and most obvious thing that people started to do is apply it to decompilation and try to summarize what the behavior of a compiled function is and at this point there's probably a dozen or more uh implementations of this my favorite is from uh Tim bazaco uh also known as Mr Fraser on Twitter and he's a reverse engineering and deop fation expert and does trainings and things like that and he's come up with a very nice integration that will rename your functions for you based upon the actual behavior of the decompiled uh code and then there was another group that actually said well rather than use a decompiler why don't we try to see if the model can decompile on the fly now this is like asking a human can you read uh btes out of a hex dump and convert that to op codes in your head now some people can I can do a few um but this is not this is not actually the direction I think that we should be going in like we have concrete tools that do great decompilation have had 20 years of investment in them and there're the tools that we all use so what we really want is agents that will run these things and then analyze them but nevertheless there is a group out there trying to figure out um Can these models actually take a binary and output a decompiled code or from disassembly to decompile code and you know the gbt 4 if it's uh optimized level zero can decompile it to a 92% distance uh accuracy from I guess what would be considered good everything else is down and literally you know less than 15% accuracy like don't do this so that that's this slide is here to tell you that people are trying to do llm on things that aren't the right way to apply them now what is more interesting is when you get into the space of having uh domain specific languages or uh specific API sets that you want to train it on and get it to learn how to develop things like fuzz harnesses and so um if you're not familiar uh Google hosts over 10,000 open-source projects we're doing continuous fuzzing they've got some number of thousands of cores always fuzzing these 10,000 projects and uh at this point they contributed a uh infrastructure to llvm compiler where if you just pass a couple of flags you can uh Supply a C file that has a call back um that is defined and it becomes a fuzzer so you can take a normal library off the internet and write one function that calls those Library apis add a flag to clang when you compile it and now that will be a self-hosted inmemory fuzzer okay and so they scal this out into a project called OSS fuzz and you can win money by improving these fuzzers so this is a takeaway for you all you can make $5,000 by spending a weekend around with GPT or whatever to improve the harnesses or make a new harness for a popular library that doesn't have coverage yet and they've written now a couple of blog posts uh detailing how they're trying to do this internally and so um the main takeaway from this little chunk of the talk is that um if you write multi-page prompts with ex examples of the you know an example input and output that you want these are called multi-shot prompts and um essentially you know the more that you the more effort that you put into your prompting the better results you will get out because essentially you're priming a pump right you are getting uh the weights activated that are contextually relevant to your end question so the the better you can do at providing an example of what a true positive result is the more likely you are to have your question answered with a true positive as as opposed to a questionable possibly hallucinated result and so um this is an example you don't need to read it off the screen but this is part of the the the start of the prompt is you know you always give it a personality so your security Testing Engineer who wants to write a C++ program to execute all lines in a given function by defining and initializing its parameters in a suitable way before fuzzing the function through this particular uh function that you explore which is VM fuzzer test one input now this is like if you were teaching uh a young student how to perform the task you really need to be pedantic and detailed and so it goes on from there to actually give a bunch of real world examples with code and the results and then a set of rules around how to use this API ultimately I this blog post is like 10 pages long but the rules ultimately come out to be something like there must be at least one call to this uh API that's exposed to receive the fuzzy data um all variables must be initialized before referencing them don't name them the same thing don't use go twos and C code uh you know general rules that you would teach a young student um and so um followed by three examples of true positive results and at the end of the day um they did you know 1 1300 benchmarks across 297 projects yada yada there's ad Json format and then the end result is that this framework manages to um improve 160 projects code coverage for their fuzzing and the maximal points of 30% Improvement now if you yourself had improved one of those fuzzing harnesses by 30% you'd be making $5,000 that weekend so you know this is casual stuff you can do while you're learning about AI while you're learning about fuzzing and finding bugs um they'll pay you to do it so that's been going on and then on the academic world um Brendan goitz um MOX known online has been a head of a group that's been researching how to apply um llms to various ctfs and Bug finding and this whole category of space and there was a a good series of tweets where he talked about how he just took a random C file that was a gif file uh parser and um asked it to write a function to generate gifts that would achieve code coverage on all that code and uh it did it got 92% code coverage just from reading the C source of the Target and uh also by the way found four uh memory safety bugs one hang and five signed integer overflows so or signedness bugs so yeah so I mean literally found apparently 10 bugs um just from reading the C source and was able to generate test inputs that would allow you to uh activate most of the logic in the code so then the response to that on Twitter was something along the lines of um sorry this is just more data so um and then uh somebody was like well gift parsers I mean the rfc's out there there's 100 gift parsers of course these llms have read all the code available on the internet why don't we try something a little more esoteric and so they did vrml uh which if you're old enough if you're old yeah thank you yeah if you're old enough then you know that we tried to do VR in the 90s and that didn't work so well but uh but but also we know that when you try to Google now it's hard to find things from the 90s so these these uh models don't have as much access to that as before and um oh I'm sorry this is a different example so that's an anecdote uh the besides the Prov okay so besides providing a parser this time they just provided an RFC I guess and it got like 40% coverage 50% coverage something like that but also the vrml which which also performed kind of middling now there is a group out there um 06 sorry so this is still Mox and he is using agent-based systems to solve ctfs and he uh threw it against the Seesaw 2023 project which was uh an annual um CTF competition that was published after the models had been trained and the and gave it some function calling capabilities now when you train a model to do function calling what you're really saying is that you're going to get it to essentially give you a Json formatted um input and you're going to run python commands in your inference Loop yourself so you're going to run something that either uses python to um use pytorch or llama CPP or something like that you're going to have a loop where every word that somebody sends over the API is now converted into tokens and embeddings and the search through a model and um now you can you can become agentic where you have multiple kind of personalities or prompts that work in tandem um as it takes the input question and tries to solve or run different tools or things like that so so you train it to Output a Json format that are the input parameters to various set of commands that you define so in this case they have the commands such as run a Unix command or create file disassemble code decompile code check to see if the flag solution is correct and so on and so forth um and with a human in the loop it was quite successful and uh was able to you know benefit people using it and uh but without it pretty much the results are simply that gp4 is kind of the only capable tool in this space and you know there's some stats uh based upon what different categories and so on so forth and then there is the yes so then there's this group by uh professor Daniel King they've actually now come out with a series of papers I originally wrote this slide deck in February or March um so this is their previous paper they just came out with one a couple weeks ago that is trying to generate exploits and whatnot now what I would say here is I'm critical of some of these academic papers because in this particular case uh there was something like 12 challenges they were kind of arbitrarily chosen their blind squl injection um that it's very Corner cased and because in the academic world you have to find the problem space that your solution fits as opposed the other way around and uh so but long story short really what this just shows you on the slides is that again gp4 back in March was the only capable model now since then we have Claud Opus and sonnet and some things like that that are in the public but the the main point here is that self-hosted models are not good at doing code analysis generating exploits finding bugs or anything like that without quite a bit of effort and that effort does come in the form of developing your own data sets evaluation criteria and your own agent based systems now I've been using a new system from dread node that came with an example of how to beat Bandits on overthe wire. org so if you're a um a CTF player or a um war game player you should be familiar with over the wire used to be pull the plug way back in the day and this this is a simple challenge where you basically just need to know how to use unix commands um you know how to use SSH certificat and logging in so but you go to the web page it gives you a challenge description and it will beat the first 12 or 13 levels of this war game so it's not writing exploits yet but it is using Linux tools and stuff without having a tool-based specialized programming and I have a live demo of that but I don't know if I'm going to do it today so we'll see if we have time now let's say you do want to do code analysis all right I've already said two two or three times the GPT for and Opus are definitely the most capable platforms for that but what when we talk about code analysis and vulnerabilities and security we want to self-host and we want to bring this internally or we we at least want to build um you know uh systems that have a workflow where we can do some prior criteria or analysis before we go pay for an API right you can invest 10,000 today in a couple of gpus in a server and save yourself 50,000 over the next couple of years if you get some pre-filtering criteria into your workflows for doing this type of stuff so that's the motivation you may also be working under contract with ndas and whatnot and you don't want to send code out to open AI or Google so there is uh a whole Lane of code analysis tuned models and they are generally taken from a standard language model and then further trained on code bases and those tend to be the best like as it turns out codee analysis code models they're not really analysis models the the code completion models the models that are tuned for code they end up losing some capability in the general knowledge space and reasoning capability but are able to do things like code autoc completion quite well and so these models aren't really meant to find the bugs for you they're meant to complete your code in the most likely outcome of what everybody else has typed on the internet when they're writing code similar to you so there is a leaderboard where you can see how well these perform and um this leaderboard is this slide is specifically just the self-hosted available models and we can see that um the categories here are human eval python Java JavaScript it it breaks it down into individual languages and their success levels are middling at best you know somewhere between 50 to 60 70% uh accuracy not too good and you can see over time going from 2021 to 24 kind of we have a gradual Improvement but this is not an exponential growth we're not seeing a hockey stick and capability Improvement and uh at the end of the day these tests have uh these evaluation Suites have evolved over time there's now one that is EV Val plus and uh there's another one called mostly basic python programs but uh this one is showing the commercial models versus is um open source models and again GPT 4 uh turbo in Spring was leading the pack on um both human eval Suites and deep seek coder which is a model out of China um has a 33 billion weight model meaning that you would ideally want you know 48 gigs of video memory or so to run this um but if you run it at a 4bit quantization which is a compression you can run it on a 24 gig kind of 4090 essentially um so again the accuracy is still down at 75% level you need to have these models are best applied when you know the outcome you're looking for and you're trying to scale the the um processing but that you can confirm the output is accurate right because these accuracies are not where we need them to be um for trust level stuff this is the the Python Programming one coming up here kind of same idea um there's a couple of different models that are competing for the best Python Programming capabilities but again it's still below 70% and so this is why you need to hook these up to rag type systems that are using databases or other concrete information sources and allow it to use its general knowledge of the of the language the programming language but be able to reference things like API docs and and whatnot now so what I've been doing with this here's my contribution to the space currently is um you know we have a big problem in security where undocumented patches that are security relevant are out there in the wild at scale we often times at a patch is fixing a a security vulnerability but it wasn't documented as a cve right so I want to be able to look at any diff any code change and determine is this a security bug or not so as it turns out surprisingly enough uh I came up with a unique way to do this it's been done in Academia twice or so with limited databases but I actually went um from the direction of I know just through doing a lot of fuzzing and building tons and tons of packages from Linux distributions I know how Linux packages their um Deb files so on Debbie and and auntu you get the vanilla source code and then a patch set gets applied to it okay and so uh in that patch set if that happens to be a cve related patch the file name itself will tell you the cve that it's related to so I developed a ground truth database of 10,000 true positive uh security related diffs okay and so I was able to do this by essentially looking at the um all all cves that were ever uh associated with a Buu and Debian from the osv database and then getting the package name and then installing all the dros and Dockers and getting those package versions and then extracting the patches and so on and so forth so so I have a 10 double the best academic database for ground truth and evaluating these data to evaluate the models on their capability of determining is this a true pos positive patch so I show it a patch that I know is a security relevant patch and I ask it is this a security relevant patch and we expect it to say yes every single time um and so that's the evaluation criteria to get the Baseline of what are these models capable of doing and then that data set also is something that I'm going to be using for fine-tuning and showing it um you know making models realize that yes this is in fact a patch and improving the statistics so so I did that and then I started with what most people do simplified prompts of a sentence or two and then a and then grew those prompts to try to give it more information about what I'm actually looking for so I'm going to not read the whole thing off the screen but in general the you know review the diff looking for indicators of a security fix uh we're looking for things like modifications of memory management adjustment authentication authorization um alteration and settings permissions error handling reporting you know the the the types of things that you know when you go look at a piece of code that are in the back of your mind that these are the general things I'm looking for now in reality what I'm moving towards is a cwe specific prompt for every different bug class okay um but originally I was evaluating should I give it just the diff or should I give it the full source code should I give it the slice of functions that flow into this code change and so on so I have essentially a couple of prompts in the slides you can look at later uh one was for the patches one was for the functions and then I ran it across uh I wrote a test Suite that runs it across a bunch of models and the end results after you know months of doing this across dozens of models are essentially that um again these are March uh April kind of results but um mistal for public Al hosted API that's the left uh image here mral medium Claud Sonet and gp4 turbo were the only ones that had better than 80% accuracy they were the best performing publicly available apis now Sonet just came out with a brand new model literally this past week and it's outperforming Opus as well and I've the whole time I was really surprised but impressed with Sonet capabilities because they Nam these Opus Sonet Haiku based upon the scale of the model right and so Opus is the largest and takes a bit longer to do its inference and processing son's a bit shorter and faster but has a lot of great knowledge and of course you know hius are short and and quick um for simple kind of processing so it is interesting to me that their middle performing model um is actually the best at code by 10% on my stats and then also that they just updated it so I'm going to have to rerun these benchmarks again but then uh the M the middle U model or the middle column here is showing you that essentially llama 3 wrecked the whole um competition as far as self-hosted models go and even the well the 70 billion weight model in this particular case a six-bit quantization outperformed the rest but essentially the 70 billion weight model is finally competitive with some of the uh public open source ones and just to show you that I did indeed test other 70 billion weight or more models uh you can see in the right hand column the code specific models or models that have Miku was supposed to be um a leaked model from mixol for example but um you know these are single digit or less than 20% accuracy and so so llama 3 really is the model that you want to start with if you're doing your own training and tuning now as I said before I started with those single paragraph kind of descriptors of I'm looking for security bugs now this is what I'm inputting is I have fullon Json U formatted description of each bug class so and describing and and of course I use GPT to help me write some of this stuff but buffer overflows occur when data exceeds the buffer storage capacity overriding adjacent memory and then the steps that you would do yourself mechanically to analyze code for that bug class and so you know identify buffers uh especially fix sized buffers analyze the input functions that flow into those buffers including things like string copy Sprint of Etc and and so on and so forth so you now you get detailed uh descriptions of each little type of thing that you would do in your own mind behind the scenes and not only that but I have a friend uh OG hacker Raptor um who wrote All these srep um patterns and I was like you know what would be great is if you wrote that for wegley and the main reason is because wegley doesn't require compilation it uses Tre Setter and so I have over 200 static analysis uh patterns that I'm feeding into the model along with the bug class descriptions and so number one it can read the patterns and it can some somewhat understand Rex and then also it can get the output from wegley and run it directly and then also have the description of the bug pattern so we're refining and getting more detailed in what it is that we're giving the model and therefore you can expect better output okay so let's say you go down this path you want to you know do hacker with models well there is a threat landscape involved here because this is a rapidly moving and very immature software space and ultimately if you use if you self host you're either going to be writing uh python again using pytorch probably used to be tensorflow but tensorflow is really kind of out of voke like nobody's using tensorflow anymore um you're generally going to be using pytorch or you'll probably you use a python wrapper for uh llama CPP called llama CPP python or you may write native code and rust or something and it'll still be llama based uh CPP based kind of infrastructure so my old I used to work at tals Cisco tals I created the uh vev Team there and my colleagues that are still there actually did find one of the first format bugs in the model format that is loaded by llama CPP so when you download llama 3 you can download it in a pickle format you can download it in a GG UF format and in that there was a file format bug of course generally the way these formats work is there's uh metadata at the top that describe the layout and um the geometry of the graph essentially that's held inside of it which is again this multi-dimensional Matrix and so at the top though there's something that's going to say that there's this many layers and these are the dimensions of those layers and maybe it'll say um you know we're using uh 32-bit float or 16bit float or or whatever type of quantization and then the inference engine loads in those weights into video memory and knows how to parse and read through that graph and so anyway so that's a file format bug this would have been something that I could go on hugging face which is essentially the GitHub for doing llm stuff I could post a model I could then go on Reddit and say hey check out my new model you would download it and you would get owned I mean that's the attack scenario right and uh I wouldn't normally be a person that advises people go to Reddit but as it turns out but as it turns out uh in the llm space essentially uh Reddit machine learning and uh local llama are two subreddits that are highly active and really how to keep up with the intense churn that's going on here so anyways because of that you can find unsuspecting people to hack through these types of attacks so so obviously file formats are an issue and uh when these bugs are found the response from the developers are less than uh ideal so uh here's another bug that was in the server side infrastructure so this one is if you're hosting a model and you have an application using it and you don't secure that endpoint then other people can connect to it and send via Json data some configuration to that model that will result in vulnerabilities and so in this case there was a directory reversal in an upstream package it wasn't even in the code that all they had to do is pull a new code and the response was yeah we eventually need to update that uh I kind of thought this was going to happen uh unfortunately I'm really busy so and it's only on windows so who cares you know uh so as it turns out I also so uh in the process of doing my code analysis work I as I said I was forcing a grammar to make it do a yes or no output and to evaluate the prompts or maybe I said that or not but in doing that well whenever you have a grammar parser in a server side thing and you can supply a grammar well I'm a fuzzing guy to me that is like you know I gota I got to poke at it and so sure enough I poked at the grammar parser and found multiple remote memory corruption bugs integer overflows whatnot and as it turns out Mozilla has forked llama CPP into a project called llama file it's actually a very interesting project um where I don't know if you follow this particular researcher um but there's something called Cosmopolitan libc it's a crossplatform cross architecture portable libc and so essentially uh Justine tunny is the author of all this she binds a loader to the model itself so you have one 4 gig file 8 gig file whatever it is and that's the model and The Interpreter all put together and it can run on a Mac OS Linux OS Windows Etc so I submitted these bugs to Mozilla trying to get cves for them um because again the high churn and the the ecosystem isn't they're not really up to security um General workflows like they don't know how to get a cve this one random dude Georgie goov or whatever wrote Lama CPP he's an excellent programmer but this is not a product you know so uh so anyways Mozilla did Fork it and it is a product from Mozilla so I submitted them to Mozilla and um long story short this is just showing it got patched back in May I submitted sometime in April it took a few weeks uh but we got that done so what this was though is that in multiple different ways in a HTTP post request to the Llama CPP server you had several memory Corruptions depending if they were using the GPU acceleration or not um also the the comment here is actually not fully accurate it reduces it down to being like it's an unclosed quote but but there there was quite a bit of flexibility in what you can do and these are persistent servers and so this this I believe would have been exploitable um so so I did mention that if you're not using the native CP C++ parsers you might be using Python and pytorch which is from Facebook and everybody's using pytorch because it has kind of a a more comfortable API it's more pythonic it's it's more modern than the the tensor flow older stuff but unfortunately uh because it's python we know that pickle files are deserialized Json and data formats but they have um they're actually objects and they can have initialization functions when you load them so if I upload a pickle um version of a model when you load it I can execute arbitrary code on your machine by Design um in the P torch infrastructure so somebody finally realized this you know a few years ago and said hey maybe we should make a format doesn't do that and so so they made something called safe tensors and safe tensors are um just a custom format that isn't supposed to dilize now then they got real smart they're like well why don't we just go ahead because uh hugging face is literally GitHub infrastructure based why don't we go ahead and make a GitHub bot or git bot that goes out and sends PRS to automatically update your pickle models to the safe tensor model that seems legit okay well now here's what happened though um in the process of doing so so they have an agent that goes out and makes a Docker and uh if you accept their PR or actually without a PR automatically goes out grabs your model converts it and then pushes a PR to you well in that process of course in that Docker it's still vulnerable to the pickle vulnerability so it's opening your you know attack model and of course because this is an agent and it's built in a distributed automated system there are environment variables holding tokens that are security tokens that are authenticating the agent yes okay some of you are following so yes so what ended up happening is that somebody made a model that had a pickle payload that grabbed that environment variable and became the agent and so they could send PRS to everybody on hugging face and uh do whatever they wanted in your in your repost so so yeah so uh so this did happen um it was reported before it became super malicious but they did have something like I believe 100,000 or so PRS actually accepted uh that were non-malicious but as a proof of concept and we're talking Google and Microsoft and you know because there's millions of models out there we you only hear about a couple of dozen of them that are that cost $10 million a piece to even start to create uh for the self-hosted ones so that's why only meta and and uh you know Microsoft and a couple of companies Moll are are putting those out but there are all the academic works are all on hugging faces as well so there's millions and millions of models um and so yeah so if you become GitHub or hugging face obviously you wreck the threat landscape and so on so I think that's I don't even know what time it is but I think we're close to being there and yeah so uh I'm going to in conclusion before I get to my final slide um the CTF I run the CTF I'm rich in Seattle uh it is live there you should go to the wiki check it out thank you uh it's a very interesting CTF this year totally unique from last year there are um there's one pable challenge there are RF uh audio visual Transmissions there when you go on board your shady bucks uh tag take a look at that tag uh so on and so forth bureaucracy is in full swing now as well please come by I'm also part of bureaucracy and also if you come by a bureaucracy you're going to see a dope ass robot and uh by tomorrow or so I'm going to get it talking to you so it's going to be a fully interactive audio visual speaking robot and the stretch goal is that there might be a CTF challenge in there as well um also uh in 2018 at tour tour Camp I launched undercurrents doio which is a full ansy BBS that is fun to use year round but also maybe dropping some Clues there for the CTF and I don't think we're phones online yet but we have a few phone numbers you can reach me at or us at which is 4 CF is the line please leave me a voicemail call for fun I don't care um dial beer if you want to talk to bureaucracy and uh I think we'll have a fact uh four facts is our fax line and uh with that just uh final comments here is number one if if you've only use chat GPT or the public web apis I highly encourage you to uh actually use the apis not just the web interfaces you can in fact for free and don't confuse this because musk has the grock with a K uh AI there is grock with a Q it's a custom Asic created by the people that created Google TPU the tensor processing units it's actually now uh lpu uh language processing unit I think um but long story short a a new company is out there and they're giving you free inference on the largest self-hosted models so grq uh you can get a free account right now and have unlimited uh like free access to 70b 70 billion weights llama 3 for example and this summer in theory uh there's going to be a three or 400 billion weight model coming from meta and this will be the first time that we're going to see an open source model compete with what the commercial products are really out there so um be on lookout for that that is supposed to drop the summer and I believe that will be on grock and it might be when they start charging but regardless use the apis they can be done for free Google collab also lets you write jupyter notebooks and run them for free and 16 gigs of uh video so for testing out software things like that there's so many free resources and hell if you want to run some fuzzing machines who says you can't do it in a jupyter notebook but uh um so so so get involved um pay attention of course to uh meta I would say the Llama series and the mistl series are the two more interesting ones that are self-hosted um you know install a plugin and use it in your IDE and and you will accelerate your capabilities um and then if you're really wanting to do real work though get into agent systems uh write descript put effort into your prompting and you will get better results out now these are not knowledge engines in in the sense that you can expect true facts all the time but they are able to deal with language and if you hook it up with true facts and a language interface then you can talk to it with language it can grab the true facts without you writing SQL queries and things like that um and also look into structured output so um I'm using the BNF based grammar formatter in llama CPP primarily but there are other things like um Jason former and um gorilla and there there's a number of systems out there um I would also say that the one that's not well known if you're going to write an agent system is this dread node.io it's two security researchers they run the AI Village at Defcon um it's a very lightweight system it's not like Lang chain where it's the kitchen sink it's actually a a very nice kind of like if you write a a video game you have a a event Loop and you have objects and actors and actions and and things that think can can happen and this is architected in a similar manner so it's it's really really nice to to work with an agent um and so yeah so we're on the cusp I mean I don't think we're there again we're in the 70 80% accuracy range so you can't just ask a model one sentence to solve The World's problems but if you put a the effort and you can make a do work for you so thanks for your time uh thanks for coming thank you and uh happy to answer any questions yep sorry Alternatives with rag systems alternative with rag systems so uh so yeah so if you're going to get into rag I would absolutely recommend the txt aai API to start with and then um when you start doing this what you're going to going to see is there's this whole world of vector databases right these databases are columnar databases that are designed to look at arrays of floating Point numbers which are what these embeddings actually are now what I would say is avoid this entire new SAS World um and actually stick with the things that you know redus and postgress have Vector extensions in fact PG Vector is the post grass Vector extension and it was just uh improved in the last week or so for uh like Cloud scale um remind me your SC time scale yes so PG Vector time scale i i i in fact had read about it and then I just uh met up with David last week and he was bringing up that his friends involved but um so PG Vector was the original postgress Vector database extension now PG Vector time scale is actually Cloud scale um distributed version of that so I highly recommend not getting distracted and if you do use one of these SAS uh even if you can self-host it they all have Telemetry in them every single code base I've read in the AI space are trying to put Telemetry into your code the whole point of doing self-hosted model inference is to avoid sending data back to some Mothership so so yeah so do look at the code if you're going to run it because literally I've caught three different projects that I was thinking about using and you know I'm not trying to be pedantic but don't put Telemetry in my code make it default off or something or make a prompt like I shouldn't have to read the code to realize they're sending data back but but yeah so so my my general advice is don't worry about the Alternatives look at PG Vector time scale and at redus vs store extensions any other question yes what's what's the structure of the safe tensor file format such that it doesn't have the issues that pickle has oh that's a good question uh so I'm going to not answer that question and tell you what I was going to do instead uh so so so the solution to this whole problem actually is model signing and I actually propos this as a work project but I don't think we're going to to pursue this path and Google just came out Google or meta in the last three weeks or so came out with kind of a large scale effort that they're going to go across this but but the the end solution if you need to solve um a supply chain issue is to have verification involved right and trusted entities that are supplying that information or or data and so the real solution is to have a party that will take the meta and uh Microsoft and various other models make sure that they're up to Quality run a benchmark against them sign them saying that here's the number one this is trusted data number two here's the results this is the version that did it you know build in all of this kind of like uh metadata around that model and then simply Fork the public infrastructure of pytorch or llama CPP and add the signature verification into that so I I propose that we're not going to pursue it so I'll throw that out there for anybody else who wants to do it yeah any speculation or guesses as to why the medium-sized model did better than the larger ones for the code analysis challenges you well unfortunately the so you're talking about like Claude Opus um and sonnets being better at code um generally they don't publish the data sets that they you know are training on so it's it's hard to infer um so General I mean I I don't have any confidence level of guessing I would just be guessing but but long story short that that's another benefits of understanding uh um how models are trained in the open source space is that you generally are getting some understanding of what they've been trained upon and also and you know a lot of the security context and model hacking right now is this prompt injection right well we're Shady tail we're phone freaks we got we know a little bit about inband signaling right so obviously this is a classical problem of not having a separation between user input and the data processing or no validation so uh the whole prompt injection is just kind of and as it turns out uh the post fine-tuning steps that remove some of the ability to say bad things or you know read you the Anarchist Cookbook or whatever uh is reducing the capabilities of the models themselves and so what you really do want the raw models and eventually I think as compute becomes cheaper we won't want the fine tune models like right now you want the instruct models because you can ask it questions and not just do an auto Complete because for an auto complete you need to have a lot of prompting you need to do the full on here's examples of inputs and outputs and now you now I'll ask you a question now you autocomplete it so you would have to spend hours to create a good prompt for a base model but the base models are more capable than the instruct models and as they continue down this brand protection is really what uh prompt injection is all about is like meta doesn't want their chatbot talking about building bombs um because it's brand protect it's it's simply marketing um it's reducing the quality of the models and at some point you'll be able to quickly fine-tune your own models within 24 hours to get it to only say the things that you want but but yeah in general uh as far as the why mediumsized models are better than larger models the best guess would simply be that they the larger models are overtuned for behaviors that are reducing some of the capabilities that are inside it yep uh I said you had a big database were you using that just to test uh that the CV were found by the LM or did you like make an embed what was the question okay yeah so so the question was with the uh the 10,000 cve patches that I've extracted and have true positive in my database um was I just testing uh to see if those patches were security relevant or using it for fine-tuning I think so yeah so the reason that I uh on the slide where I was showing this performance metrics says I've been waiting for a model that's worthy of spending the cost to do the fine tuning now you can do fine tuning for hundreds of dollars or maybe $1,000 or something it's not going to be in the millions or tens of thousands of dollars so it's not a huge cost but I do eclip sium a startup so we do have limited resources so so Ive I've been building this data set with the intent of doing fine-tuning um code llama based upon llama 3 might be a good basis but yes so it is in preparation for doing that and also trying to layer um classifier models like smaller models that are very fast that you know just rejecting certain things before it gets to the heavier weight processing so it's an ongoing project I mean this is kind of the timeline is at the end of this year I I well this summer I'll be TR I'll be tuning models but by the end of this year I should have some good results maybe a hushcon or something like that all right let's than the speaker all right thank you everybody he [Music]