NorthSec 2025 - François Labrèche - How not to do ML

Uh so welcome to my talk everyone today on how not to do machine learning. Uh so today I'll be showing the negative impact of improper CV feature selection when doing a live exploit prediction model. All right. Real quick about me, Logan did introduce me uh very well. So I'm the research lead for the XDR threat prioritization team at Sophos. Uh I have a PhD in computer science applying machine learning to cybersecurity problems. And for the most part what I do is I make mistakes and I present them at conferences. All right. So the agenda today uh first off I'll show the context of of what the prediction model does, what's the problem we're trying to solve. Uh I'll give an overview of the actual model itself, but then I'll go into a number of critical issues with the model, the data, how we encode features. I'll show a revised model and then some actual predictions the model did in April. And then I'll show a bit how we can apply the same model to other threats and conclude. All right. So let's start with the context. So here the background is really vulnerability management. Uh so we're trying with we now have over 40,000 vulnerabilities being published yearly as of 2024. But out of all of these vulnerabilities only a small minority actually have exploit for them. And even fewer are actually uh attacked uh used by attackers. So here there's a benefit uh there's a use for machine learning in creating a a predictor that can predict ahead of time of all the new vulnerabilities that we see every day which ones will have an exploit or not. And there can be many benefits to this. Uh we can it can help with identifying the true priority of vulnerabilities. It can help with vulnerability remediation essentially knowing which vulnerabilities to patch first. And lastly it can also help with uh tracking threats uh knowing where to focus your uh research. But with this comes a problem where predicting exploits with the data we use presents a unique opportunity to incorrectly encode features in multiple ways. And what's tricky about that is that these issues mostly happen when deploying the model in a live environment doing real predictions. So they're they're more sneaky to catch let's say. So essentially a model that is built using incorrect features can have a drastic impact on the true performance. Here true performance meaning when you actually want to use the model in a real real world scenario. All right. So the focus of the talk is not the threat prediction model itself but yeah I'll go I'll do an overview of the model. It's mostly it's essentially based on the past literature but yeah. So the methodology is encoding features built from CV metadata and open source intelligence. We train a model and we do predictions. So it's a pretty straightforward supervised machine learning problem. Uh essentially here in this approach the model algorithm itself will be agnostic. So we don't uh it's we're not looking at specifically which model will perform better. We're looking at the feature encoding and we'll have a model as a benchmark. So but yeah we do use for the people that want specifics we do use a random forest classifier. And here the the two last tidbits the balanced class weights and threshold moving essentially is just to uh uh fix the address the class imbalance. Here class imbalance essentially meaning we have a lot more vulnerabilities that do not have exploits that do have exploits. Do some quick fixes for that. Uh so yeah we encode features using CV data that we all know. And we'll evaluate the model using 10-fold cross validation which is basically the standard of evaluating models. Uh for those that do not know 10-fold cross validation you know I'll go into more details later. Uh and then we train a model we can apply it to all new vulnerabilities that come out. And we can know which vulnerabilities will have exploits. We'll patch them right away even before an exploit comes out. We're good. All right. So this is the overview of the model. So there's a lot of text here but for the most part here on the left in blue we have the vulnerability details. Uh that's things we all know from CV reports. We have the description. We have the vulnerable products. We have the list of references. Uh we have the CVSS score for the vulnerability. We have the publication and modification dates. And lastly we have online discussions here at the bottom regarding the CV. So everything is public open. And with this information we encode a number of features here in the middle in green. The first part at the top uh which is we see that we encode the description as topics. I'll go into more details for this just in the next slide. But it's how we encode the description. Uh we also look at the description length and number of word as a feature for the model. The number of vulnerable products, number of references. And then we can simply take the CVSS features as is for for features for the model. So the score, the different CVSS features. Then we encode the dates of the CV, the publication modification modification date. And lastly we look at the number of online mentions. We encode this for the model. Now there's more fancy things we could do with online discussions. Here we really only want to have some signal of is the disc is the CV being discussed or not? Is it trendy or is there no discussions on the CV. And with these features uh we add the label which is basically is there an exploit or not. And then we train the model on historical data to get a model that we can use on new vulnerabilities. All right. So relatively quickly let's go into the uh topic modeling for the description. So essentially the CV description is text we have to encode it in some way. There's multiple ways to do this. Uh in our case what we use is something called LDA latent Dirichlet allocation which essentially it's a uh clustering algorithm for text. So you give it all of the vulnerabilities descriptions that have existed so far and you train a topic model. Essentially it'll group all vulnerabilities into 30 global topics. And the good thing about LDA is that you can see these global topics just by representing them as word clouds. So here we have three example word clouds and we can quickly see which vulnerabilities uh they refer to. So in the left we've got denial of service vulnerabilities. In the middle we've got cross-site scripting vulnerabilities. And lastly on the right we've got WordPress plugin injection, SQL injections. So for the most part with this we get a vector of 30 values for each vulnerability where each value is how relevant this specific topic is for the vulnerability. All right. And now the label we want to predict is a binary value. So essentially is there an exploit or is there no exploit for this specific vulnerability. So we only look at prioritizing vulnerabilities. So in in this specific scenario essentially if there's one or more exploit we'll want to prioritize the vulnerability. So we don't look at if there's multiple exploits uh or what type of exploit. So zero exploit or one or more exploit. The data used is the data from the NIST National Vulnerability Database for the CV details or the description, the dates I mentioned. For exploits we look at Exploit DB, Packet Storm, uh proof of concepts on GitHub. Now these are not all exploits but it's enough exploits that the model should be able to perform. And for online discussions we look at dark web forums with our partner Flare. Uh we look we crawl Reddit uh Reddit posts mentioning CVs. Uh we look at Pastebin Telegram chats. With this with all of this data we encode the features and then we can train the model and do the 10-fold cross validation to essentially know how the model is performing. So essentially this is the standard way to evaluate machine learning models. What we do is we split all of the data into uh 90% used for training and then we leave 10% off for testing since we will train on the 90% applied to the to the 10% left we'll know how the model performs and we we repeat the process with a different 10% for the test we do this 10 times and we aggregate the results. Essentially with this you'll know globally how the model performs because all the data has at least once to train and to test. Here are important metrics to look at. So some of you might have seen how we evaluate machine learning models. In our case we'll look at the accuracy which essentially is the overall performance of the model. How many things did it get right or wrong? And we'll look at the recall as well. Now the recall here is essentially the ratio of exploits correctly predicted. So specifically exploits. In our case we want to know want to predict exploits more than things that will not have exploits. So this is why we look at the recall. And then we do all this when code the feature we do the 10-fold cross validation and we train the model we get these results. So we can see we got 93% accuracy. That's a good number we like to see. We have 83% recall which is also good. For the most part the model uh predicts things correctly. Here I also have precision and F1 score just to have the values. The precision essentially means that we'll have some false positives but for the most part the model performs well. We can deploy it to production company will be happy. Now if we do this we can see that the actual live performance drops drastically. Actually it's we still have 92% accuracy but we can see that the recall is down to 2%. Now essentially what this means is the model is predicting every new vulnerability as not having an exploit hence it still has good accuracy because most vulnerabilities don't have exploits. Uh but the thing we're actually using the model for is to predict exploits and it's not predicting almost any exploits. So essentially this model is completely useless. Uh so here just small note to do the live performance what we do is we run as a daily batch job. So we have the model trained and we apply the model to all new vulnerabilities from the past two weeks that don't have an exploit yet. And then we can evaluate the live performance because we do see over time the exploits that do come out. So we know when it's right when it's wrong. All right. So what happened with this model that we deployed to production? Well essentially future information was led in the training of the model through the feature encoding essentially inflating its performance on historical data making the model appear really good on paper and actually hurting it when you deploy it to new vulnerabilities. Um and rendering it essentially useless. So so for the following part of the talk we'll explore what actually led to the poor performance and we'll fix it we'll fix it along the way. So the first issue with this model I presented is building features from CV publication and modification dates. So here essentially when you encode features for historical data the model can use this because there's trends and types of vulnerabilities and new vulnerabilities come come into play at some certain times through time. But essentially with new vulnerabilities the publication date will always be today. Essentially the model has never seen that date every time we apply it. So it's just hurting the model more than anything else. And here the solution is pretty easy. We simply remove the features from the model. If we do this and we redo the historical evaluation we can see that we still have relatively similar results. The results drop a bit but it's still a good good enough model and if we deploy this to production now we can see that the live performance uh we now have a 300% increase in recall to 6%. Now it's still completely useful useless but it's a step in the right direction. Which brings us the second issue and this one is only specific to the evaluation and it's mostly there for anyone doing machine learning in cyber security. But the issue here is doing standard 10-fold cross validation which by default does a random split of the data. So now most most of the time this is perfectly valid to do in machine learning but with cyber security you get new types of attacks that come appear over time. It's an attack defense scenario so you cannot give future data to the model. So for the most part here with the normal split it's the cross validation I mentioned before. And for the fix what we do is what we call a time series split. So essentially with time series splits you will always train the model on data that is older than the one you're using to test its performance on. And this is what we see. So the training data is always older than the testing data. And if we do this essentially the evaluation will just be closer to reality. So we're not deploying a new model but we can see that the performance on historical data dropped a bit more. We're still not nearly close to the live performance we got before. Which brings us to the third issue which is building features from online discussions. So here when we encode the features the online discussions from the past essentially it's a cumulative sum of all the discussions that happened or maybe the peak of discussions. But with new CVs that come that come out maybe the discussions have just started maybe it's trending over a couple of days maybe a couple of weeks maybe even a couple of months when the the CV is actually used in an attack. So for the most part when we apply the model to new vulnerabilities the number of online discussions won't match what it's seen in the past so it won't be able to do correct predictions. So here again the solution is removing the features. There's ways to do this with more work into encoding into historical data but here we'll just need we just want to know what the impact is so we'll just remove the features and see how it performs. So here I have the results for the historical data. So we can see the accuracy slightly drops 1%. It's mostly performing similarly. We've got a higher recall even but lower precision so a bit more false positives. And if we deploy this model to production we can see that now the model starts to perform better. So we've got a great increase in performance. The accuracy is at 88% performs well and the recall more importantly increased to 42%. So now the model is actually starting to is actually usable. It won't catch everything. It will have some false positives but we still have 88% accuracy so it's better it's actually predicting correctly. And bear in mind also when doing the exploits might still come out maybe weeks later than when we assess the performance so it's a bit under representing the performance but gives an idea of how the model performs. Now there is still one issue left and this one is the trickiest issue of them all is and this one is building features from CV metadata that is actually not available when the CV and here I've got citations of approaches that use this including my own. And the reason for this is that CV metadata is generally the most information you can get on a CV so that's what's being used to do prediction models. But actually it was shown by Can Mamadi that CV metadata is actually not available most of the time on the day that a CV publication happens. Most of them you won't have the CVSS score. You will not have the vulnerable products. You might have one reference that will be the advisory. So essentially you're trying to predict on data that is not there. So here the only guarantee we actually have is the vulnerability description. So what if we built a model using only the features built from the vulnerability description? If do this on historical data, we have similar results. Okay, that's good. But then if we deploy this to production, here on the right, we can see the new results. And now the model starts to suffer. The accuracy is close to being random. Uh the precision is low, but surprisingly we do Well, not surprisingly, we do have a slight increase in recall. So, there's a couple of vulnerabilities that it can now correctly predict because it's not using the data that is not existent. All right. So, I showed a stronger CV metadata model. A model that can be used right away on day one. So, what do we do with all this? Well, the features that aren't good, we just remove them. We scrap them. And now what do we do with the CV details that we might might not have? Well, essentially what we can do is combine both model into a single framework. So, when we see a new CV that comes out, essentially we know if there's a CVSS score, if there's vulnerable products or not. If we do have the CV metadata info, we can simply use the stronger CV metadata model. If not, we'll use the minimal model to at least get these quick predictions to know if we need to patch something right away or not. And that's what we did. And we've deployed this in production. It's been running. And I do have some noteworthy predictions. So, it's not predicting everything every exploit, of course. But these are some of the predictions in April that the model did get correct. So, these are exploit predictions from April. So, we've got the Gladinet Center Stack Deserialization vulnerability that was added later on to CISA's known exploited vulnerabilities. Got a SQL injection. NTLM spoofing that was also added to CISA. And lastly, Linksys RC vulnerability from April. So, this is just a subset of some of the vulnerabilities it does get right to show a bit the the the uses for this model. But there is also extensibility to this model. So, we've been predicting the publication of exploits. But we can also just swap out the label and use something else and do the whole same the same feature encoding. And we also did that by using CVs used in malware as a label. So, essentially here what we do is we look at CV at malware signatures that reference CVs and we use this to know if there's a malware using CV or not as a label. And we did the whole same thing, same approach, but predicting this. Now, the same issues arise when you're trying to predict malware that use CVs. So, here we have the initial features as I showed right at the beginning of the talk with before the four issues. So, we've got very good results, 99% accuracy. We've got 85% recall. Here we have a very high accuracy because it's very imbalanced. So, almost all vulnerable Not almost, but the vast majority of vulnerabilities aren't used by malware. Um but here if we apply all the corrections I mentioned, we can see that the results drop. We still have 99% accuracy, but the recall drops to 54%. So, on paper the model appears to have lost in performance, but in in reality these corrections actually leads to better live predictions. So, these are some of the successful predictions the model did for malware uses of CVs. So, we've got a buffer overflow in Ivanti Connect Secure. We've got a couple of use-after-free vulnerabilities got correct. And RC in Tomcat Tomcat. And lastly, we've got a VMware out-of-bounds write. It also predicted correctly that would be used by malware. So, we can see these types of vulnerabilities are very different than the ones for uh exploit publication. We've got a lot of use-after-free buffer overflow vulnerabilities. So, just this is to show like depending on the label, the model can adapt to the feature encoding. All right. So, essentially this talk it was shown how four feature encoding and evaluation errors can actually inflate the evaluation of a model on paper and not translate at all to live performance. And two methods for optimizing successfully the live performance of exploit and malware prediction models were shown. And I've shown a couple of these types of predictions the model can do can do correctly to then help with vulnerability prioritization management. And yeah, these are the references that were throughout the the the talk. And yeah, that's it. Thank you.