Security on Autopilot: Low-Touch Automated Security for Drupal Projects / Other / Nathan ter Bogt

Kia ora. Uh good afternoon everyone. Uh welcome to my talk about security on autopilot uh and how we can set up automated security testing uh set it up once and use it going forward. Uh if that if you think that means that we don't need to do any work, then you're in the wrong talk. Uh I'm going to start with some security myths and why teams like to throw them around and the reasons that I completely disagree with them uh and let's just have a look. Uh we've all heard it, we'll get to it later. Be honest with yourself. You won't. You likely have put off your functional tests as well. Uh did you get around to writing your functional test? Uh it'll slow down delivery. It won't slow you down, you're just deferring the time investment. Uh doing things further down the development process is generally more time expensive and doing it when you have a security incident is definitely more time expensive. Uh we're too small to be a target. Nobody is too small. Attackers have automation too and the automation doesn't care how big you are. Uh and finally you passed a penetration test. That's great for today. Tomorrow there's another 10 CVEs. Uh so what do we want to do about it? We want to look at ways to automatically protect ourselves uh and respond to incidents as quickly as we can. So what we want to achieve? We want to quickly identify the issues in our application and our dependencies. And we want to keep our application dependencies up to date. Uh we want to pick up issues early in the development process so that they don't make it to production but just as importantly, we want to pick up pick up issues in our running environment. Uh what does success actually look like? That we're notified by an automated system that a security issue exists and it's in which environment and how long it's been there and the risk level of that incident. Uh fingers crossed, we've already had a pull request created by an automated system. Uh and it has already run all of the local tests, code quality, functional tests, etc. Uh and you're just going to have to look at merging it. Uh and we've limited Uh we've kept our application dependencies up to date, but that's to limit the footprint of a PR for a security fix. So, we want to know we have a security issue by scanning our application, not by an incident. We want to know well before any incident, and we want to know that it's fixed and available and ready um before it even gets to that. Uh I'm going to refer to Formula 1 for a little bit uh to help give a bit of context. But, in Formula 1, the FIA has a process called scrutineering. The car's placed in a purpose-built booth and 3D laser scanned and lidar is used to see if the car dimensions are to spec. Uh they check the weight of the car, and they check engine engine displacement and things like that, and make sure it's all up to code. And they do these checks while the car's static. It's not moving. That's what SAST is. So, we are doing all these tests for our application code, our dependencies, and our runtime environment while not running the application. Uh we're specifically scanning the the assets we have and not doing a dynamic test against the application. Uh we have three different layers of that test of that testing. Uh the first is the application code, then which is our code, there's the application dependencies, and then there's the environment that we put our application onto. Uh the application code. The application code is your code. We're assessing your application's code for possible issues. Our automated tests, uh tooling like PHP Stan, can pick up things like poor error handling. Uh and that can stop an attack before it becomes an issue. An example would be say you don't respond to a exception properly in session authentication. Uh additionally, if you see a common issue in your team, uh you can automate the detection of that to save on your code reviews and uh just the time taken to get code to production. Uh an example of that is your team uses a lot of direct query APIs in database for Drupal, but you should really be using select and update so that you don't have SQL injection attacks. Uh I consider PHP Stan to be foundational and the first thing that you need to implement it in an automated security testing setting. Uh the reason I say that is as we start to automate it dependency updates, uh and application like runtime updates, PHP Stan can pick up things like function definitions changing but or the function used to throw an exception instead of a boolean, all that stuff. So, you're getting that feedback as part of your build system and your workflows. Uh, we should run PHP Stan on every single pull request. Uh, as I said, it's picking up issues early uh, and making sure they don't make it to production. Uh, uh, but also on integrations to your main branch, so that you're testing that as lots of work is happening. And while this is not really a CI CD task, a code review is also an essential part of the application code checking. Uh, we've had great success at automating parts of the code review using Copilot and Claude uh, to help reduce the load on our reviewers, but not remove the load of our reviewers. Uh, allowing the humans to pick up architectural issues that could create something like a DDoS vector uh, or catch edge cases for like a login bypass. Uh, you can automate it, use AI, but you still need your humans to do that overall architecture. Uh, and I will put in here, you also need automated unit and functional tests, uh, but that's a whole different talk. Uh, the next level is your application dependencies. Uh, we have tools like Dependabot that help you manage your dependencies, uh, but this is about protecting yourself against someone else's code. Uh, this is scanning your dependencies, doing maintenance and management of updating them. Uh, regular maintenance PRs that upgrade your dependencies are important because if it doesn't update cleanly, you have time to fix it. You don't want to do that when you have a security incident and you have to upgrade 200 packages and all their issues to fix one security bug. Uh, something that works for me, uh, and just a little win is that if I have a bug in a specific dependency update then I'll pin that version I'll commit the chan- changes and create a task, like a Jira ticket, for unpinning it. The reason for that is if you do that, the automation keeps going. All your other dependencies keep upgrading, you keep getting pull requests, things aren't breaking and you have time to come back, fix the pin version, and put it up to the whatever you need to. Uh we should run the maintenance workflow against your main branch at least once a week. Uh but in our case we run it on like what might say Oh, that one's daily. Uh but on some projects we run it weekly, but we'll run it on Wednesday morning so that we have time to do it during the week. Uh, another thing on here, this is specific to Dependabot. But this is a new feature they added in, cool down default days. Uh, that'll protect you from a zero-day exploit. It'll only give you updates for something that's at least a day old. Uh tools like Dependabot, uh, and Renovate, and there's a couple of others uh, will send you a pull request when there's a security issue. So, this this is dependency maintenance, but when you actually have a security issue, you'll get that pull request within a couple of hours of when that's released. Uh because we've been taught on top of all the maintenance upgrades and making sure we merge them and looking after all our dependencies, it should be a small, easy, low-impact task to review that security update and merge it. Uh the tools so far go hand in hand. Without the maintenance pull requests, without PHP Stan checking our code, and without a solid unit test suite, security PRs are going to take a long a lot longer to test and a lot longer to confidently merge. The third stage of our static testing is the runtime environment. So, your application can be super secure. You've done everything so far. You've done all your dependency management. You've done all your PHP Stan, etc. But, you put it on a Linux kernel with a TCP vulnerability. And that's taking your whole site out. Uh you're no better off if you don't also test this layer of your code. So, here in our case, we're using tools like Trivy. Uh that will take a Docker container and scan that for all installed versions on that container and give you a report of, you know, OpenSSL has an issue, Engine X has an issue, all that kind of stuff. Uh it should be run again for all your pull requests and all your merges to the main branch, same as what we're doing for everything else. Uh if available to you, depending what tool set you're using, uh GitHub in this case does it. Uh a lot of these static tools can export a file called a SARIF file. And that's a static analysis results interchange format. Uh I don't remember that. I call it SARIF. Uh but then you can use UIs like GitHub security scanning to inject all those SARIF and get a report. Uh this is a report from an example project we have. But more importantly, you do this for 10 projects, and then there's a dashboard higher than that where you can check how all your projects are going. And here's the kicker. Unless you're a rockstar doing continuous deployment, a lot of the static security testing needs to be done twice. You need to ensure that you scan your development environments as well as your deployed environments. Uh we definitely don't want to introduce a issue through our development, uh so that we need to do those on every pull request. But if you only release every month or so, then what happens in your main branch doesn't matter. Uh you need to know you have a major security issue on your production server ASAP. Uh so your production site is where the attacks are most likely to happen. It's the environment that has the data to exploit, and it's the environment where your reputation is most at risk. So you need to pull down your production environment, and you need to run all of the same testing suite. Now, back to Formula 1. So let's say our car passed all the scrutinizing, weight checks, engine checks, etc. But, we take the car out on the track and we fail at 300 km an hour. Uh we passed the weight test because we didn't use enough carbon fiber to cope with the forces on our wings. They fell off and the car fell apart. That's dynamic application security testing. So, this gap is filled by standing up your environment, running it, and then setting some other security tools onto it. Uh it's testing the run running application for security issues, things that our static test can't pick up. It's issues like missing content security policy headers, insecure TLS supported versions, or worse, SQL injection issues by a forms or URL query parameters. Uh if these sorts of things are exploited, uh it'll allow an attacker to wipe your database or export all the data out of it. So, we have a couple of tools for these. Well, one tool, but it's got a couple of different versions. So, we'll go through that. Uh OWASP ZAP is a baseline tool that can we can leverage to find these types of issues. Uh it targets the OWASP top 10. Uh and if you haven't looked at them recently, then I suggest you have a look at that URL. Uh they change every few years, and they change the top 10 based on thousands of recommendations from different security companies as to what people being exploited by. Uh so, to run this baseline, uh you need a deployed working site. It crawls the site for a limited time. Uh generally a minute, but you can change it if you want to. Uh and it will report on all sorts of things that can be improved. The baseline scan is safe to run against any endpoint as it specifically won't try to destroy your site. Uh we again run this test on every single pull request and every merge to main. Uh once it's deployed, uh just to test that we haven't injected any new issues into our products. Uh if OWASP ZAP was the um one that's safe to run, then this is the unforgiving big brother. Uh you pass it a seed URL. It will crawl the entire site until there are no other URLs to find. Then it will actively assess your pages for forms and things, and then attack with SQL injection, cross-site scripting, all that kind of stuff. Uh if you have a vulnerability, it will corrupt your website. I can tell you that from experience. Uh and it's considered unsafe to run on any production site for obvious reasons. Probably your QA site, too, if you want to get work done that day. Uh but it is worth calling out that the full scan is slow. It takes a while to set up an environment that you can scan, like a safe environment that you are allowed to scan. Uh and we only run it about once a week. And we run it in completely separate different workflow, nothing to do with our pull requests and main branch. Uh we feed any issues that it comes up with back into our products through just general Jira tickets. So, all of these things can be set up with GitHub actions and workflows. So, you can have pull requests running all PHP stand and Trivy, all those sorts of things. Uh and then you can have I don't know, wherever you run your functional tests against your deployed environments, you can have OS running and making sure CSP headers are all working. Uh but we need all of those underlying layers, like functional unit tests or functional tests, unit tests. Uh PHP stand again. We need all of that working in order for this to be fully automated. Uh what you don't want is Dependabot sending you a security PR and you don't have any tests. You don't know whether it's actually going to work when you merge it. Uh I've gone a bit faster than I thought. But I'll go over these and then I will take questions if they'll I'll show you a demo of the OS report and then we'll do some questions. Um but SAST is a security test of your code, your dependencies, and your application runtime environment. Its purpose is mostly to let you know your application was safe. Uh well, what is as safe yesterday as it is today? The CVEs change every single day, and if you're not scanning often enough, you won't know that you have an issue. Uh DAST, however, is a security test of your running application. Mostly to ensure you're protecting yourself from SQL injection attacks, bad security headers, misconfiguration of your environments, etc. They cover two different security problems and they largely don't overlap. You need both. Uh we don't want to write code that creates a new security issue and that's why we run all those workflows for every pull request, every main branch. Uh but we don't want any issues in our production environments, either. So, we need to run them. Pull down your containers that you're actually running, run the same test suites against them. Uh keep on top of your maintenance pull requests to limit the splash zone of a security pull request. Uh a single package with a single patch version is a lot easier than 200 packages with major versions. Uh finally, automate everything. When you the goal is when you start work in the morning, you do a quick check of the security reports. Hopefully, you already have a pull request available, the test suite has already run, and you have to click the merge button. Finally, that URL there, a lot of these techniques are live and running on that environment. Uh so, you can pull down that repository, have a look at see what's in there, um and uh that'll continue to evolve. This example project is actually the test Drupal project that we run on our QA clusters. Uh my name's Nathan Taber, I'm a sysops engineer at Previous Next and Skipper. Uh as Vladimir said, I've been a Drupal developer since Drupal 4. I looked up how long that was and I decided to say Drupal 4. Uh I've been living in Auckland now for a couple of years. Uh and Skipper Hosting is launching in New Zealand. Uh if you think of No, any questions? Now, I will quickly show you that example project is particularly bad at the moment. Uh and here's the report to prove it. So, this is an OWASP I think it's a baseline report. I don't even think it's the full report. Uh but this is what you get out of an OWASP scan. Uh you get your high, medium, and lows. You get the reason and what the issues were. But then if we scroll down here, we get this massive report. Uh this is what it comes up with in in a minute. You can imagine what a full scan against your site is. Uh So, it has found that there's a dangerous JavaScript function running in your code with a dynamic test. And it'll tell you the exact URLs that it found it. Uh feature policy is not good enough. Like all that kind of stuff. Uh so, if you just run that uh and then go through and create tickets for whatever you think is important, uh you'll be better than 90% of the people anyway. >> Did you say yeah? >> So, yeah, it depends on the project. So, if it's an example Drupal project that we just have as our testing suite, then obviously weekly, fortnightly is fine. Um but we do have a couple of projects where we do need to know daily. Uh so, we'll run those daily. Um and it's important to say that that daily weekly is only for maintenance pull requests. The security pull request from Dependabot is going to come no matter what. >> Yeah. Is it? That's the style of it. >> Yeah, okay. >> It's getting >> Yeah. Uh so, as I said, um we try to do really quick scans for pull requests and merges domain. And the reason for that is to get really fast feedback to the developers. Oh, there was a bug. I'll fix that. Which is why we do the baseline scan for 1 minute. We just want to know did we put something really obvious in that we shouldn't have done? So, we can get that result back. Uh the expensive scan, like the full scan with the whole environment setup, I guess you could Uh, we generally do it weekly, but you could consult with your client and say, "It's going to cost about $4 in workflow." How often? Like, where's your risk level? >> Yeah, yeah. >> Yeah. >> How do you >> Yeah, and and I do see a lot of people set up all of this type of testing and workflows and everything, and they just do it against their main branch. Like, it is really I can't stress how important it is to run the test against your actual production working code. We have code, well, there is client code in a cluster somewhere that was last deployed 18 months ago. Like, the security doing the security on the latest main branch, who cares? Yeah, so there was um, a talk yesterday uh, on the OSB scanner. Uh, so that you can actually run the OSB scanner in your container that you have deployed on production, if you want. Um, and Renovate and Sneak can target within a file system. It doesn't have to be the main branch. You can say, "Mount the file system, scan that, and get the results back." Um so, there is a couple of different ways to do it. Uh but, at least for our clients, most of them are doing deployments every 2 weeks or so. So, as long as we scan the container for Nginx, OpenSSL, etc., the actual application ones are getting fixed fairly quickly.