DEF CON SG1 - CSIT Village - James Tan - Uncovering Hidden Threats: CSIT Qpproach

All right. And for our final talk of today, we have uncovering hidden threats, where we go behind the scenes of threat investigation. James Tan walks us through real workflows, real incidents, and how complex threats are detected and handled in practice. It's the reality of cybersecurity beyond the theory. Let's give James a warm welcome. >> Thank you everyone for coming to my presentation. So, today I'll be sharing about how CSIT approaches threat investigation in our day-to-day work. So, before we continue, some introduction uh introduction to myself. I joined CSIT as a threat investigator not too long ago, since December 2023. Before joining, I studied information security in National University of Singapore, and was a long-time CSIT scholar since my polytechnic days. During my internship, I felt that the work aligned with my technical interest in cyber work, so I went ahead to apply for a position here, and the rest is history. Working at CSIT gave me plenty of opportunities, which I'm very grateful for, including being able to share the work that we do at Def Con today. Without further ado, let's get right into the content. So, a brief overview on what we'll be going through today. Firstly, I'll be going through the broad roles of a threat investigator before diving into how our workflow looks like when it comes to dealing with cyber incidents. When it comes to threats, there's a long list of creative threats out there in the world, so I will broadly summarize them into different threat actors according to their motivations and severity. I'll also be showing a real case study that occurred sometime last year, and we can take a look at what events unfolded as time went by before wrapping up with some key takeaways that we learn from it. Being a threat investigator places us at the forefront of cyber attacks, and these are real attacks happening around the world. We're not simply preparing for something that could happen, we are actively defending against things that are already happening. But, what exactly does a threat investigator do? Although the specifics may differ between different agencies and organizations, we broadly categorize it into four main areas. Firstly, risk assessment. Risk assessment is more of a preventive stance with the aim to lower the risk of existing threats. We look at global threats and implement best practices such as hardening guidelines to ensure that we are well protected. Doing so will help us potentially cut down the known threats and elevate our security posture. However, cyber threats are ever-changing, so we will still have to be vigilant in our detection. As such, part of being a threat investigator is to detect the threat as early as possible. This can come in a range of various telemetries collected from various endpoint sensors. After these telemetries are collected, we have to make sense of the data by incorporating rules or filters to dynamically suss out any form of malicious activities. This can come in the form of rogue processes running, network connections made to unknown servers, or even user accounts being added or modified. Threat monitoring remains the very first step in a successful threat investigation, and without a robust monitoring system, threats may remain undetected for a long period of time. So, how do we overcome this? And that will be the next role, threat analysis. This can come in the form of a more active lookout for existing threats, where we will actively look out for suspicious activities before our sensors are able to detect them. For example, whenever there is a newly released exploit in the wild, we would expect a surge of attempts leveraging that same exploit, which might affect systems that have not been able to patch themselves in time. In order to mitigate this, we will have to assume the role of an attacker once in a while and look out for such vulnerabilities before others can make use of it against us. But what happens when an actual intrusion occurs? This will be the final role of a threat investigator, where we have to respond to incidents. Only for confirmed cases, it is important to know the impact and intent of the adversary. What damage has been done and how can we mitigate the intrusion to prevent further damages to our systems? Depending on the profile of the adversary, their motivations can vary drastically. To some, they are financially motivated. Others want to steal data or to simply destroy and kill and cause chaos. By knowing the adversary's intentions, it gives us a better idea on the immediate next steps to take. Now, let's zoom in to the actual threat investigation work itself. It all starts with an initial detection. Where our sensors first make contact with the possible intrusion. This may or may not be a false positive, so we will need to perform an initial assessment of the situation first. The initial assessment seeks to answer a few key questions, mainly for our stakeholders to get a good sensing of the severity of the attack. At this point, time is of the essence, so we will need to act fast to get a rough picture of what is going on and whether it is even a legitimate attack in the first place. Once an intrusion is confirmed, a preliminary report will be generated to inform the relevant parties of the situation before we continue to analyze the specifics of the incident. To me personally, this is the most interesting part of the job. Most of the technical work happens here in order to find out the details of the intrusion and answer even more in-depth questions such as, what capabilities does the attacker have and what are the attacker's intentions and possibly identity as well. Once the facts have been established, these findings are placed into a technical report to once again update the relevant parties on the details. Together with this report, we will also give our recommended follow-ups in terms of damage control and prevention of reoccurrence. Broadly, this is the workflow that we follow whenever a detection occurs. Thankfully, due to our vigorous hardening and proper defenses in place, successful intrusions do not occur regularly, but the amount of attempts that we see is on a daily basis. So, before any threat investigation can be done, we must first understand what are the types of threats that are out there. Threats refer to a very broad term for anyone that may attack us. But behind every cyber threat, there will always be a human behind a computer. And these are our threat actors. Depending on their motivations, their steps taken will vary greatly. Starting with the lowest of severity are our script kiddies. As the name suggests, these are individuals who use pre-existing exploit scripts and tools to launch cyber attacks. However, they may lack in capability and knowledge and can be seen as simply trying their luck by mass exploiting anything and everything around them. They are mostly motivated by curiosity and seek out thrill and ego. But they are also more likely to make mistakes since they may not fully understand the tools that they are using. Most of their attempts do not succeed as their methods are dated and should have been patched against already. Although this group of threat actors may seem harmless to most, it is still important to meet minimum safety guidelines to prevent attacks from them. Moving up the list, we have our hacktivists. Hacktivists use cyber attacks to promote a particular ideology, cause, or agenda. Due to their motivation, their attacks can be highly targeted against whoever they are campaigning against and as such, it could be more severe. One common example would be Anonymous where they would deface certain websites to send a message about their cause. However, hacktivists are normally not interested in creating any real damage to their targets and it is mostly for embarrassment or disruption as long as they are able to get their message across. Next, we have our criminal organizations. Criminal organizations are usually financially motivated and will employ a range of tactics to get money out of their victims such as scams, crypto mining, gambling, or ransomware. For this group of people, their capabilities start to improve and they are more careful with their operations to avoid being exposed easily. As you all know, scams are always on the rise. Scammers are getting more creative with their delivery of their scams, such as creating fake restaurant booking sites or masquerading their phone numbers when cold calling their victims. All this shows their sophistication in cyber knowledge when it comes to achieving their objective. And lastly, we have our state-sponsored actors. This group of people have the backing and funding from their respective to conduct covert operations to serve their nation's needs, including cyber warfare. Due to their financial backing, they will also have very sophisticated tools and equipment to launch their attacks. They also have to be even more careful when it comes to being stealthy, as any exposure of their operations would have serious repercussions in a nation's reputation and relationship with others. I'm sure you all have heard of the term APT before, and it was used alongside the recent news about UNC3886. In this case, UNC3886 is attributed to a state-linked espionage group. An APT is a loose term to describe any threat that is literally advanced and persistent. And it is commonly used for state-sponsored actors, although not always the case. Among all these threat actors, they all rely on the same thing. Access. Without a good access, none of their operations can take place. Some gain access via phishing attempts, where they will ask a victim to visit a link or scan a QR code. Others will try brute forcing or guessing of credentials. But today, we will zoom into one of the strongest method to achieving a good access, which are zero days. Now, I'm sure all of you are aware what are zero days. As the name suggests, zero days are vulnerabilities that are unknown to the developers, and no patch are currently available. Since it has not been publicly announced, the exploit is as good as being used on the same day that it has been discovered, hence the term zero day. Zero days are extremely important for threat investigators as they give us valuable insights to how an attack occurred so that we can better prevent it in the future. Speaking of zero-days, we will now look into a case study that used a possible zero-day just last year. So sometime last year, we were made known to a possible intrusion. Our sensors picked up a suspicious process running and upon receiving the alert, we immediately started looking into it. To better understand the incident and for the purpose of this presentation, I will break it into four stages. Along the way, you can try to figure out the nature of the adversary, taking note of the traits that they exhibited and understanding the human behind the threat. We will first look at the adversary's initial entry. As mentioned earlier about zero-days, we hypothesize that the adversary exploited an application zero-day vulnerability to perform remote code execution against his targets before uploading their payloads. So let's see how do we arrive at this conclusion. During our preliminary investigation, we found these log entries near the time of our detection. There are three data points that can be observed from this small log snippet alone that helps us to arrive at some form of conclusion. The first thing we notice here is that there's a pattern. Notice that for each post request, one letter is being removed from the username. This is a classic sign of brute forcing where the attacker is guessing for a valid username to exist on the server. The second thing to notice here is the closeness of timestamps between each post request. This indicates that the brute forcing was automated and it is likely a scripted guessing instead of a traditional word list. Another thing to notice here, as of every brute forcing attempt, would be the success criteria being error code 302 instead of 404. This means that the username, user N in this case, exists on the server and the post requests are being redirected. Although this snippet of log alone, we uh through this snippet of log alone, we established that the adversary requires a valid username to exist on the server for their exploit to work. And it is done through an automated brute forcing. We also now know that the adversary is capable of or in possession of such a brute forcer. So, we continue to trace the logs related to this username, and we arrived at yet another log snippet. Once again, there are three important points to take note here. Firstly, is the repeated post requests to the same destination but getting different responses. This raises some suspicion that the response received may contain some information that is important to the attacker. But this data point alone is insufficient, as this could be normal user activity. So, we we still need more supporting facts. The second thing to notice here is the behavioral pattern of the request. A few post requests were sent to a search files function before one sent to the upload files, and then again to the search files function. Again, it still looks like normal user behavior, akin to doing normal directory traversal and looking for suitable place to upload a file. However, the third observation helps to confirm our suspicion and that there will be a user agent in the request. Normal user behavior would access the page via some browser like Mozilla or Chrome. However, the logs captured here are Python requests, which means that the requests are being sent by a Python script. Again, this log snippet gives us more insight on the behavior of the adversary. The adversary was making use of a Python script to send crafted post requests to these two particular functions. As of this moment, our investigation efforts pivot into looking at these two affected functions, as it seems that these functions are important to the adversary. So, let's take a look at the next slide. The contents of post requests are normally not captured at the log level, but we are interested to know what exactly did the adversary send to the compromised system. Given the urgency of the case, we started by performing a memory core dump of the affected web process in hopes to find some remnants of the adversary's traces. And true enough, we were able to see the request sent by the adversary to the search files function. Here we can infer that the vulnerable parameter in the post form is the search value parameter, and this is also where the adversary injected his malicious payload by performing a simple escape before placing his reverse shell command. This answers the question as to how initial access was established. The attacker must first be able to find a valid username that exists on the victim system before crafting a post request targeting the search files function to perform remote code execution. One thing we noted here is that the attacker does not need to be authenticated to perform this exploit, which highlights a flaw in the code flow when it comes to checking of authentication and processing of requests. Another thing to note here is our suspicion of this being a zero-day. Back then, we attempted to find any reports regarding this vulnerability, but we couldn't find any. And since this was the first time we have seen an attack vector against this vulnerable function, we had a suspicion that this was likely a zero-day. From all these data points, we conclude that the adversary exploited a zero-day to perform some form of remote code execution. After we know how the adversary got in, we can now look at what they did. To summarize, the adversary planted malicious payloads before hijacking the victim's hosted websites for their own purposes. Let's start by looking at what the adversary typed after getting access. So, in this a snippet alone, this is the bash history of the adversary. We can split the adversary's actions into three main groups. Firstly, they deleted this shared object file named mod_intercept from the Apache web server's module folder. Afterwards, they downloaded their own Sorry. Afterwards, they downloaded their own file from another remote location and replaced the deleted file with the same name of mod_intercept. One thing to note here is that the file and folder name of this remote location looks like it has been masqueraded as a font file, noting the file extension of {dot} e0t. But after bringing it in, it is now renamed as mod_intercept. Afterwards, the adversary modified the httpd config file to include this new mod_intercept file. They used the sed command to perform an inline modification, specifically to insert into line 113 of the configuration file. Lastly, the adversary restarted the web service to load this new modified config file. As of this point, the adversary successfully brought in his own file, masqueraded it as a seemingly legitimate file on the system, and configured the web service to load this malicious file. Now, I would like to draw your attention to this particular line here. Notice that the adversary used the touch command with a {dash}r flag. Now, what exactly does that do? Using the touch binary with an r flag, it sets the timestamp of a file to be the same as a reference file. In this case, the adversary was setting the timestamp of the file that they brought in, which is mod_intercept, to have the same timestamp as another legitimate file in the same folder. In this case, it would be mod_allow_methods. The adversary did this in order to better hide his malicious files within the list of other legitimate modules, so that it will not be immediately obvious which file was the one that they newly brought in and replaced. But if we look at the next section here, we notice that the adversary did something interesting. They attempted to do the same for the httpd config file by hiding the timestamp. However, his reference file HTTP config_backup does not exist. The adversary, for some reason, decides to make a copy of the config file as a backup before performing the touching again. But, we know that this won't work as the timestamp of the original config file has already been changed. This mistake helped us identify roughly when their activities happened, which allowed us to narrow down our investigation efforts greatly and focused on the time span. This data point us profile the threat actor a little bit more. And we now know that the adversary is attempting to hide their traces, but they are simply not careful enough. Now, we can look at the files planted by the adversary itself. Earlier, we know that the time span to be looking out for using that as a clue, it brings us to this particular file on the system in the system's temp directory. This file is named as auto_monitor, and we can see its contents here. The first part of the file, we can see a bunch of obfuscated gibberish. But, we can see that this gibberish text is being decoded and saved to a file named temp_baf. Note that this is stored as a hidden file due to the dot prefix, which means that the adversary is still somewhat mindful to hide their traces. In the second half of the file, we see that this tmp_baf file is being copied to different users' web folders and renamed as default.php. As of this moment, we do not know why it needs to be copied to every user's web folders, but let's look at the contents of this tmp_baf file first. Here, we can see the contents of the file. Right from the start, we know that it is heavily encoded with multiple go-through code redirections and a bunch of gibberish variable names. Although we can try to eyeball and perform static deobfuscation, that will simply take too long. Instead, we spin up a virtual environment and host this file on a PHP server. And it is reviewed to be a GUI-based web shell. This GUI-based web shell has basic functionalities such as file and folder manipulation, directory traversal, as well as remote code execution. Knowing that this is a web shell, it makes sense as to why the adversary wanted to copy it to every user's web folder. By placing their web shell in as many publicly accessible places as possible, they achieve some form of redundancy in terms of their access for the machines. If we piece this artifact together with our earlier analysis of their keystrokes, and given the closeness in timing, it is likely that the adversary used the exploit to upload this web shell before using the RCE functionality to establish a reverse connection back to their systems. We also now know that maintaining access is important to the adversary. However, their chosen method is extremely noisy, and there was not much thought to hide their activities other than obfuscating the code and renaming of their file to seem legitimate. From these two data points alone, we know that the adversary does not really care if they were caught or not, which may imply that this is just one among many other victims out there. Now, recall earlier that they modified the web server's configuration file. So, now let's look at it. Here are two versions of the config file, with the left side being an original copy, and the right one being a modified one. If you were to try to eyeball the difference between the two files, it would be rather time-consuming. And also, since each web server's configuration file differs from system to system, it is also not reliable to simply use a diff tool to compare. However, recall from the adversary's keystrokes earlier that the adversary added a line specifically to line 113 to include their file as a module. And we can see here, buried among the other legitimate lines of other modules. Again, another way of hiding their traces by hiding in plain sight. So, what does this mod_intercept file do? Unfortunately, the module is a data file, so it is hard to show it on a slide, but we will briefly sum up what the file does. As the name suggests, the legitimate process of the legitimate purpose of the folder intercepts requests to the web server and processes it. In this malicious file, the adversary configures the web server to act as a proxy instead, redirecting all traffic to a hardcoded webpage that belongs to the adversary. Up to this point, we now know that the adversary is performing some form of hijacking by redirecting traffic entering their this web server to another site of their control. So we have some idea of their intention. But this is still not sufficient to confidently conclude. So, let's now follow the redirect and see where we end up on. And so we end up here. It should be quite obvious that this looks like a gambling site with foreign words. So we now know that the adversary is redirecting visitors to their gambling site. However, this raises another question. Imagine yourself as the owner of a gambling site. Naturally, you will want as many people as possible to visit your site. So it doesn't make sense for gambling site to be hosted behind a proxy as that makes the site not obvious to many and to some even tedious to enter. So why did the adversary decide to place their gambling site behind a proxy instead of simply defacing and replacing the victim's hosted site entirely? Answering this question will allow us to know the intention of the attack. To summarize, the adversary was likely performing a search engine optimization hijack against web servers hosting popular webpages. So let's see how we arrive at this conclusion. If we were to do a quick search on some of the more unique keywords about the gambling site, we can see that some of the search results are actually legitimate websites. However, the contents displayed are related to the gambling site instead. Why is it so? To explain this, we have to first understand how a search engine indexes your page. When a web page is created or updated, search engines update their index regarding their known pages. Even when these web pages have been hijacked, the index still retains information about the original website, such as its authenticity and its backlinks. Knowing this, it makes sense why the adversary was targeting legitimate websites. The adversary was leveraging the high search rankings of these websites to put themselves higher on the search engine rankings. This allowed for greater chance of tricking users to visit their websites, which is a classic case of SEO hijacking. This is especially dangerous when the original website is a particularly popular one. However, a more popular website or web server would naturally have more defenses in place. As such, it is a relief that the adversary was limited in capability in this aspect. With this information, we can now piece the sequence of events together. The adversary began their attack by performing an automated brute forcing to find a valid username that exists on the victim system. Afterwards, they gained initial entry by exploiting a zero-day vulnerability using the valid username, which allowed him to perform a basic RCE. They went on to bring in several malicious payloads from their C2 server, such as an obfuscated web shell and a web server configuration module. This module was used to turn the victim's web server into a proxy to perform site redirection to their foreign gambling site. To achieve some form of redundancy and persistency, the adversary copied their web shell into all other users' web folders in the event that they need to re-access their victim again. After running through their initial entry, the actions performed, analysis of their artifacts, and their intentions, we concluded that the adversary was likely linked to a crime syndicate. In this case, we believe the main objective of the adversary was financial gains in the sense that more people visited their sites, the richer they will get. We also noticed a wide scale reach of their attacks, evidently from other web other affected websites which they resolve from search engines. However, we also note that stealth was not their top priority. The adversary made mistakes in their operations and were not too concerned about hiding their traces entirely. This aligns well with their widespread reach in the sense that getting caught would not impact their operations much anyway as they have many other victims. And that wraps up the sharing of our case study. Although this is just one case, it is important to acknowledge that every case is unique, each having their own traces to discover and lessons to be learned. This case in particular reminds us again that the cyberspace is an ever-changing threat. Cyber attacks happen around all the around the world all the time, even right now. New vulnerabilities and threats emerge daily, each having their own unique intentions, methodologies, and capabilities. As the threat landscape changes, we also have to continuously learn and experience these variable threats around us. Some even close to home. And we must never underestimate any and every threat we face. This case also taught us that the scale of each adversary cannot be underestimated. As technology becomes more and more ingrained into our daily lives, this also means that cyber adversaries can now command a large amount of resources and impact many people easily. Especially when AI is so accessible now, these engines can be used to help adversaries build more resilient tools or offer obfuscation techniques that would have taken much longer for a human to come up with. Naturally, this also means that we as defenders can also leverage AI to help us in our investigation work. Being on the defensive angle means we will always be fighting the unknown. Threats evolve constantly and get more sophisticated over time. And as much as we can try to prevent this, we are limited to the knowledge of what we know. And that is why it is important to perform a constant review of our security posture and to occasionally assume the role of an attacker. And with that, I've come to the end of my presentation and I'll be happy to take any questions offstage. Thank you.