Adversarial Simulations

By Max Boehner

Do you have detections in place that have never alerted? If so, how can you be sure they work? If you’ve ever asked yourself this question, keep reading.

Adversarial Simulation (AdvSim) is an activity in which tactics, techniques, and procedures (TTPs) are performed in a controlled manner to simulate behavior of adversaries. In other words: executing attacker tools and techniques. There are multiple reasons for running AdvSim but for now, we will focus on the aspect of testing detection mechanisms.

To start off, here are some things we will need:

A machine to run our adversarial techniques (AdvSim box)

A user account with local administrator privileges on the AdvSim box

A detection tool, most likely a SIEM (Security Information and Event Management)

The AdvSim box should be representative of a typical system in your environment and there needs to be a mechanism for forwarding its logs into the detection tool [check out our webcast on this subject!]. Using a production machine to run tests is not a good idea, as some tests may jeopardize the integrity and availability of the AdvSim box. Ideally, a virtual machine is used that can be reset to a safe snapshot if anything goes wrong (learn how to do this in the next article!). A simple test to ensure that things are set up correctly is to run a program like Notepad on the AdvSim box and then check whether the execution log can be found in your detection tool.

Once we know that logs are flowing, we can move on to picking a detection to test. Assume that we have a detection in place that looks for adversaries launching renamed Windows executables in order to avoid detection. Such a detection can be found in the Sigma project.Do you have detections in place that have never alerted? If so, how can you be sure they work? If you’ve ever asked yourself this question, keep reading.

Adversarial Simulation (AdvSim) is an activity in which tactics, techniques, and procedures (TTPs) are performed in a controlled manner to simulate behavior of adversaries. In other words: executing attacker tools and techniques. There are multiple reasons for running AdvSim but for now, we will focus on the aspect of testing detection mechanisms.

To start off, here are some things we will need:

A machine to run our adversarial techniques (AdvSim box)

A user account with local administrator privileges on the AdvSim box

A detection tool, most likely a SIEM (Security Information and Event Management)

The AdvSim box should be representative of a typical system in your environment and there needs to be a mechanism for forwarding its logs into the detection tool [check out our webcast on this subject!]. Using a production machine to run tests is not a good idea, as some tests may jeopardize the integrity and availability of the AdvSim box. Ideally, a virtual machine is used that can be reset to a safe snapshot if anything goes wrong (learn how to do this in the next article!). A simple test to ensure that things are set up correctly is to run a program like Notepad on the AdvSim box and then check whether the execution log can be found in your detection tool.

Once we know that logs are flowing, we can move on to picking a detection to test. Assume that we have a detection in place that looks for adversaries launching renamed Windows executables in order to avoid detection. Such a detection can be found in the Sigma project.

For the purpose of this test, we will focus on the PowerShell executable. If we want to test this detection manually, we could do this by opening a cmd.exe terminal and copying the legitimate powershell.exe to the desktop of user Advsim and naming it svchost.exe. Then we simply run that fake svchost.exe.

C:\Users\advsim> copy C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe C:\Users\advsim\Desktop\svchost.exe

C:\Users\advsim> C:\Users\advsim\Desktop\svchost.exe

Now all that is left to do is check our detection to see if it fired. If it did not, you may need to dig into your event logs to determine why. This detection relies on the logs containing something typically called OriginalFileName which is an attribute included in Portable Executables (PE) at compile time. This information is included in Sysmon Event ID 1 logs but not in built-in Windows logs.

Now that we have tested our detection manually, you might be wondering if there is an easier and more automated way to do this. Luckily, there is a project called Atomic Red Team that does just that.

Atomic Red Team is a library of so-called atomic tests intended for testing detections. For extra awesomeness, these tests are mapped to the MITRE ATT&CK® framework. There is an atomic which performs a similar test to the manual one we executed. It is named “Atomic Test #5 - Masquerading - powershell.exe running as taskhostw.exe” and can be found under the category “T1036.003 - Rename System Utilities”.

Now, how do we execute this atomic? There are multiple tools out there that can run them. We will be using Invoke-AtomicRedTeam because it is free and easily installed via PowerShell.

To install both Atomic Red Team (the atomics) and Invoke-AtomicRedTeam, we open a PowerShell window as administrator and run the following commands. Please note that you will need to disable any antivirus or EDR solution on the AdvSim box to avoid problems. The atomics contain scripts for simulating malicious activity so some of the files will look like malware.

IEX (IWR ‘https://raw.githubusercontent.com/redcanaryco/invokeatomicredteam/master/install-atomicredteam.ps1’ -UseBasicParsing);

Install-AtomicRedTeam -getAtomics

If you are prompted about the installation of NuGet/PowerShellGet, allow it.

In the same PowerShell window, we now have access to the Invoke-AtomicTest commandlet. Before executing our atomic, we can look at its details. Remember that we want to execute test #5 for technique T1036.003. To obtain details about the test, run:

Invoke-AtomicTest T1036.003 -TestNum 5 -ShowDetails

This tells us that the test will copy powershell.exe into taskhostw.exe and subsequently run that copied file. This is not exactly what we did earlier but similar, and if we wanted to, we could use custom arguments to specify the target file name to be svchost.exe. For the sake of simplicity, we will skip custom arguments. Now run the commands to first execute the atomic and then perform clean-up, which will remove our renamed PowerShell file:

Invoke-AtomicTest T1036.003 -TestNum 5

Invoke-AtomicTest T1036.003 -TestNum 5 -Cleanup

Once again, we will want to verify that our detection fired. If our manual test was detected, then this should typically also have worked.

Armed with this knowledge, we can look at how to do AdvSim at scale. One of the benefits of using Atomic Red Team over manual testing is that it is easy to create scripts that import and utilize the atomics. This allows us to assemble sets of atomic tests for our detections and execute them periodically to ensure our detections are working as expected. Think of this like unit tests for detections.

Another interesting use case for Atomic Red Team is running atomics for techniques relevant for your environment and testing if you already have detections for those in place. You could even assemble chains of attack techniques based on threat intelligence reports and run those. Lastly, if you are very confident about your detection capabilities, you should consider having AdvSim performed with the specific goal of trying to evade detection. This can be useful to determine how strong your detections are when faced with advanced adversaries that try to be stealthy.