Cobalt Strike Blog | Cobalt Strike Research and Development

Post-Mortem of a Metasploit Framework Bug

Two weekends ago, I ran my Advanced Threat Tactics course with a group of 19 people. During the end exercise, one of the teams was frustrated. Their team server was incredibly slow, like mollasses. I asked the student with the team server to run top and I noticed the ruby process for msfrpcd was consuming all of the CPU. I mentioned that I had seen the issue to which the student leaned back, crossed their arms and responded “oh, great, I guess my 2-cores and 4GB of RAM aren’t enough–harumph!”

I wasn’t fibbing to sweep the issue under the rug. I have seen this behavior before, for the last year actually. It frustrated me too, but I was never able to isolate it. In this blog post, I’d like to share with you the story of this bug and how I managed to isolate it. I hope this will help you with tracking down issues you encounter too.

To scan through a pivot, Armitage has a feature I call MSF Scans. This feature will run auxiliary/scanner/portscan/tcp to discover services and follow it up with several auxiliary modules to further enumerate the open services.

I noticed on some virtualized systems that following this process would lead Ruby to consume CPU for an entire core, making the Metasploit Framework non-responsive. On Amazon’s EC2, a micro instance would nearly always trigger with this problem. It’s this reason I recommend that Armitage and Cobalt Strike users use a high CPU EC2 instance.

When a thread is so busy that it consumes all of the CPU, we refer to this problem as resource starvation. This busy thread is preventing other threads from running as often as they normally would, making the whole system feel slow.

I took a look at the virtual machine I gave out in class to see if I could reproduce the problem. Most of the time, when I would run a scan–everything was OK. If I opted to run multiple scans at once (uncoordinated teams sharing one Metasploit Framework instance do this a lot), then  I was much more likely to trigger this problem. When I ran multiple scans through a pivot enough times, I would reliably trigger this CPU starvation condition.

Reliably reproducing a problem is the first, and often hardest step in actually fixing it.

Next, I had to figure out where this problem was happening. In Java, there’s a way to dump a stracktrace of every running thread to STDOUT (use kill -3 [pid] to do this). Ruby has a gem called xray which will sort-of do this. By default it only dumps the current thread. If I wanted to patch Ruby, apparently I can get it to dump all threads. I decided to look for another option.

The Metasploit Framework has a threads command. Typing threads by itself will list the threads spawned by the framework:

If you type threads -i [number] -v you will see a stacktrace for that thread.

You may also use threads -k [number] to kill a thread.

Armed with this information, I opted to trigger the CPU starvation condition and kill threads one at a time until the CPU spinning stopped.

I had an inkling that one of the threads created by the auxiliary/scanner/portscan/tcp module was the cause of this CPU use.  I kept examining and killing threads until the only ones left were the MeterpreterDispatcher and MeterpreterReceiver threads.

When I killed my meterpreter session, the CPU use went to a normal level. When I killed all jobs and threads related to my portscan, the CPU use stayed at the high level. Conclusion? The problem is in the MeterpreterDispatcher and MeterpreterReceiver threads–somewhere.

I dumped the stacktrace for these threads. I then started at the top and looked at thread_factory.rb. I had this crazy notion that each framework thread checks out a connection from the postgres database. Maybe I exhausted this pool somehow (or something wasn’t giving back to this pool) and this would cause further thread creation to block, possibly forcing some code into a busy wait state. This assumption was not correct.

I took a look at the next spot in the stacktrace, packet_dispatcher.rb.

Line 255 of packet_dispatcher.rb is the start of the code for the MeterpreterReceiver thread.

Line 307 of packet_dispatcher.rb is the start of the code for the MeterpreterDispatcher thread.

You may use p to print any Ruby data structure to the console or wlog to log a string to the framework.log file in ~/.msf4/logs. I added a few p statements to these two threads so I could understand what they were doing. For example:

p "The size of @pqueue is #{@pqueue.length}"

When I triggered my CPU consumption condition, I noticed something strange.

MeterpreterDispatcher would loop inspecting the size of a variable called @pqueue. At the beginning of this loop, @pqueue would always have one or two items. This thread only sleeps when @pqueue is empty. This is OK, because the MeterpreterDispatcher loop clears @pqueue during each iteration.

How do values get into @pqueue? I took a look at MeterpreterReceiver. This thread will add values to @pqueue. According to my debug output though, the MeterpreterReceiver thread was not adding values to @pqueue when my bad loop was hit.

I looked closer and noticed that at the end of MeterpreterDispatcher there is a check. The thread will try to process a packet and if it can’t, it will insert it back into the queue. Interesting.

If MeterpreterDispatcher can not process a packet (for whatever reason), it adds it to the queue. The queue is no longer empty so it is guaranteed to try again without sleeping. If MeterpreterDispatcher can not process the packet again, it adds it to the queue. Bingo… I found my resource starvation opportunity, almost.

MeterpreterDispatcher has a check to take a packet off of this treadmill. If this check takes incompletely processed packets from the queue, then I’m wrong again. I examined the code and saw that the packet timeout value is 600s or 10 minutes. When a packet is not processed, it’s added to the queue, again and again until it’s processed or 10 minutes pass. This explains why the problem would show up and go away if I left the framework alone for a time.

At this point, I wrote a simple patch to sleep when the packet queue is populated entirely with packets it couldn’t process. I submitted my pull request and after egypt was able to verify the issue, the issue was closed. I’m always amazed by the responsiveness of the Metasploit Framework dev team.

I hope you enjoyed this post mortem. I wrote this post because I’d like to encourage you to dig into the weird issues you encounter and try to solve them. Sometimes, you’re the person with the right environment and situation to trigger a hard to reproduce issue. The Metasploit Framework team did a wonderful job providing us tools to inspect what the framework is doing. With these tools, you can isolate these complicated issues–even if you’re not much of a Ruby programmer. Fixing bugs is an important way to contribute to an open source project. A module may delight folks with their new found powers, but a bug fix will save stress, frustration, embarrassment, and potentially counseling costs for thousands of people. I’m happy this one is fixed.

Cobalt Strike and Armitage Updates

Updates to Cobalt Strike and Armitage are available now. I spent the past two weeks testing these programs through unit tests, a class QA focus group, and multiple penetration tests of my local lab from Amazon’s EC2.

This update fixes several bugs in both programs and I was even able to contribute a few fixes to the Metasploit Framework.

As usual, the trial version of Cobalt Strike is available at: (changelog)

Use msfupdate to get your hands on the updated Armitage release, or go to: (changelog)

For either application, I recommend that you update the Metasploit Framework to get revision 15972. A CPU resource starvation condition was mitigated last night. This bug would cause a Ruby thread to take over the CPU for up to 10 minutes at a time, making Armitage and Cobalt Strike barely usable. I have a blog post coming on this particular bug.

P.S. Strategic Cyber (*cough*me*cough*) is at the Maryland Cyber Challenge and Conference today and tomorrow. Stop by our table to get one of the nifty penetration testing lab DVDs with exercises and target VMs paired to our online course.

Advanced Threat Tactics Training

I share a lot from my experiences playing on exercise red teams. I talk about the tactics to collaborate, persist on systems, and challenge network defenders in an artificial environment. Armitage was built for this role.

I speak little about my experience working as a penetration tester. I used to work for a security consulting firm providing “red team services to a DoD customer”. My job was threat emulation. My partner and I would plan and execute actions over a long period of time. All of our activities were double-blind. To protect our work, my boss would meet with our contact in a public area set aside for smokers, hand over our plan, and gain approval to execute at that time.

Last October, I was asked by the LASCON organizers in Austin, TX to teach a one day course at their conference. I opted to teach a course on threat emulation. This is when I wrote Advanced Threat Tactics with Armitage. The course briefly introduced Armitage and the Metasploit Framework. A lot of time was spent on how to get a foothold using tactics these tools don’t directly support. The lecture portion ended with two talks on post-exploitation and how to move inside of a network.

The capabilities missing from our tools made up the Advanced Threat Tactics portion of the course. In these three lectures and labs, I taught:

  • All attacks start with reconnaissance. How do you perform reconnaissance before a targeted phishing campaign? I introduced the concept of a system profiler and how to build one.
  • What do you do if client-side applications are patched? Think like a criminal–you care about the end and not the means. Here, I introduced the idea of hacking with features. It’s important to know how to look at an attack surface and recognize opportunities to get code execution. Sometimes the simple ways work best.
  • Once you have an attack, you need to make sure it passes anti-virus. You also need to think about command and control and how you will go through a restrictive firewall. In this portion of the lecture, I introduced students to these ideas and tools available (at the time) to help them with this process.
  • Once you have your attack put together, it’s important to package it in a convincing way and get it to your target. Here I taught how to send a pixel perfect phishing message. I made students do these steps by hand. Nothing says fun quite like stripping headers from a message in a text editor and then typing SMTP commands by hand to exchange email with the target’s mail server.

My course helped students think creatively about how to get a foothold in a network and use that foothold to achieve a goal. The missing capabilities in the penetration tester’s toolbox have become the road map for Cobalt Strike.

Fast forward one year later. I’m teaching a two-day Advanced Threat Tactics course at OWASP AppSec USA. The heart of the course is still the same. It’s a two-day opportunity to learn how to think creatively about the hacking process and execute the tactics through several guided labs. The two-day time frame allows me to add a lab and lecture on evading defenses. I have also expanded the post-exploitation and maneuver lectures.

The best part of the course is the exercise though. The course ends with an exercise that lasts several hours. You have the opportunity to work with a team and assume the role of different threat actors attacking a simulated enterprise.

  • As a hacktivist, you’ll break into the ACME corporation, discover their dirty secrets, deface their website, and publish their email spool
  • As an actor interested in economic espionage, you will gain access to the ACME corporation, find the source code to their secret project, and steal it.
  • And, as a nation we face risk of sabotage through cyberspace. As this threat, you’ll find and manipulate a control system that leads to the destruction of a nuclear reactor.

I wrote this course for a broad audience to include novice to experienced penetration testers and network defenders. I teach the Advanced Threat Tactics by request to organizations who have the resources for 12-15 students. For individuals, the best opportunity is to attend Advanced Threat Tactics at a conference. The next run of Advanced Threat Tactics is at AppSec USA in Austin, TX. The course is Tuesday, 10/23/12 and Wednesday, 10/24/12. If you’d like to sign up, there’s still space available.

Dirty Red Team Tricks II at Derbycon 2.0

Last year, I spoke on Dirty Red Team Tricks at Derbycon. This talk was a chance to share what I had used at the Collegiate Cyber Defense Competition events to go after student networks. During this talk, I emphasized red team collaboration and our use of scripts to automatically own Windows and UNIX systems. I also released the auto hack scripts at the event.

This year, I had a chance to update this talk and show what is different about this year. At this talk, I emphasized the use of bots and how they helped us play the game. I also talked about the use of asynchronous command and control to better hide our presence on student systems. I released Raven, the asynchronous C2 agent I developed for this year’s CCDC event. Raven is the prototype of Cobalt Strike’s Beacon feature. I also released a few other Cortana scripts discussed in the talk. This talk also covers a neat Windows persistence trick using DLL hijacking against explorer.exe.

Thanks to Adrian “irongeek” Crenshaw‘s amazing speed, I’m able to share both videos with you today. It’s best to watch both videos in order.

Let me know what I should cover in next year’s Dirty Red Team Tricks III.

Strategic Cyber at Derbycon

Day 2 of Derbycon 2.0 – The Reunion is about to start. Strategic Cyber LLC is near the Capture the Flag room exhibiting Cobalt Strike, answering questions, and talking about hacking.

The airline destroyed my portable monitor (boo!), so we’re working off of laptops, but it’s OK.

We have several goodies that we’ree handing out too. Goodies include Armitage stickers, a limited number of Cobalt Strike stickers AND pen test lab DVDs.

Pen Test Labs

Yes, pen test labs. Our free pen test lab consists of three virtual machines.

  1. An attack virtual machine with a 21-day trial of Cobalt Strike that starts when you first run Cobalt Strike
  2. The awesome Metasploitable 2 virtual machine from the Metasploit Project
  3. A workstation victim virtual machine with self-contained email infrastructure

These virtual machines provide a quick and safe way for you to experiment with the offensive tools and techniques. The DVD also includes several step by step labs tied directly to the free Penetration Testing with Cobalt Strike course.

We have a limited number of DVDs available and they moved fast yesterday.


The big topic around the table is Beacon, Cobalt Strike’s new covert command and control payload that mimics the C2 of advanced malware and RATs. This is an exciting capability leap for penetration testers. We will be answering questions and demoing aspects of Beacon at the table as well.

Dirty Red Team Tricks II

Sunday at noon, I will be delivering the Dirty Red Team Tricks II talk at Derbycon. Last year’s talk was quite a hit. I provided the kit and process we used at the Collegiate Cyber Defense Competition to work together as a red team and stay hidden on student systems. This update to the original talk will feature 2012’s tactics. You don’t want to miss.

That’s about it. I look forward to seeing you at the con.

Beacon – A PCI Compliant Payload for Cobalt Strike

TL;DR Beacon is a  new Cobalt Strike payload that uses DNS  to reduce the need to talk directly to Cobalt Strike. Beacon helps you mimic the low and slow command and control popular with APT and malware.

In the interest of helping you verify vulnerabilities for compliance purposes, I’d like to introduce you to Beacon, a new feature in the latest Cobalt Strike update.

Beacon is a PCI compliant payload (if PCI means Payload for Covert Interaction). Beacon offers long-term asynchronous command and control of a compromised host. It works like other Metasploit Framework payloads. You may embed it into an executableadd it to a document, or deliver it with a client-side exploit.

The next time you have to run an exploit to check the box, why don’t you exploit the CEO’s system and use Beacon to quietly maintain a lifeline into the network until everyone is gone for the night? Then you can inject Meterpreter into memory, load Cobalt Strike’s Covert VPN, and run your favorite vulnerability scanner

What is that you say? Your customer has decent network monitoring? They’ll block your beacon before anything can be done about it? OK! Beacon can phone home to multiple domains. If one gets blocked, that’s OK. If you own a few domains and have a few NS records to spare, Beacon can check for tasks using DNS requests. It doesn’t need to communicate with you unless a task is waiting for it.

Beacon’s features include

  • Check task availability using HTTP or DNS
  • Beacon to multiple domains (who cares if that first one is blocked)
  • Capable of automatic migration immediately after staging
  • Tight integration with Cobalt Strike. Deliver beacon with social engineering packages, client-side exploits, and session passing
  • Intuitive console to manage and task multiple beacons at once

Beacon is available in the latest 21-day Cobalt Strike trial. You may download it at

Licensed users may use the update program to update their Cobalt Strike installation to the latest version.

If you’re at DerbyCon, make sure you stop by the Strategic Cyber LLC table for a demo.

Are you headed to OWASP AppSec USA in Austin, TX? I’m teaching a two-day Advanced Threat Tactics course. In this course, I will show you how to evade defense technologies, gain a foothold in a modern network, and carry out post-exploitation. It’s a great way to learn more about how to use technologies like Beacon.

Information Security Movers and Shakers Interview

My friend Vivek Ramachandran, founder of, recently launched a new blog interviewing “Information Security Movers and Shakers“. I’m certainly a shaker as I find myself quaking in my boots nearly every day. Sometimes from excitement, sometimes from doubt. 🙂 Welcome to entrepreneurship.

Vivek’s questions created a great opportunity to cover the Armitage and Cobalt Strike story to date. In the interview I talk a little about how I got started, the differences between Armitage and Cobalt Strike, and just what the heck is threat emulation. I also go more into the Cobalt Strike roadmap than I have done publicly before.

Go read the interview and enjoy.

Delivering custom payloads with Metasploit using DLL injection

I’m very interested in supporting alternative remote administration tools in Cobalt Strike. Meterpreter is awesome as an active RAT, but I need something less chatty to hold my accesses when I’m not using them. I plan to talk about about this in my upcoming Dirty Red Team Tricks II talk. In this post, I’d like to talk about how to deliver a custom payload with one of the Metasploit Framework’s existing stagers.

Problem Statement

Part of the value I offer with Cobalt Strike and Armitage is a workflow around the Metasploit Framework. As such, any third-party stuff I create has to integrate well into the framework and feel like a first-class part of the tool’s workflow.

When I sat down to look at this problem, I had the following requirements in mind:

  • I want to use my third-party payload with remote and client-side exploits
  • I’d like to use my third-party payload with generate executables (e.g., I want them to work with psexec, I want to be able to embed it in an existing executable, etc.)

My ideal integration point is Cobalt Strike’s listener management feature. Through this dialog, users may start multi/handlers for different payloads, configure them, and name them. Later, users may pick from these listeners when setting up a client-side attack or generating a social engineering package.


Matt Weeks covered a few options for using custom playloads in Metasploit 4. He talked about how to use the generic/custom option to embed your own shellcode. He also covered the EXE::Custom option.

The best solution to my problem is the dllinject payload shipped with the Metasploit Framework. dllinject allows me to specify a reflective DLL and load it using the http, https, or tcp stagers.

To use dllinject, I need to:

  • create a reflective DLL
  • patch the reflective DLL to make it compatible with the dllinject stager
  • deliver the patched reflective DLL to the dllinject stager

How to create a reflective DLL

A reflective DLL is one built with Stephen Fewer’s Reflective DLL Loader code. To build a reflective DLL, you will need Visual Studio 2008. The dllinject stager will not load an arbitrary DLL for you.

I tried Visual Studio 2010 initially, but the DLL would crash the process I injected it into when using it on versions of Windows before XP SP3. I spent some time tweaking the different compiler and linker options to mitigate this with no dice. Visual Studio 2008 Express is still available for free and it works, I recommend that you use it.

Next, create a project and import the Reflective DLL files into it.

You will also want to right-click the project, navigate to Properties -> C/C++ -> PreProcessor and define the REFLECTIVEDLLINJECTION_VIA_LOADREMOTELIBRARYR and REFLECTIVEDLLINJECTION_CUSTOM_DLLMAIN constants. You’re just defining these as true essentially.

This will allow you to use DllMain as the entry point once the DLL is reflectively loaded. See ReflectiveDLL.c.

Now you should be able to compile the project without any issues.

To test your reflective DLL, use the inject program included in the package.

Now you’re all set.

I want to deliver the DLL to the stager, myself…

To take advantage of dllinject, I have an additional problem. I plan to have my custom payload communicate to my web server running on port 80. I can stage dllinject from a different port, but this isn’t as clean. I’d like to deliver my custom payload to the stager from my web server. This introduces a few additional steps which are worth documenting if you need to do something similar.

Patch the Reflective DLL

The DLL inject stager will not accept the reflective DLL as-is. It must be patched. The payload handler for dllinject handles this step for us automatically. Since I want to host the DLL myself, I must patch it myself.

The code for this patching is in reflectivedllinject.rb. I’m opting to cheat and force the Metasploit Framework to patch my DLL for me. To do this, I will automatically launch a reverse tcp handler for the dllinject payload. Connect to it, download the patched DLL, and save it to a file I can host.

Here’s the Cortana/Sleep code that does this:

# connect to the reverse tcp dllinject handler
$handle = connect("", 65002);
# read a 4-byte integer stating the size of our data. I use I- to account for the byte order
$bytes = bread($handle, "I-")[0];
$data = readb($handle, $bytes);
# save to r2.dll
$h = openf(">r2.dll");
writeb($h, $data);

Deliver the patched Reflective DLL

Now that I have a patched reflective DLL I can host it on my web server and deliver it when the stager requests it. The stager communicates to a 4-character random URL. This random URL is hard coded in the stager when it’s generated. I wrote a regex in my web server and instructed it to deliver my hosted DLL for any requests that match /[A-Za-z0-9]{4}. One other note, the Content-Type of the delivered DLL should be application/octet-stream.


With that, you now know enough to deliver your own RAT or agent using the Metasploit Framework’s DLL inject payload. An eventual Cortana goal will be to create an API that allows Armitage and Cobalt Strike to act as a way to control administration tools beyond Meterpreter. I’m still deciding what that will look like, but… having an integration point to deliver a custom payload is a good start.

A loader for Metasploit’s Meterpreter

Recently, there was an interesting discussion on the metasploit-framework mailing list about the staging protocol for Meterpreter. egypt let loose with some wisdom about what it would take to write a client to download and execute a payload from a Metasploit Framework multi/handler. mihi completed the discussion by advising where to place the socket value, so meterpreter could pick it up.

The basic process is:

  1. Connect to the multi/handler
  2. Read the length of the payload into a 4 byte unsigned integer in native byte order
  3. allocate a buffer with Read, Write, and Execute access
  4. copy the socket file descriptor from step 1 to the EDI register
  5. Read the payload from the socket into our buffer
  6. Cast the buffer to a function and call it

My ears perked up at this discussion, because it’s something I knew I’d have to dig into soon. Our friends at the anti-virus companies are doing a great job of picking up Metasploit’s stager, no matter how it’s encoded or which template executable I throw at it. Trust me, I tried. Before releasing Cobalt Strike, I had this idea to license a multi-AV engine and operate a cloud service to encode binaries again and again until they were clean. The prototype of this idea was a complete failure. Since then, I’ve been meaning to investigate writing my own client.

Armed with this guidance, I wrote a quick client for a Metasploit reverse_tcp multi/handler. The code is on GitHub.

If you’d like more information on how AV is picking up executables generated by the Metasploit Framework, read Facts and myths about antivirus evasion with Metasploit

Covert VPN – Layer 2 Pivoting for Cobalt Strike

Currently, I’m debating a class of social engineering “packages” to force SMB requests against an attacker controlled system. Ideas include packages to generate LNK files, host a WPAD server, etc.

This created a bit of an identity crisis though. I see Cobalt Strike as a tool for a penetration tester to emulate the capabilities of a motivated external actor. Sadly, many awesome SMB attacks require a physical presence on the target’s network.

To put this issue to rest, I decided to build a feature to allow a motivated external attacker the ability to work as-if they are physically present on the target’s network. This feature is Covert VPN.

Deploy Covert VPN

Covert VPN is a layer 2 pivoting capability for Cobalt Strike. It creates a network interface on your system that is bridged into the target’s network through a channel of your choosing. Covert VPN can tunnel its traffic over UDP, TCP, or HTTP channels

Once an interface is active, you can sniff packets, start rogue services, use external scanners and attack tools–pretty much whatever you want.

Covert VPN is in the latest version of Cobalt Strike.