TL;DR Rapid7 wrote a blog post claiming that their exploits are better. I think the Metasploit Framework’s coverage is fine, but some other vendors do better with AV-safe client-side exploits. Over time, memory corruption exploits will become less relevant to penetration testers. Let’s talk about how penetration testing is evolving, not who has “the best” exploits.
Let’s talk about the players in the penetration testing software field. There’s Core Security, Immunity Inc., Rapid7, Saint Corporation, and my outfit Strategic Cyber LLC.
Generally, we act like politicians on the campaign trail fifteen months before an election. We either act like the other parties don’t exist, take very light jabs, or look for ways to cooperate.
Today, Rapid7 has a blog post on Open Source vs. Pay-for-Play exploit packs. In this post, a Product Marketing Manager at Rapid7 makes his case as to why Rapid7’s hybrid open source and commercial model yields more reliable and relevant exploits than other commercial-only contenders.
Nico Waisman, a Regional Manager and accomplished Security Researcher from Immunity Inc., had an interesting reaction to this particular post:
https://twitter.com/nicowaisman/status/284354002710327296
The end of Rapid7’s blog post invited opinions, so here’s mine.
First, the Rapid7 blog post labels Metasploit Framework contributors like myself as the Rapid7 Security Community. I am not a member of the Rapid7 Security Community. I am a contributor to the Metasploit Framework. Refer to us as the Metasploit community, please.
Next, I can see where this post is coming from. Core Security labels their exploits as commercial grade. I perceive this as a light jab against the open source Metasploit Framework. I read Rapid7’s post as a response to the commercial grade label.
After I published this post, a Core cofounder and former CTO responded to the commercial grade label. My perception of this label wasn’t the intention of it. Thanks for clarifying. The tweets are below.
https://twitter.com/4Dgifts/status/284439074767519744
https://twitter.com/4Dgifts/status/284441225640165376
For remote service exploits, it is my belief that all products have similar coverage. The most common remote service exploit to demo is 2008’s ms08_067_netapi. As we turn near 2013, I believe all products have this one well covered. There are other useful remote exploits, but I’m not aware of a magical remote service exploit in any product that by itself, makes the product a must-have. No one has an edge here.
Now, if your work involves penetrating systems, not verifying remote service vulnerabilities, then client-side attacks matter to you. Again, I believe the Metasploit Framework has good coverage of client-side attacks. However, some of its pure commercial competitors have an edge in this area.
The Metasploit Framework’s client-side attacks are eaten alive by anti-virus products. The problem is so bad, that part of my roadmap involves porting a few key attacks to Cobalt Strike so I can give my customers options. Core Security tries to stay ahead of some anti-virus products. I haven’t read a blog post from Immunity and Saint about this topic, so I can’t speak to how they handle this problem here.
Now of course, anyone can modify the Metasploit Framework’s exploits to evade an anti-virus product and submit a pull request. This is rare though. My guess is that if someone modifies a Metasploit Framework client-side exploit they hold onto it to get the most use out of their modification. I expect pen testers to have the skill to modify an exploit to pass AV, but many penetration testers find themselves squeezed to mimic a threat in a tight timespan, anything we can do as vendors to help them is welcome.
Access strategies change over time though. 10 years ago, the game was memory corruption exploits against remote services. 4-5 years ago, the game shifted to memory corruption exploits against user applications (client-side attacks). Organizations continue to become smarter about vulnerability and patch management. Software will continue to become harder to exploit. Despite this progress, organizations today get owned with executables disguised to look like PDFs.
As memory corruption exploits become less relevant, we must focus on reconnaissance and look for opportunities to abuse information disclosures, design flaws, configuration mistakes, trust relationships, and the behavior of systems. David Kennedy’s Social-Engineer Toolkit is an example of this. Its Java Signed Applet attack uses existing functionality to get access and it is constantly updated to stay ahead of anti-virus.
I believe organizations will one day assume an attacker can get a foothold. At that point, a pen tester will add value by helping an organization assess their ability to detect, frustrate, and contain an attacker. Our tools will need to evolve to better support this service offering.
How should they evolve? Let’s start with these questions: How do you maintain access to a system without tripping an alarm? How do you establish Command and Control when facing a very restrictive firewall and web proxy server? How do you carry out those neato insider threat attacks from a foothold? How do you quickly identify privilege escalation opportunities? How do you automate your engagement? How do we as vendors better help our pen testers match capabilities to opportunity? How do you manage large-scale penetration testing infrastructure to better mimic an adversary with control of multiple hop points? These areas are stagnant in penetration testing tools and ready for innovation.
As we get better at mitigating vulnerabilities, in what other ways will pen tester service offerings evolve? As more organizations trust cloud services, we’re seeing social engineering attacks that take advantage of differing vendor policies about which information is safe to give out vs. which information authenticates you. Who is working to address this?
Successful attacks are just as much about a lucky opportunity from good timing as they are about good products and planning. A two-week window is hit or miss in terms of opportunity. What would an economical year-long penetration test look like? How can we as vendors better support the next penetration testing service models?
Attackers continue to evolve. Penetration testing is slowly evolving. We’re not away from the vulnerability verification mindset yet, but we’re getting there. I believe that swinging swords around who has better exploits is irrelevant. Vendors who want to lead should discuss where the field is going and work to help it get there.
"I have a dream that hackers will one day not be judged by the color of their hat, but by the content of their exploits." – @jack_daniel
— Humor/Parody (@SecurityHumor) December 27, 2012