Joel Spolsky is one of my favorite writers on the topic of software development. He coined a 12-step Joel Test to determine if your company had sane development practices. A lot of these are very common sense, but I’d like to share a little about how I work and this seems a good way to do it.
1. Do you use source control? Yes
I have a private git repository for development of Cobalt Strike. Armitage uses Subversion still (mostly because I’m too lazy to move it from Google Code).
2. Can you make a build in one step? Yes
Yes. I use a fairly standard Apache Ant build file for my Java projects. I think Ant is cumbersome for some things, so I tie multiple build steps together with a shell script that carries out all steps to create trial and production Cobalt Strike packages to deliver to my end users.
Cobalt Strike has a couple of sub-projects written in C for Windows and Linux. These sub-projects each have their own build process for their respective development environments. I’m toying with using a cross-compiler to build the Windows code where I can get away with it.
One example where I can’t use a cross compiler is Beacon. It is tied to a specific Visual Studio environment because of the Reflective DLL injection code it depends on.
3. Do you make daily builds? Sort of
I make builds when I finish a bug fix or make intermediate progress on a feature. On the days I write code, I am doing full builds of Cobalt Strike multiple times.
4. Do you have a bug database? No
I don’t have a formal database. I still track bugs and notes in a text file. When I start growing development beyond myself, I will pick a simple solution to work with. I do not have a bug database for customers to report bugs into. I still handle this over email.
Armitage uses the Google Code bug tracker and I stay on top of this.
5. Do you fix bugs before writing new code? Yes
If I can reproduce the problem and I’m confident I know what the bug is, I will fix it before I work on a new feature. I prefer having a few features that work extremely well over a myriad of features that half work. When I have a bug report, I will go quite far to try to reproduce it. I have an MSDN subscription and I use it to replicate environments when I need to. However, I’ve had bugs that are too hard to reproduce. Sometimes, I’m put in a situation where I have to wait for more clues before I can track down the bug.
6. Do you have an up-to-date schedule? No
I do not work on a schedule. Generally, I aim for a release every 1-3 weeks. I aim to have ~10 logged changes per release. I know which features I intend to build and they have a prioritization. I do not tie these features to specific dates because some genius suggestion, customer request, or bug report may come along and take priority.
In November, I was contacted about Cobalt Strike’s spear phishing tool. I had a trial user who really needed the ability to authenticate to an SMTP server and he was willing to provide access to his infrastructure for testing. I dropped my current development tasks and took advantage of the opportunity to add SMTP authentication to Cobalt Strike’s spear phishing tool. I had always planned to add this, but it became a higher priority when someone asked for it.
7. Do you have a spec? Yes
Another classic Joel Spolsky series is Painless Functional Specifications. In this short series, Joel describes how to write a functional specification to describe a product. I wrote a specification for what would become Cobalt Strike and built the product from it. This spec helped me build my initial product, but I don’t use specs for maintenance. Sometimes, when I’m planning to build a significant feature, I will write a spec and send it to my trusted board of users who have the opportunity to chime in on it.
8. Do programmers have quiet working conditions? Yes
I work from home where I have a comfortable and quiet setup. Sometimes I listen to music, but often times I prefer to work in crisp silence. I once worked for a defense contractor where phones would ring and people would have meetings on their speakerphone in the cubes around me. This was a miserable experience. I will never do this to a programmer. 🙂
Sometimes, I work from Affinity Lab. Affinity Lab is a coworking space in Washington, DC with ~60 companies. It’s Strategic Cyber’s official business address too. Affinity Lab is less quiet, but I go there when I need to be around people and accomplish administrative tasks. I sometimes write code there too, when the change of scenery is enough to jumpstart the brain.
9. Do you use the best tools money can buy? Yes
This is a very strong yes. Anything I can spend to grow my business, improve my product, or make myself more efficient is a write-off. Taxes incentivize me to spend! If I don’t spend the money, I get to keep a portion of it. If I spend it, the full amount goes to grow my business. I commit money to software, hardware, contractors, and licensed technology quite regularly. I try not to be foolish with this though. For example, I’m on the fence about upgrading my MSDN Operating Systems subscription to a full MSDN subscription. The full subscription is quite expensive, so I don’t know if I will extract that value out of it. Generally though, when a case can be made, I’m quick to invest back into my business. I work off of a maxed out MacBrook Pro and an Apple Thunderbolt display.
10. Do you have testers? No
Yes, I have testers, they’re called users. When I have enough changes to cut a release, I do so. Cobalt Strike changes make it to customers and trial users very quickly. When the product was under private development, I had a team of beta testers who acted as a stand-in for the customers I would eventually get. I still use this team as an advisory board.
Generally, my releases focus on one feature area making them easier to test. I don’t believe in monolithic point releases. Sometimes, I will introduce a bug or error, and if it’s a show stopper, I recut the release or create a special build for the customer that needs it. These cases are very rare.
In terms of user feedback, Cobalt Strike benefits greatly from building on Armitage. I try to keep the code for the two interfaces as close as possible. This gives Armitage’s significantly larger user base a chance to chime in on something that will improve both products.
I spend a lot of time testing the foundation I build on too. Cobalt Strike builds on the Metasploit Framework which is one of the fastest moving projects I’ve ever touched. Something that works today, could change, and ripple into my product in an unexpected way. I also have the pleasure of serving a user community that likes to use the latest development version of the Metasploit Framework. Talk about a catch-22!
I mitigate this scenario with unit tests. Before I cut a Cobalt Strike release, I update to the latest version of the Metasploit Framework. I run several Cortana-based unit tests to exercise the Metasploit Framework, automatically hack into hosts, and do basic post exploitation. The unit tests help me test the Metasploit Framework and exercise my interfaces to it. I also exercise a few basic scenarios. Occasionally, I stage a node on Amazon’s EC2 and conduct a penetration test against an enterprise network lab environment I host on a Shuttle PC.
Each release, I publish the version I tested with my release notes for customers to match their environment to. If someone chooses to update to the latest version of the framework’s code. That’s up to them. If they encounter a problem, they can downgrade to the last tested version.
Of course the best testing is real world use. I don’t do services / pen testing work now (development is a full time job!), so >exercises are the closest I come to real world use with my tools. When invited, I play red team in exercises and war games. These opportunities provide valuable feedback that helps me make my tools better. Generally, I spend March and April doing nothing but exercises. These months are so busy that I sometimes leave at the tail end of one exercise to catch a flight to the next one.
11. Do new candidates write code during their interview? Not Applicable
I’m not interviewing developers yet. When that time comes, I will not hire from a typical interview. I will carve out a small project, hire someone on a contract basis, and see how well we work together. This will answer the questions that an interview is ill-suited for. I’m okay risking money on a test project to see how well someone works out.
12. Do you do hallway usability testing? No
The Joel Test asks, how often do you observe a stranger trying to complete a random task through your software? I do usability testing, but I don’t do it by pulling people out of a hallway. I do it through classes. I teach two classes. I teach a free 4-hour workshop on Armitage and Cobalt Strike at various conferences. When hired to, I teach the Advanced Threat Tactics course.
I don’t get a lot of usability feedback from the 4-hour workshop. Generally there isn’t a lot of time for the labs and the labs are very scripted. However, I sometimes receive a gem of a suggestion or see where something could be made intuitive.
The usability goldmine for me is the Advanced Threat Tactics course. The class ends with a capstone exercise. The exercise takes place in my enterprise network lab environment. The environment is seeded with data and services to create the sense of a living world. I put my students into teams and I assign each team one of four goals. The students are expected to get a foothold in the lab environment and iterate through the network attack process until they accomplish their goal. Some of the goals are very open-ended (e.g., you’re a hacktivist, expose ACME for their wrong doing). The exercise is where I observe how well my toolset and its workflow clicks with users.
The host labels feature added to Cobalt Strike and Armitage last month came from a January run of the Advanced Threat Tactics course.
Advanced Threat Tactics is so effective for usability testing, that I also have a private invite-only session I run. I ran it twice last year and I plan to run it later this year. During this private session, I invite a mix of people. I always make sure I have two people who have never hacked before. I also invite some of the most experienced penetration testers and researchers I know. This is an intimidating group to “teach”, but it’s a useful way to see how different skill levels approach the toolset.
My goal is to make sure Cobalt Strike is valuable to experienced pen testers without forcing them to learn a new way to do everything. The fact that Cobalt Strike provides full access to the Metasploit Framework console helps greatly with this. I also watch to make sure that novice users are able to get things done with Cobalt Strike after a reasonable amount of instruction.