How to maximize ROI of Bug Bounties and penetration tests

Graphic from securestate.com

I’ve been doing security bug hunting, penetration tests and managing in-house bug bounty programs for various companies, for over half a decade already. During that time I learnt that it doesn’t really happen too often that hiring company knows exactly what to do with security engagements results.I’d like to help and suggest what you can do to fully benefit from what you paid for.

FYI — Later in this post I’ll be using pentester/bug hunter interchangeably. Although I know there is a quite a difference between those professions, in a context of this article it doesn’t matter.

Many companies working with security researchers don’t really utilize the potential of those people. Just patching specific issue reported by bug hunter isn’t quite the right way to do it. There is a whole lot of things you can do around it.To take full advantage of these quite expensive exercises, companies should do their homework — by which I mean root cause analysis, searching for similar issues and developing proper regression test cases.

Many don’t quite get it. Exactly this happened to scoop.it. They fixed my vulnerabilities and pretty much forgot about next steps. Either forgot or didn’t know them at all. I believe it would be good to propose a sane guideline for companies that have no idea how to approach this.

Let me give you a bit of a background on my experience with scoop.it, so you have a full understanding of what I’m talking about here and why.

A while back I got that idea to put all of my old bug hunting stories in one place. I started reading my old bug reports, clicking through proof of concepts(PoC) I used to provide, and out of nowhere my browser got blocked by alert box from scoop.it domain saying just “42”.

May look like nothing but in a matter of seconds I knew what happened — my old XSS did strike back. In the old days I used to use my favorite number — 42 — as a PoC for XSS vulns.I didn’t know why it happened though, because all issues I reported to scoop.it were fixed back then and there was no doubt about it — this shouldn’t have happened now.

In 2013 I’ve been doing a lot of bug hunting for free, mainly just to improve my skills.Quality CTFs weren’t as popular as they are nowadays so free bug hunting was quite a good way to simulate real life pentests. Bounties almost didn’t exist either at that time so most of the people I knew that were doing bug hunting, were driven mainly by gaining knowledge about large computer systems and apps, creating business relations with corporations or just trying to save the world.That’s not the main subject of this post though, so let’s move on.

One day I saw scoop.it on an advertisement and it seemed like a good site to bash around.Scoop.it was kind of a social network app that had tens of thousands of followers on Facebook and good marketing exposure, so I thought it’s a good idea to check if I can break their ecosystem.“They have thousands of customers, they store massive amount of data so let’s save the planet and ensure their users’ data is safe with them” — I thought and started grinding.

Oh boy, that was ugly vulnerable web app. I reported dozens of XSS vulns which are pretty awful bugs to have, especially in social network app.It was bloody hard to find anyone technical via their support system, but luckily after many back and forths I finally reached the tech team and managed to get dozens of vulns fixed.I verified their fixes and moved further with my life. The whole thing eaten up a couple of weeks of my life which was just about enough time spent working for one company.

I assumed that if someone reported them dozens of security vulns, they will take it from there, run internal pentests or hire 3rd party company to do so.It does seem to be a good sign that it’s time to improve security if random person on the Internet can find handful of severe vulnerabilities within days, doesn’t it?

Well, after three years I found that I couldn’t be more wrong. Not only I saw regressions on fixed issues(pic above), but they also introduced additional vulns in places that were safe in 2013.Left side of the screenshot is from a PoC video I made in 2013 and right side is the same field nowadays.

I tried to reach out to scoop.it team, but it’s as terrible experience as it was in 2013 — I sent out a bunch of emails and haven’t heard from them so I decided to go with public disclosure.They have many XSSs in other places anyway, so I don’t feel much guilty for not waiting more as other companies can learn from this. And maybe, just maybe scoop.it will feel a bit ashamed and fix their crap.

Let’s make it clear — I don’t want to point fingers at scoop.it. It’s their business and I really don’t care what they do in their own playground. There are many companies behaving the same way if not worse, I’m just using them as an example. They are just the lucky ones I found in my mailbox very recently. Most of the times I’m far from judging anyone, but seeing such posture I surely won’t do another free web pentest for them.I just hope this will somehow reach them and other companies with lack of infosec knowledge and educate them a bit.

Now that you see this shit is real, here are a couple of things you can do to ensure you spent your money well.

From the day one, build a relationship with pentesters. It really isn’t easy to find great pentesters, so once you’ve found them — make sure you’re in good relations with them. You never know when you’ll need next pentest and how big your budget is going to be so it’s good to know people.Sometimes you don’t need full pentest, just a small assessment and I’m sure there will be someone to help you figure that out. Long term relationships with infosec pros are good to have.Of course you don’t need personal relationship if you want derpy security checklist for business show-off for prospects etc, but I’m talking here about real security. If all you want to have is a checklist for a couple of bucks then who cares.

During actual tests, be responsive to pentesters. If they ask you about your ecosystem details, tell them as much you know so they can adjust to your needs. Tell them what’s really important for you, what data has business value and what are mission critical components of your network/application. It’s completely fine to share with them your opinion on strong and weak spots of the target network.

When an engagement is done don’t limit yourself to taking paper report with raw data. Have a chat with those folks. Ask them about opinion on your architecture and write down less technical advice they may have. Pentesters are one of those people who you can listen hours and still learn something new. They have seen lots in their lifes, so they often have much to say and will be happy to help if you just ask for it.Don’t be a jerk, don’t send formal letter demanding additional information. Kindly ask for it. If you’re too formal then you’ll most likely receive only what was on a contract and nothing more.It’s no one’s business to help someone who can’t appreciate it.Schedule a short call with people who actually were pentesting your stuff and speak with them.Don’t filter their thoughts and ideas through an account manager or a support agent.If you take their time, but on next pentest they find the same issues they’ll be pretty pissed off and likely won’t waste private time on you anymore. So if you expect someone to spend extra time, be cool and at least try to address raised issues.This is a way to build relationships I mentioned earlier.

Really listen and then execute. Don’t be mad at pentesters because they showed you how shitty is the code you wrote. Humility is really important here and you shouldn’t take anything as a personal offence. If you messed something up, be grateful that you can now learn how to fix it.

All ^these will improve quality of your products and leave you with a feeling that money was spent well. You should look at pentests as an opportunity, and not as an additional burden in your day-to-day work. If pentests are pain in the butt, then you’re doing something wrong.

While InfoSec management is busy with above, let’s take a look on what are the other technical things to be done.

Learn technical aspects of the issue, why did it happen in the first place and how you can prevent it from happening in the future — this is called root cause analysis.Maybe you need to create a secure framework, maybe change coding guidelines or do an in-house training for specific development team. It can be that you’re using vulnerable 3rd party library and package upgrade will do.Whatever the reason may be, always ask the question — what can we do now to make our codebase safe against this for the future and how do we ensure that even new-hires won’t reintroduce the issue.If this can’t be prevented in automated fashion, then figure out how do you monitor it so you can immediately jump in, if someone reintroduces the issue.

Look for similar issues in the same or other products which weren’t in the scope of a pentest. Sometimes by replying the most simple test cases pentesters did, you can find other bugs.You should look for that especially in those applications/features that were developed by the same people who wrote vulnerable code in pentested application.

Create solid test cases to cover reported bugs. You need to have solid regression testing in place. It’s just stupid when I come to the same company after months/years and find the same bugs in the same places. It wastes pentesters’ time which he could utilize on doing actually valuable work.You can write regression automation to do verification after each release of the product.You can use free tools to scan for those low hanging fruits. This is crucial — don’t waste time and energy on regressions. Do it well once so you don’t need to worry about it. Learn from scoop.it examples provided above.Security regressions happen, quite often actually. It doesn’t take much to reintroduce an ugly bug. One missed commit or a commit of new-hire who doesn’t know yet your secure coding guidelines can make a lot of mess.

Finally, once you are done with everything else, have a two brief sessions with two audiences to officially close pentesting engagement.One group should be — architects/managers/product owners and have their commitment on next steps. Security is never ending game, you need to keep finger on a pulse all the time.Another good thing to do is a presentation to engineering teams with the summary of lessons learnt from the pentest, so others can learn something new. Either to use it in your business or for engineers’ personal benefit. Any infosec education is always good so share whenever possible.

Additional thing I also recommend to do once in a while is reviewing test accounts left by pentesters. They drop tons of payloads during their tests so this is pretty good regression testing resource. Again — see my scoop.it’s screenshots. If they took a look on content I published on my tests account, they would know they messed some stuff up.

To summ it up — here are the action items I recommend you to take during and after penetration test/bug bounty cycle:- provide all information to pentesters as requested.- don’t play weird games and don’t block pentesters as they do their job. You’re all in the same team and should share the common goal to make your organization safer- be humble and thoroughly follow pentesters recommendations in terms of issues remediation- find the root cause of every single issue and learn what you can do better next time- look at the big picture and think about other places where the same issue may exist but wasn’t found by pentesters- have your engineering teams learn from pentesters and study pentest report- ensure developers have complete understanding of all issues- cover identified bugs with solid regression test cases- once in a while check changes made by pentesters, visit their profiles used for testing to catch unexpected regressions

While it may sound like a ton of things to do, it actually isn’t that much. How often do you get solid pentest? Not that often, so you can do a full cycle of above once or twice a year — depending on your business needs.

When it comes to bug bounties, it is usually easier as you get small amount of issues constantly in opposite to one big report with tons of vulns in it — unless you’re size of Facebook or Google and receive hundreds of bug reports per day, but then this article isn’t for you anyway.So for bug bounties you likely can skip some of those steps according to your needs.

It’s a raw guideline which worked for me and a couple of companies I helped. Hopefully it can help you either. If your only excuse used to be — “I don’t know how” — then it doesn’t work anymore.

Pick whatever suits you and adjust, just don’t limit yourself to only fixing reported issues. Go beyond that and do more.

Enjoy.