Picture from http://blog.linguistica-international.com/4-ways-translation-can-send-your-roi-through-the-roof/ Having worked on both sides of the fence, I want to share my biggest lessons learnt during my career that entailed: being a penetration tester and red teamer being an accomplished bug bounty hunter working as an internal QA engineer, Security Engineer and Security Architect a’ka blue teamer running and maintaining bug bounty program for a handful of companies worked as a head of security reporting to the board of directors for maximum ROI of security initiatives, including penetration tests and bug bounties Here is a list of action items I recommend you to take during and after penetration test/bug bounty cycle. This list is based on the most common gaps I’ve noticed at the companies I’ve worked with and the things that to me made a huge impact ROI-wise. By implementing these steps, you’ll be able to get much higher return of investment from penetration tests. If you’ve already spend your money, let’s make sure you’ve spent it well and squeezed max out of it. Provide all information to pentesters as beforehand. You want to make sure that pentesters don’t waste too much of their time on reconnaissance. Provide them with documentation on the product, video recording how your product works, and a list of APIs and endpoints you want them to test. In general, penetration testers will know what to go after, however if you have hidden APIs, or want to speed up the process, it’s better if you provide them the information beforehand. 2. Don’t play games and don’t block pentesters as they do their job. You’re all in the same team and should share the common goal to make your organization safer The mindset shift is a huge thing. I’ve met number of security teams, that were literally fighting with external pentesters, because they were afraid of their position and ego. You hire pentesters to do the work for you and to provide maximum value to your business, so it’s not smart to waste your money while you argue with pentesters and block their testing environments. Depending on corporate culture, it may be a good idea to have a separate — such as compliance/audit — team to coordinate external testing, to avoid conflict of interest. 3. Be humble and thoroughly follow pentesters recommendations in terms of issues remediation Don’t fight it, when pentesters provide you information about the issue severity and its guidance. It may sometimes happen that with internal business knowledge, you can perform better risk analysis and assess that the issue has different severity for you. However, keep in mind that most often than not, pentesters know what they’re doing and it’s not their first assessment(I hope!) so when they provide you an information about the issue, it’s likely to be legitimate. It’s generally good idea to follow their remediation guidance, to make sure you’ve addressed the issue in-depth and then request re-tests. 4. Find the root cause of every single issue and learn what you can do better next time. Stay focused on the big picture and think about other places where the same issue may exist but wasn’t found by pentesters. As opposed to simply fixing individual bugs, dig deeper into your codebase to find out if the same issue doesn’t happen in other applications. Then review your software engineering practices to see what could be done better to stop those bugs from appearing in the first place. 5. Have your engineering teams learn from pentesters and study pentest report Don’t keep it all for yourself and have everyone learn from the report. Use it as an interesting exercise and learning experience. Pentests at most companies happen once or twice per year, so why not maximize the outcome. Thanks to that software and QA engineers can learn how to apply that knowledge in their day to day work such as to add it to existing test cases. 6. Ensure developers have complete understanding of all issues You need to take ownership over the reports and don’t just throw it at employees. You want to make sure everyone has good grasp of security awareness, so they can address them well as well as use that knowledge in the future to build more robust apps. If something is complicated it won’t get easily remembered and put into action. 7. Cover identified bugs with solid regression test cases You don’t want pentesters to find the same bugs over and over again. Not only it’s a waste of money, but that’s an additional exposure, because if regression pops up, attackers may abuse it before your next pentesting engagement. We’re performing penetration tests to improve our products and company, so if you’re not going that extra mile, you’re leaving a lot on the table. 8. Once in a while check changes made by pentesters, visit their profiles used for testing to catch unexpected regressions. It makes a lot of sense to clone the activities that were performed by pentesters, and build standalone testing scenarios out of it. When you login to the pentesters’ account in the testing environment, you may notice much more bugs than they reported. Security testers, focus on reporting security issues, and most often don’t waste their time on reporting UI/UX issues, that may have been identified during security tests. So just checking their environment to see if e.g. not expected encoding didn’t break the UI, will be beneficial, and again will be something to share with internal QAs so they can improve their test cases. 9. Review logs to catch unidentified bugs Some things aren’t exposed to the user, which means that security testers may have touched some fragile system and maybe had even broken it, but they weren’t aware of it. If you review the logs, you may be able to find some unhandled exceptions, Internal Server Errors, and other indicators that may allow you to improve robustness of your code and systems. Thanks to making yourself comfortable with the logs generated by penetration testers, you’ll be also able to create security monitoring around your access/application logs, to detect potentially malicious hacking attempts. If some keywords happened to appear in logs only during pentesting engagement and never before, then attackers may perform tests in the same fashion. If you create alerting around that, you may be able to spot the attackers and allow yourself to respond to the threat. In my career I’ve met only a handful of companies that taken such wide range of activities during penetration tests. If you follow this guidance, you’ll optimize your spendings and should be more satisfied with general return of investment in security tests.
