Tester Focus and Bandwidth

We have discussed a lot in the last five articles, and each of those topics have a great impact on the quality of a pentest, however, no other topic has a greater impact than a tester's focus and bandwidth. Without the appropriate amount of time, free of distractions, a pentester will not be able to cover the scope, devote dedicated time to investigate vulnerabilities, or dive deeper where the clues suggest.  

Instead, testers will compensate for this lack of bandwidth in the following ways:

Often, pentesters will rely more and more on automated tools when stretched thin. While automation is often a good thing, cutting corners with tools that lack validation will only create more customer confusion down the line.

Another way they may try to conserve time is by not covering the entirety of the scope, opting instead to focus on what they perceive to be the most important areas. 

An unfortunate side effect is that often collaboration with customers and fellow testers will be reduced.

Finally, when finished with their investigation, pentesters may need to move on quickly and will thereby spend less time and effort on reporting.

While, in a vacuum, these may not seem like too big a concern, together and in the full context of the pentest they can have detrimental effects on both the quality and value of the work. The key here is making sure that all testers have exactly what they need - from a well defined scope, including exclusions, to the right level of access. These items need to be in place before the test begins so that  if issues arise during the test, they are resolved as quickly as possible and without the need for the tester to engage. In fact, most negative impacts on a test occur when the testers have to chase down  these issues. Instead, we should have others working on this, and ideally resolved before the test even starts.

“If the preparation and operational approach does not have a mature process, it will negatively impact the focus and effort a tester is able to put forth on testing because they will have to split their time on tasks that should have been handled.”  - Jonathan Stines, Senior Manager, Product Security M&A at Salesforce

We take so much time to align on expectations during the sales cycle: scoping appropriately, understanding the needs of our customers, involving the right internal team members for knowledge transfer, staffing the right testers based on experience and customer needs, and kicking off the test in time. To let all that work go to waste by asking our testers to chase it all down again is mind numbing at best. Instead we should have a system and people in place to help when misalignments occur during the delivery of a pentest. We already have Technical Project and Customer Success managers work to enable testers to dig into these needs for them. We need to also have an expectation of customers (internal and external) to be responsible for these items, after all, they own and know the details much better than the team staffed to deliver the test.

Here are some examples of common hurdles and who, in my opinion, should be working to resolve them:

• Access to the testing environment is not available.
• Credentials have either not been provided or are not operational.
• The customer revises the scope during the test period.
• The customer is non-responsive.
• An environment is down or slow to respond.
• The tester is non-responsive.
• The tester is outside the scope of the test.

When access to the environment is blocked or credentials have not been supplied, the tester should notify the Technical Project Manager (TPM) to seek assistance. While this is the responsibility of the environment owner, the TPM assigned can work with customers to ensure these technical issues are resolved.  Similarly, when the customer is either non-responsive or is asking to test outside of the agreed-upon scope or their environment becomes unresponsive, the tester should reach out to the TPM who will work with the Customer Success Manager (CSM) to re-align scope and ensure the customer is available and responsive.  When the customer feels the testers are either non-responsive or are not providing appropriate updates or testing, it is crucial that this be brought up right away to the TPM.  They can then work to ensure testers have what they need and are indeed testing to the customers' expectations.  If alignment is not quickly reached, the TPM team should work to replace testers as appropriate.

This area may seem obvious, however, we see a lot of issues come up during tests when testers do not have what they need or their focus is otherwise split.  It is pivotal to ensure testers can focus and have the space that allows them to use all of their knowledge and experience in the appropriate technical areas. This will ensure the level of testing is adequate and properly aligned to the scope, technology stack, and customers’ expectations.

Next up in my article series, I’ll discuss the need for in-test communication and how to align when things don’t go as expected. Thoughts on this series so far? If you have anything to add to my list or if I missed anything you find absolutely essential when hiring or vetting pentesters please send me your thoughts at [email protected].