Commissioning a Penetration Test That Produces Fixable Findings

If your acceptance committee asks for a penetration test report before a new system goes live, your first decision is not which tester to hire. It is what you will ask them to test, and under which conditions. The report you get back can only be as useful as the scope you wrote at the start.

Take a hypothetical case. An organization receives a report dozens of pages long, with five High findings. None of them says which URL the issue was found on, which account role was used, or what has to change before the finding counts as closed. The development team reads it end to end and still does not know where to begin. Every gap in that case could have been closed while the scope and the reporting requirements were being written.

This article is written from the buyer's side. We will not cover how systems are attacked or how any tool is used. We cover what you, as the system owner, have to decide yourself: which kind of engagement to buy, how to write the scope, how to read severity scores, how retesting works, and who owns the fixes.

How a VA differs from a penetration test

A Vulnerability Assessment (VA) looks for known vulnerabilities across as wide an area as possible. It relies mainly on scanning tools, and a tester then removes the false positives. The output is a list of vulnerabilities that may exist in the system.

A penetration test goes deeper. The tester tries to prove whether a vulnerability can actually be exploited and, if so, how far it reaches in terms of data or privilege. This work depends more on human judgment than on tools, so in the same amount of time it covers a narrower area than a VA does.

The two are not substitutes. A VA suits recurring checks that catch newly disclosed vulnerabilities in the software you run. A penetration test suits moments when the system changes significantly, such as before go-live or after adding a module that handles sensitive data. Name the engagement type explicitly in your TOR. If it only says "security testing", each bidder may read it as a different kind of work, and the proposals will not be comparable.

If that go-live also replaces an older system, validating the migrated data before cutover is separate work that needs its own plan. We covered it in Complete Is Not Correct.

Write a scope that tells the tester what to test

The scope directly shapes the quality of the report, because a tester can only report on what they were authorized to touch. We recommend putting five things in writing.

• In-scope assets. List them: domains, API URLs, and the mobile app build under test. Be just as explicit about what is out of scope, especially third-party services your system calls, which your organization has no authority to let anyone test.

• Authenticated or unauthenticated. Testing without an account only shows what an outsider sees. If the system has several user roles, prepare test accounts for every role, so the tester can check whether one role can reach another role's data.

• Test windows. State the permitted days and hours, and the periods when testing is not allowed, such as month-end close or peak usage.

• Production or staging. Staging reduces the risk to a live system, but the results only reflect reality if staging is configured like production, including software versions and service account permissions. Wherever the two environments differ, record that difference as a limitation in the report.

• Rules of engagement. A document both sides sign before work starts. It names each side's emergency contact, the conditions that stop testing immediately, such as unusual slowness or encountering real personal data, and how evidence is stored and deleted after the engagement.

You also need to decide how much information to give the tester. Handing over API documentation and an architecture diagram up front lets the tester spend time looking for weaknesses rather than guessing at the system's structure. The trade-off is that the results say less about what an outsider with no inside knowledge could do. If your committee's question is how well the system holds up against outsiders, write that goal into the scope directly.

If the system will expose APIs to automated systems or AI agents, their accounts belong in scope too. We wrote about the gates an organization should pass before giving AI agents access to real systems in this article. All of the conditions above belong in the TOR from its first draft.

Methodology references: WSTG is a testing guide, the Top 10 builds shared awareness

"Testing will cover the OWASP Top 10" sounds thorough. But OWASP itself describes the Top 10 as a standard awareness document for developers and web application security. The most recent edition OWASP lists is the Top 10 2025 (source: OWASP Top 10, checked 30 September 2026).

Our view is that the Top 10 works well as a shared vocabulary with executives, but it should not be the only acceptance criterion for a test. It groups risks; it is not a list of what has to be tested.

If you want to reference a testing methodology, name the OWASP Web Security Testing Guide (WSTG), which is a guide to testing web application security directly. Ask the tester to attach the checklist they actually used as an appendix. You can then see which items were tested, which were not, and why.

What a CVSS score tells you, and what it does not

When a report scores findings with CVSS, the first question is which version was used and which metric groups went into the score. The current version is CVSS v4.0, published by FIRST in November 2023. It has four metric groups: Base, Threat, Environmental and Supplemental.

The CVSS v4.0 specification states that the Threat and Environmental metrics are specified by consumer organizations, because they are best placed to assess the potential impact in their own context. It also recommends that consumers enrich the Base metrics with these two groups, so the score becomes a more comprehensive input to risk assessment.

The specification also defines labels for this. CVSS-B is a score from Base metrics only, CVSS-BT adds Threat, CVSS-BE adds Environmental, and CVSS-BTE adds both. In your TOR, ask for every score in the report to carry the matching label.

The practical reason is that an external tester does not know which of your systems hold personal data, or what other controls are already in place. In a hypothetical example, a CVSS-B 7.5 finding on a test server with no real data may matter less than a 5.3 finding on a page that displays your organization's customer data.

The CVSS v4.0 severity scale rates 9.0 to 10.0 as Critical and 7.0 to 8.9 as High. That helps with a first ordering, but deciding what to fix first remains the system owner's job.

What a report you can act on must contain

The Reporting chapter of OWASP WSTG v4.2 lays out an example report structure. The introduction covers the scope, the limitations of the test, and the timeline. An executive summary follows, focused on business impact.

In the findings section, each item has a reference ID and title, an assessment of likelihood and impact, a risk rating, a description with the steps needed to reproduce it, remediation guidance, and references. The appendices hold the test methodology, an explanation of the severity rating, cleaned-up tool output, and the checklists used.

The WSTG says itself that this is one possible approach, and that it fits consultancy-style reports best. We suggest using it as the base for the report requirements in your TOR, adding two things we consider necessary for a development team: every finding must name the affected asset and the account role used, and it must state what has to change before the finding counts as closed.

Read the limitations section carefully. If part of the system went down during testing, or test accounts for some roles arrived late, the report should say so here. That is how the committee knows how much ground the results actually cover.

Retesting, and who owns the fixes

NIST SP 800-115, published in 2008, was written to help organizations plan and conduct security tests, analyze findings, and develop mitigation strategies. In that sense, testing does not end on the day the report arrives.

Decide in advance that every finding gets an owner who is a named person, not a team, and a due date set by your organization's own policy for each severity level. If your organization has no such policy yet, write one before the first report arrives.

In the testing contract, state how many retest rounds are included and how many days after the report they must happen. The retest report should give a status for each finding, such as fixed, partially fixed, not fixed, or risk accepted. A risk-accepted finding should carry the system owner's signature, not the developer's.

If the system is under a maintenance contract, check whether fixing findings from a test report counts as work under that contract, and which severity scheme sets the fix times.

A limit we should state plainly

Writing a scope this detailed takes real time from your internal team, especially preparing accounts for every role and making staging match production. If your system is an information website with no login and no personal data, one round of VA plus a configuration review may be enough. A multi-role penetration test earns the effort when the system has several permission levels or holds data whose exposure would cause real damage.

Objections we often hear

One question we hear is whether it would be better to let the tester, as the expert, define the scope. The tester's knowledge is valuable for designing how to test. But only your organization knows which data matters, which services belong to third parties, and which periods are off-limits. Let the tester propose a scope, but the approval signature should belong to the system owner.

Another belief is that a report with no serious findings means the system is secure. A report can only vouch for what was in scope during the test window, and the limitations section is there to spell out that boundary. Code deployed after the test date is not covered by that report.

Where to start

1. Check the requirements of the regulator for your sector first. Requirements on penetration testing and its frequency differ by sector, so this article gives no figures. Read them in your regulator's own published rules.

2. List your assets, user roles, and the third-party services the system calls. That list becomes your scope and your out-of-scope list at the same time.

3. Write the required report contents and the number of retest rounds into the TOR before it is published, rather than negotiating them after the first report arrives.

4. Name the owners of the fixes and the person with authority to accept risk before the report is delivered.

If you keep only one thing from this article: the quality of a penetration test report is set on the day the scope is written, not on the day the tester starts. A clear scope and a named owner for each finding are what turn a report into fixes that actually get finished.

If your organization is preparing a system for go-live and would like to talk through the scope with the development side, you can reach us at 088-983-9386 or [email protected]. Our office is in Bangkapi, Bangkok. You do not need complete documents before we talk, and there is no need to decide quickly. You can find more about us at superdev.tech.