NullPointerException When Generating Commercial Tools Average Scorecard #268

Summary

BenchmarkUtils v1.3 throws a NullPointerException when attempting to generate the Commercial Tools Average scorecard if 2+ commercial tools are included in the results.

Environment

• BenchmarkUtils Version: 1.3 (also present in current main branch as of 2026-02-09)

• Test Suite: BenchmarkJava v1.2

• Build Tool: Maven 3.x

• JDK: 17

Steps to Reproduce

1. Run BenchmarkUtils scorecard generation with results from 2 or more commercial security tools

2. Ensure all individual tool result files parse successfully

3. Execute: mvn org.owasp:benchmarkutils-maven-plugin:create-scorecard -DconfigFile=<config.yml>

4. Observe error after "Tool scorecards computed." message

Expected Behavior

BenchmarkUtils should successfully generate:

• Individual tool scorecards

• Vulnerability comparison scorecards

• Commercial Tools Average scorecard (comparing commercial tool performance)

• Overall summary scorecards

Actual Behavior

Process fails with:

Tool scorecards computed.

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

Error: Failed to execute goal org.owasp:benchmarkutils-maven-plugin:1.3:create-scorecard

Caused by: java.lang.NullPointerException

    at java.io.Reader.<init> (Reader.java:168)

    at java.io.InputStreamReader.<init> (InputStreamReader.java:112)

    at org.apache.commons.io.IOUtils.copy (IOUtils.java:1430)

    at org.apache.commons.io.IOUtils.toString (IOUtils.java:3406)

    at org.owasp.benchmarkutils.score.BenchmarkScore.generateVulnerabilityScorecards (BenchmarkScore.java:966)

Root Cause Analysis

Location: plugin/src/main/java/org/owasp/benchmarkutils/score/BenchmarkScore.java at line 966

Problematic Code:

// Line 962-966 (current implementation)

// Resources in a jar file have to be loaded as streams. Not directly as Files.

InputStream vulnTemplateStream =

        CL.getResourceAsStream(scoreCardDir + "/commercialAveTemplate.html");

String html = IOUtils.toString(vulnTemplateStream, StandardCharsets.UTF_8);

Issue: The code concatenates a File object (scoreCardDir) with a string path, resulting in an invalid classpath resource path like:

• Generated path: results/scorecard/commercialAveTemplate.html (invalid classpath resource)

• Expected path: scorecard/commercialAveTemplate.html (valid classpath resource)

When getResourceAsStream() receives an invalid classpath, it returns null. This causes IOUtils.toString() to throw a NullPointerException when attempting to read from a null InputStream.

Comparison with Working Code: The vulnerability template loading at line 914 correctly uses a string literal:

// Line 914 (CORRECT implementation)

final String VULNTEMPLATERESOURCE = "scorecard/vulntemplate.html";

InputStream vulnTemplateStream = CL.getResourceAsStream(VULNTEMPLATERESOURCE);

Proposed Fix

File: plugin/src/main/java/org/owasp/benchmarkutils/score/BenchmarkScore.java

Location: In method generateVulnerabilityScorecards(), approximately line 965 (line number may vary by version)

Change:

// BEFORE (incorrect):

InputStream vulnTemplateStream =

        CL.getResourceAsStream(scoreCardDir + "/commercialAveTemplate.html");

 

// AFTER (correct):

InputStream vulnTemplateStream =

        CL.getResourceAsStream("scorecard/commercialAveTemplate.html");

Patch File (for v1.3)

Note: Line numbers may vary in different versions. Verify line numbers before applying.

--- a/plugin/src/main/java/org/owasp/benchmarkutils/score/BenchmarkScore.java

+++ b/plugin/src/main/java/org/owasp/benchmarkutils/score/BenchmarkScore.java

@@ -963,7 +963,7 @@ private static void generateVulnerabilityScorecards(

                

                 // Resources in a jar file have to be loaded as streams. Not directly as Files.

                 InputStream vulnTemplateStream =

-                        CL.getResourceAsStream(scoreCardDir + "/commercialAveTemplate.html");

+                        CL.getResourceAsStream("scorecard/commercialAveTemplate.html");

                 String html = IOUtils.toString(vulnTemplateStream, StandardCharsets.UTF_8);

                 html = html.replace("${testsuite}", BenchmarkScore.TESTSUITENAME.fullName());

                 html = html.replace("${version}", TESTSUITEVERSION);

Alternative: Direct sed Command

If the patch fails due to line number mismatches:

sed -i 's|CL\.getResourceAsStream(scoreCardDir + "/commercialAveTemplate\.html")|CL.getResourceAsStream("scorecard/commercialAveTemplate.html")|g' \

  plugin/src/main/java/org/owasp/benchmarkutils/score/BenchmarkScore.java

Impact

Severity: High

Affected Users: Anyone running BenchmarkUtils scorecards with:

• 2 or more commercial security tools in their results

• BenchmarkJava, BenchmarkPython, or BenchmarkNodeJS test suites

Workaround:

1. Apply the patch manually to BenchmarkUtils source before building

2. OR exclude all but one commercial tool from results directory

3. OR set showAveOnlyMode: false in scoring config (if commercial averages aren't needed)

Tested Solution

Patch has been validated with:

• BenchmarkJava v1.2 (2740 test cases)

• 14 security tool result files including 4 commercial tools

• Successfully generates all scorecards including Commercial Tools Average

Additional Notes

• Bug exists in both v1.3 release and current main branch

• Similar pattern used correctly for other templates (vulntemplate.html, template.html)

• File location: plugin/src/main/resources/scorecard/commercialAveTemplate.html is correct

• Bug only manifests when commercialAveragesTable.hasEntries() returns true (2+ commercial tools)

References

• Resource loading pattern at line 914: CL.getResourceAsStream(VULNTEMPLATERESOURCE)

• Resource loading pattern at line 67 (ToolReport.java): getResourceAsStream(BenchmarkScore.SCORECARDDIRNAME + "/template.html")