OWASP Benchmark Project

Overview

The OWASP Benchmark for Security Automation (OWASP Benchmark) is a free and open test suite designed to evaluate the speed, coverage, and accuracy of automated software vulnerability detection tools and services (henceforth simply referred to as 'tools'). Without the ability to measure these tools, it is difficult to understand their strengths and weaknesses, and compare them to each other. Each version of the OWASP Benchmark contains thousands of test cases that are fully runnable and exploitable, each of which maps to the appropriate CWE number for that vulnerability.

You can use the OWASP Benchmark with Static Application Security Testing (SAST) tools, Dynamic Application Security Testing (DAST) tools like OWASP ZAP and Interactive Application Security Testing (IAST) tools. The current version of the Benchmark is implemented in Java. Future versions may expand to include other languages.

The OWASP Benchmark and Hdiv

Accuracy score

Hdiv Detection (IAST) scored a 100%, which comes from a 100% true positive rate minus a 0% false positive rate.

Open video

OWASP Benchmark Scorecard

Usage

How to run the analysis of the OWASP Benchmark

If you wish to regenerate our Benchmark results, please proceed as follows.

Requirements

Java, Maven and Git have to be installed in your environment.

JDK >10 LDAP issues

For now Benchmark code is not fully compatible with modern JDKs and LDAP test cases are faillng because of this open issue https://github.com/OWASP-Benchmark/BenchmarkJava/issues/54.

CSP headers and toolbar

If you want to test the use cases manually and see the toolbar you should activate the flag

-Dhdiv.toolbar.change.csp.headers=true

to allow the toolbar to be displayed properly

Steps

  1. Download the OWASP Benchmark Utilities from GitHub and execute a Maven install:
    git clone https://github.com/OWASP-Benchmark/BenchmarkUtils.git
    cd BenchmarkUtils
    mvn install
    
  2. Download the OWASP Benchmark Project from GitHub:

    git clone https://github.com/OWASP-Benchmark/BenchmarkJava.git
    cd BenchmarkJava
    
  3. Edit pom.xml file and add the following lines to the cargo-maven2-plugin configuration properties:

    <properties>
        ..
        <cargo.jvmargs>
            -javaagent:/{path-to-hdiv-folder}/agent/java/hdiv-ee-agent.jar
            -Dhdiv.config.dir=/{path-to-hdiv-folder}/license/
            -Dhdiv.agent.debug=true 
            -Dhdiv.xss.advanced=true 
            -Dhdiv.file.level=FINE 
            -Dhdiv.log.date.format=TIME 
            -Dhdiv.workingMode=FULL_DETECTION
            -Dhdiv.log.append=true
        </cargo.jvmargs>
        <cargo.servlet.port>8443</cargo.servlet.port>
        <cargo.protocol>https</cargo.protocol>
        ...
    <properties>    
    

    Note

    Replace {path-to-hdiv-folder} variable with the path where your Hdiv folder is installed

    Save the file.

  4. Remove previous log files.

    rm {path-to-hdiv-folder}/license/*/hdivAgentLog.hlg
    
  5. Launch the Benchmark application and wait until it starts.

    ./runBenchmark.sh 
    

    License Benchmark application

    Remenber to license the benchmark application in the Hdiv console to enable IAST features

  6. In another terminal, run the Crawler and wait until it completes.

    ./runCrawler.sh
    
  7. An Hdiv log file will be generated:

    {path-to-hdiv-folder}/{server-name}/hdivAgentLog.hlg
    
  8. Move the log file to ./results/ directory:

    mv {path-to-hdiv-folder}/{server-name}/hdivAgentLog.hlg  ./results/
    
  9. Create scorecards. The following command will compute a Benchmark scorecard for all the results files in the /results directory. The generated scorecard is put into the /scorecard directory.

    ./createScorecards.sh
    
  10. Check out the results.

    ./scorecard/Benchmark_v1.2_Scorecard_for_Hdiv.html