# OWASP Benchmark: a scoring target for Java vulnerability scanners > OWASP Benchmark is a runnable Java web application with deliberately planted, genuinely exploitable vulnerabilities and a matching expected-results file, so a SAST, DAST or IAST tool can be scored on speed and accuracy instead of on the vendor's own demo. It is GPL-2.0 licensed, and the released versions are historical, with the live head of the repository being what you actually use. **OWASP-Benchmark/BenchmarkJava** — OWASP Benchmark is a test suite designed to verify the speed and accuracy of software vulnerability detection tools. A fully runnable web app written in Java, it supports analysis by Static (SAST), Dynamic (DAST), and Runtime (IAST) tools that support Java. The idea is that since it is fully runnable and all the vulnerabilities are actually exploitable, it’s a fair test for any kind of vulnerability detection tool. For more details on this project, please see the OWASP Benchmark Project home page. - Repository: https://github.com/OWASP-Benchmark/BenchmarkJava - Website: https://owasp.org/www-project-benchmark/ - Stars: 836 · Forks: 1,510 - Language: Java - License: GPL-2.0 - Published: 2026-10-20 · Updated: 2026-10-20 · Language: en - Canonical page: https://hysenlabs.com/projects/owasp-benchmark-benchmarkjava ## A benchmark whose flaws are meant to be found The OWASP Benchmark Project is a Java test suite designed to verify the speed and accuracy of vulnerability detection tools. It is a fully runnable open source web application that can be analyzed by any type of Application Security Testing tool, including SAST, DAST such as ZAP, and IAST tools. The design decision that makes it useful is stated in the README as the intent: all the vulnerabilities deliberately included in and scored by the Benchmark are actually exploitable, so it is a fair test for any kind of application vulnerability detection tool. That is the hard part of benchmarking a scanner. A test suite built from synthetic patterns rewards a scanner for pattern matching, and a real application rewards a scanner for finding the paths that actually work. Benchmark gives you the second kind of test corpus while keeping the first kind of control, because every planted flaw has a known correct answer. The scoring works through the expected results file. The repository contains expectedresults-1.2.csv at the root, and the scorecard directory and the createScorecards scripts are the machinery around it. The project also includes scorecard generators for numerous open source and commercial AST tools, and the set of supported tools is growing. The README says that scoring capability is implemented in a separate project, BenchmarkUtils, hosted at github.com/OWASP-Benchmark/BenchmarkUtils. So the deliverable is not the application alone. It is the application plus the ground truth, and the tooling that turns a scanner's findings into a comparable number across vendors. That is why OWASP runs the benchmark as a project rather than publishing a single paper, because the ground truth and the tooling have to keep pace with the scanners being measured. ## The released versions are historical, the head of the repo is the release There is a versioning quirk here that will confuse you if the README is skimmed, and it is worth understanding before you check anything out. The README states that the current latest release is v1.2, and that all the releases available on the GitHub releases page are historical. The latest release is always available live by simply cloning or pulling the head of the repository, in other words git pull. The listed tags confirm the age: 1.2beta from 2016-10-01, 1.1final from 2015-08-29, and 1.1 from 2015-06-06. So the pattern is that a version number is cut when the ground truth and the scorecards need to be frozen together, and after that the code keeps moving on the head of the branch. The expectedresults-1.2.csv file in the repository is named for the 1.2 era, and the code on the head of master may have moved past it. That gap is the practical risk in this project, and it is a real one for anyone building a CI comparison. If you check out a tagged release you get a coherent set of code, expected results and scorecards, but a decade old application code. If you work on the head of master you get the current code, and the expected results file may or may not have caught up. The repository is not archived and the last push was on 2026-10-19, so the project is being worked on. The test suite is being extended and the ground truth maintained, and that work lands on the branch rather than in a new tag. The practical rule is to pin what you measure. If you are running the benchmark to compare two scanners, record the commit you used, record the expected results file you used, and record the scorecard generator version, because otherwise a later run is not comparable to an earlier one and the number means nothing. ## Running the benchmark application itself The repository ships scripts for running the application, and the distinction between them is a security decision rather than a convenience one. The first is runBenchmark.sh, described as running the Benchmark Web Application, accessible via local machine only. The second is runRemoteAccessibleBenchmark.sh, which is like the above but allows port 8443 to be accessible outside the machine Benchmark is running on. The README lists them in a form that shows the shape of the deployment, with the Windows equivalents runBenchmark.bat and runRemoteAccessibleBenchmark.bat also present at the repository root, along with runCrawler.sh and runCrawler.bat. Local only is the default and should stay that way unless a specific analysis needs the remote variant. The remote variant exists because a DAST tool, or a scanner that needs to reach the application from another machine, has to be able to connect to it. That is a legitimate need. But the application is intentionally full of exploitable vulnerabilities, so exposing port 8443 puts a deliberately vulnerable web application on a network, and the only thing standing between that and someone else's scanner is your firewall. The repository also has a .keystore file at the root, which is what the HTTPS listener uses, and a .devcontainer directory for running the whole thing in a container. There is a VMs directory, which suggests virtual machine based setups are part of the intended workflow, and that is the right place to run the remote variant. The Maven build is a pom.xml at the root with a .mvn directory, and the source is under src. The other top level directories are where the surrounding machinery lives: data, results, scorecard, scripts, tools and docs. ## Where the benchmark does not tell you the truth The limitation that matters most is that a benchmark measures a scanner against a corpus, and the corpus is not your application. A score here tells you how a tool performs on this set of planted defects, in this framework, with this style of code. It does not tell you how the tool performs on a legacy application with a decade of different coding styles, or on a codebase with a different build system, or on the specific framework you use. Second, the planted vulnerabilities are static. They are in the source, they are known, and the expected results are a static list. A scanner with a signature that matches the planted pattern scores well, and a scanner that finds the same class of defect through a different mechanism, or in a variant of the pattern the benchmark does not contain, does not. That skews results toward tools whose detection logic resembles the corpus, which is a fair criticism of any benchmark built on synthetic defects, and this one is explicit about being synthetic. Third, and this one is structural, the ground truth requires human verification. The expected results file cannot be produced automatically from the code, because deciding whether a given line is a real vulnerability or a safe use of the same API takes judgement. That means the project needs maintainers who will review and update the ground truth as the code evolves, and the cost of that review is the real ongoing cost of the project. Fourth, the tool list is not equally deep. The scorecard generators exist for numerous open source and commercial tools, and the set is growing, but a scorecard only exists for tools someone has written a generator for. When you compare a tool with a scorecard against one without, you are not comparing scanners, you are comparing what has been implemented. The right way to use it, then, is as a first filter rather than a final verdict. It removes tools that are obviously weak, and it gives you a vendor-neutral number. It does not tell you which tool will find the one strange bug in your codebase that nobody has seen before. ## How this differs from the other ways to pick a scanner The alternative approaches to evaluating a vulnerability scanner each give you a different kind of information, and the benchmark is the one that gives you a number. Running a scanner against your own codebase is the most realistic option and the one that produces the least comparable output, because every organisation has a different set of findings and no baseline to compare against. It tells you what the tool finds in your code, and you still have to triage it. The advantage is that the result is directly actionable for you. The disadvantage is that a bad week of engineering, a large legacy module or an unusual framework will distort the picture, and you have no way to tell whether the tool missed something or whether there was nothing to find. A commercial vendor trial or a vendor-run proof of concept gives you a curated demonstration on a codebase the vendor prepared, usually with the vendor's own people helping you interpret the results. It is closer to the truth than a synthetic benchmark and much less independent. A curated public corpus of real applications, with a hand written ground truth, is the closest comparison to what the Benchmark does, and the difference is that the defects are real rather than planted. The cost is that building the ground truth for a real application is enormously more expensive than for a synthetic one, which is exactly why so few such corpora exist. The Benchmark sits between the synthetic benchmark and the real application, and its specific bet is that the planted vulnerabilities are genuinely exploitable, which makes the results less about pattern matching and more about whether a tool finds real defects. That bet is the project's central claim and the reason the OWASP project hosts it as a long running programme rather than a one-off release. A note on the scoring side. The scorecard generators live in BenchmarkUtils rather than in this repository, and the README says the set of supported tools is growing all the time. So the practical workflow is: run the benchmark application, run your tools against it, feed the results into the current generator from BenchmarkUtils, and compare the scorecards. ## Licence, maintenance and the shape of the repository The licence is GPL-2.0, with the LICENSE file at the repository root. For a test suite that is a reasonable choice, since the ground truth and the scorecards are the valuable assets and a copyleft licence keeps derived versions of the corpus and the tooling open. It also means that if you build an internal harness on top of the benchmark and distribute it, you need to think about the obligations, which is a reason many organisations keep their harness internal and only publish the numbers. On maintenance, the repository is not archived and the last push was on 2026-10-19. The recent commits are on the head of master, and the version tags stopped at 1.2beta in 2016, which is the arrangement the README describes as the normal way to consume the project. The top level layout reflects the three separate jobs the repository does. The application itself is in src, built by pom.xml, and run by the runBenchmark scripts. The ground truth and the supporting data are in data, with the expected results file at the root and a results directory for what a run produces. The scoring machinery is split between the scorecard directory, the tools and scripts directories, and the createScorecards scripts, which are the shell and batch entry points that drive the generators. There is also a style configuration, DevStyleHtml.prefs and DevStyleXml.prefs, which are preferences for a Java IDE formatter, and a .pre-commit-config.yaml equivalent in the .github directory along with the workflows. The OWASP project page at owasp.org/www-project-benchmark is where the project documentation lives, and the README says to refer to that site for all the project details. One practical point about the crawler. The runCrawler scripts alongside the benchmark runners suggest a crawling mode for the application, which is the mechanism a DAST scanner needs to explore the application rather than being pointed at a single URL. If your tool of interest works by crawling, that is the mode to use, and it is worth confirming that your tool and the crawler are driving the same navigation. ## Conclusion Adopt OWASP Benchmark if you are choosing between vulnerability scanners for a Java stack and want a fair, vendor-neutral comparison based on genuinely exploitable findings. Do not adopt it as a training target for developers, since it is built to be vulnerable, and do not run the remote accessible variant on an untrusted network. Verify first that your tool can actually be pointed at the benchmark's own endpoints, that the expectedresults CSV matches the version of the code you checked out, and that the scorecard generator you use is the current one in BenchmarkUtils. ## FAQ ### What is the OWASP Benchmark and what is it for? It is a fully runnable open source Java web application with deliberately included and genuinely exploitable vulnerabilities, used to verify the speed and accuracy of vulnerability detection tools. The planted flaws are actually exploitable so that the test is fair for any kind of application vulnerability detection tool. ### Which kinds of security tools can be tested with OWASP Benchmark? Any type of Application Security Testing tool, including SAST, DAST tools such as ZAP, and IAST tools. The README also notes the project includes scorecard generators for numerous open source and commercial AST tools. ### How do I run the OWASP Benchmark web application? Use runBenchmark.sh to run the application accessible via local machine only, or runRemoteAccessibleBenchmark.sh, which is like the above but allows port 8443 to be accessible outside the machine the benchmark runs on. Windows equivalents runBenchmark.bat and runRemoteAccessibleBenchmark.bat are also included. ### Which version of OWASP Benchmark should I use? The README states that the releases on the GitHub releases page are historical and that the latest release is always available by cloning or pulling the head of the repository. The latest tagged release is v1.2 from 2016, and the last push was on 2026-10-19. ### Where does the OWASP Benchmark scorecard generator live? In a separate project, BenchmarkUtils, at github.com/OWASP-Benchmark/BenchmarkUtils. The README says the set of supported tools is growing all the time, so the generator version matters when comparing results across runs. ### What license is OWASP Benchmark released under? GPL-2.0, with the LICENSE file at the repository root. ## Sources - [License: GPL-2.0](https://github.com/OWASP-Benchmark/BenchmarkJava/blob/master/LICENSE) - [OWASP-Benchmark/BenchmarkJava on GitHub](https://github.com/OWASP-Benchmark/BenchmarkJava) - [Project website](https://owasp.org/www-project-benchmark/) - [README](https://github.com/OWASP-Benchmark/BenchmarkJava/blob/master/README.md) - [Releases](https://github.com/OWASP-Benchmark/BenchmarkJava/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/owasp-benchmark-benchmarkjava