OWASP Benchmark for Java

Abstract

Quodeq is an open-source tool that uses a language model to review code against written standards. This report measures how well its security review separates real vulnerabilities from safe code that looks the same. Quodeq 1.12.0 was run once with each of five models on the OWASP Benchmark for Java, which has 2,740 labelled test cases. With the two large cloud models it found 83% and 91% of the real vulnerabilities while flagging 15% and 14% of the safe cases, a score of 0.68 and 0.77. Published results for rule-based scanners on the same cases range from 0.32 to 0.60. Two open models run locally found as much as the best cloud model but flagged 65% and 72% of the safe cases, a score of 0.20. Accuracy fell for every model when the evidence was in a second file, which quodeq does not ask the model to open. The run also exposed three defects in the tool itself: an overall grade that does not respond to the number of violations, a severity rating that depends on the model, and files skipped without notice. The benchmark is public and the models have probably seen it, so these results are an upper bound.

Key findings

1. Detection depends on the model. The same pipeline scored between 0.20 and 0.77.
2. Large cloud models separate safe from vulnerable code. Claude Sonnet 5.5 scored 0.77 and GPT-6 Sol 0.68, above the published scores of rule-based scanners on this benchmark.
3. Local models find as much and discriminate far less. Gemma 4 and Qwen 3.8 found 85% and 92% and flagged 65% and 72% of safe code.
4. Cross-file evidence is the main remaining error. The best model went from finding 96% with 9% false alarms to 69% with 29%.
5. The tool's own grade and severity are not reliable. The overall grade ranged from 3.5 to 4.8 out of 10 for an application built to be vulnerable.

1. Setup

1.1 Benchmark

The OWASP Benchmark for Java, version 1.2, is an application built to be vulnerable. It has 2,740 test cases in 11 categories, each mapped to one CWE. 1,415 contain a real vulnerability. The other 1,325 are near-identical and safe. The ground truth ships with the project.

Each category is one kind of mistake.

Category	What goes wrong
SQL injection	Input from the user ends up inside a database query, so an attacker can change what the query does.
Cross-site scripting (XSS)	Input is written into a web page without escaping, so an attacker can run script in someone else's browser.
Command injection	Input ends up inside an operating system command.
Path traversal	Input chooses a file path, so an attacker can reach files outside the intended folder.
LDAP injection	Input changes a query against a user directory.
XPath injection	Input changes a query over XML data.
Weak cipher	Data is encrypted with an algorithm that is considered broken, such as DES.
Weak hash	A broken hash function such as MD5 or SHA-1 is used.
Weak randomness	A predictable random number generator produces something that must be unguessable, such as a session token.
Insecure cookie	A cookie is set without the Secure flag, so it can travel over an unencrypted connection.
Trust boundary	Unchecked input is stored in the session, where the rest of the application treats it as trusted.

The first six and the last one depend on following data: the line is only a problem if something an attacker controls can reach it. The other four can be judged from the line itself. This distinction accounts for most of the differences reported below.

1.2 Runs

Quodeq 1.12.0, security dimension only, one run per model, same standards and prompts. Three cloud models (Claude Sonnet 5.5, GPT-6 Sol, GPT-6 Luna) and two open models served locally by Ollama (Gemma 4 26B, Qwen 3.8 27B). Each run is reported as it finished. No run was repeated or corrected.

2. Method

2.1 Scoring

A test case counts as flagged when quodeq reported a violation in its file under a security requirement that covers the case's CWE. A violation of an unrelated requirement in the same file does not count. From that:

• Found is the share of the 1,415 real vulnerabilities that were flagged.
• False alarms is the share of the 1,325 safe cases that were flagged anyway.
• Score is found minus false alarms. This is the benchmark's own measure. A perfect tool scores 1. A tool that flags every case, or none, scores 0.

Quodeq's requirements are coarser than CWEs. One requirement covers LDAP, XPath and code injection, and the cryptography requirements overlap. The match is therefore by requirement family and not by exact CWE, and the results are not directly comparable with scores published for other tools.

2.2 Cost

Token usage came from the providers' own session logs. Cost is those tokens at the list prices on Anthropic's and OpenAI's pricing pages in October 2026. The cloud models were run through the vendors' command-line tools, so the figure is an API-equivalent cost and not an amount billed.

3. Results

3.1 Overall

Model	Found	Missed	False alarms	Correctly left alone	Found %	False alarm %	Flags that were right	Score
Claude Sonnet 5.5	1,289	126	188	1,137	91%	14%	87%	0.77 ± 0.02
GPT-6 Sol	1,172	243	202	1,123	83%	15%	85%	0.68 ± 0.03
GPT-6 Luna	703	712	216	1,109	50%	16%	77%	0.33 ± 0.03
Gemma 4 26B (local)	1,203	212	867	458	85%	65%	58%	0.20 ± 0.03
Qwen 3.8 27B (local)	1,306	109	953	372	92%	72%	58%	0.20 ± 0.03

The ± is the 95% margin that follows from the number of test cases alone. Each model was run once, so variation between runs is not captured, and small differences between models should not be over-interpreted.

The five models fall into three groups. Sonnet and Sol find most of the vulnerabilities and flag few safe cases. Luna flags equally few safe cases but misses half of the vulnerabilities. The local models miss few vulnerabilities and flag most of the safe cases, which places their score close to Luna's although their detection rate is close to Sonnet's.

3.2 By security principle

Quodeq files every finding under a security principle taken from the ISO 25010 quality standard. Three of them cover this benchmark.

• Confidentiality. Data is only readable by those allowed to read it. Weak ciphers, weak randomness and insecure cookies are filed here.
• Integrity. Data and behaviour cannot be changed by someone who should not change them. SQL injection, XSS, LDAP and XPath injection, weak hashes and trust-boundary mistakes are filed here.
• Authenticity. The system only acts for who and what it should. Quodeq files command injection and path traversal here.

The benchmark's categories were grouped by the principle under which correct findings were filed in practice.

Principle	Real / safe cases	Sonnet 5.5	GPT-6 Sol	GPT-6 Luna	Gemma 4	Qwen 3.8
Confidentiality	384/422	100%/12%0.88	91%/21%0.71	51%/8%0.44	90%/44%0.46	99%/69%0.31
Integrity	772/643	87%/14%0.73	81%/12%0.68	53%/22%0.31	78%/66%0.11	86%/68%0.18
Authenticity	259/260	89%/17%0.72	76%/13%0.63	37%/17%0.20	99%/98%0.01	99%/87%0.13

Confidentiality is the principle on which every model scores best. Its checks can be made from the line itself: a cipher name, a random number generator, a cookie flag. Integrity and authenticity depend on whether user-controlled data reaches the sensitive line, and on these the local models barely discriminate. On authenticity, Gemma flags 99% of the vulnerable cases and 98% of the safe ones.

3.3 By category

Category (CWE)	Real / safe	Sonnet 5.5	GPT-6 Sol	GPT-6 Luna	Gemma 4	Qwen 3.8
SQL injection (89)	272/232	94%/16%	81%/17%	68%/34%	99%/100%	100%/94%
XSS (79)	246/209	92%/14%	94%/16%	56%/21%	87%/68%	100%/66%
Weak randomness (330)	218/275	100%/9%	95%/28%	55%/8%	86%/61%	100%/92%
Path traversal (22)	133/135	90%/16%	76%/10%	41%/21%	99%/100%	100%/81%
Weak cipher (327)	130/116	100%/22%	95%/8%	55%/8%	98%/16%	98%/26%
Weak hash (328)	129/107	74%/3%	78%/2%	37%/2%	69%/8%	69%/19%
Command injection (78)	126/125	88%/19%	76%/18%	34%/14%	99%/96%	98%/93%
Trust boundary (501)	83/43	69%/19%	43%/9%	2%/0%	4%/7%	24%/21%
Insecure cookie (614)	36/31	100%/0%	53%/0%	14%/6%	86%/0%	100%/26%
LDAP injection (90)	27/32	89%/31%	93%/3%	78%/28%	52%/66%	100%/100%
XPath injection (643)	15/20	100%/15%	87%/5%	87%/30%	67%/95%	100%/100%

Sonnet has the best score (found minus false alarms) in seven categories, Sol in three, and they tie on XSS.

Three observations follow. Sol has the lowest false-alarm rate in several categories, including LDAP injection (3% against 31% for Sonnet) and weak cipher. Trust boundary is the weakest category for every model, and quodeq's standards contain no dedicated requirement for it, which may contribute. Qwen's 92% false-alarm rate on weak randomness is a factual error and not a data-flow error: it reports SecureRandom.getInstance("SHA1PRNG") as weak because the name contains SHA-1.

3.4 Single-file and cross-file cases

About 30% of the test cases (814) pass the suspicious value through a helper class in a different file. Whether the case is vulnerable depends on what that helper does. Quodeq gives the model one file at a time and does not ask it to follow the call.

Model	Single-file cases (708 real, 496 safe)	Cross-file cases (194 real, 300 safe)
Claude Sonnet 5.5	96%/9%	69%/29%
GPT-6 Sol	80%/9%	81%/22%
GPT-6 Luna	53%/19%	42%/30%
Gemma 4	86%/82%	81%/88%
Qwen 3.8	94%/79%	88%/83%

Sonnet has the best score on single-file cases and loses that position on cross-file cases. Sol's detection rate is almost unchanged, while its false-alarm rate more than doubles. For the two large models, most of the remaining error is concentrated in the cross-file cases. This error follows from how quodeq presents code to the model, one file at a time, and is unlikely to be removed by a stronger model alone.

3.5 Comparison with other tools

The benchmark has been applied to many tools. Two sources report results on the same 2,740 cases.

Quodeq 1.12 with each modelOther tools, published results

Tool	Results from	Found	False alarms	Score
Quodeq + Claude Sonnet 5.5	this page, 2026	91%	14%	0.77
Quodeq + GPT-6 Sol	this page, 2026	83%	15%	0.68
CodeQL 2.27.0 (java-security-extended)	Cognium, Sept 2026	100%	40%	0.60
Semgrep OSS 1.177.0 (p/java + p/security-audit)	Cognium, Sept 2026	87%	39%	0.48
FindBugs with FindSecBugs	benchmark scorecard, 2016	97%	53%	0.44
Quodeq + GPT-6 Luna	this page, 2026	50%	16%	0.33
SonarQube Java plugin 3.14	benchmark scorecard, 2016	43%	11%	0.32
Quodeq + Gemma 4 / Qwen 3.8 (local)	this page, 2026	85%/92%	65%/72%	0.20
OWASP ZAP (dynamic scanner)	benchmark scorecard, 2016	17%	0%	0.17
FindBugs 3.0.1	benchmark scorecard, 2016	11%	10%	0.01
PMD 5.2.3	benchmark scorecard, 2016	0%	0%	0.00

Three caveats apply to this comparison.

• The scoring differs. The other rows count a case only on an exact CWE match. Quodeq's rows match by requirement family, which is slightly more lenient. The comparison is indicative, not exact.
• The 2016 rows are old. They were computed from the per-case scorecards that ship in the benchmark repository. Those tools have changed since.
• The 2026 rows come from a vendor. Cognium's comparison includes their own product, which they score at 0.74. CodeQL and Semgrep were not rerun for this report.

Subject to these caveats, the pattern is consistent. Rule-based scanners that detect nearly all vulnerabilities also flag 40% to 53% of the safe cases. The two large models detect slightly fewer and flag far fewer. The local models behave like the least selective rule-based scanners and score below them.

3.6 Speed and cost

Model	Agents running at once	Time	Files per minute	Cost
Claude Sonnet 5.5	5	32 min	88	$52
GPT-6 Sol	10	21 min	133	$20
GPT-6 Luna	10	17.5 min	160	$1.24
Gemma 4 26B (local)	2	2 h 29 min	19	local
Qwen 3.8 27B (local)	2	8 h 51 min	5	local

No best value is marked for speed. The times are not directly comparable, because the runs used different numbers of agents at once: 10 for the GPT models, 5 for Sonnet, 2 for the local ones. Per agent, Sonnet was the fastest cloud model at 17.6 files a minute. For the local models a second agent adds little, because the server does not run them in parallel and memory is the limit. On the test machine, Qwen processes about 4 files a minute with one agent and 5 with two.

Sol costs less than half of what Sonnet does for a score 0.09 lower. Gemma reaches the same score as Qwen in a quarter of the time.

4. Worked cases

A safe case that a local model flagged. BenchmarkTest01878 builds a SQL query from a variable called bar. This is where bar comes from:

String bar;
int num = 106;
bar = (7 * 18) + num > 200 ? "This_should_always_happen" : param;

7 times 18 is 126, plus 106 is 232, which is always greater than 200. So bar is always the constant and the user's input never reaches the query. The benchmark labels the case safe. Sonnet reported nothing. Gemma reported a critical SQL injection, with this reason: "The SQL query is constructed using string concatenation with the 'bar' variable, which is derived from unvalidated cookie input." The model reported the concatenation without evaluating the condition.

A real case the best model missed. BenchmarkTest01923 writes a value into the HTML response. The value comes from here:

org.owasp.benchmark.helpers.ThingInterface thing =
        org.owasp.benchmark.helpers.ThingFactory.createThing();
String bar = thing.doSomething(param);

Whether bar still carries the user's input depends on which implementation ThingFactory returns, and that is decided in another file. In this case the input passes through and the page is open to cross-site scripting. Sonnet reported nothing for this file. Quodeq had supplied this file alone and did not ask the model to open the factory.

The same trap in a helper. Many cases read input through a helper class with two kinds of method:

public String getTheParameter(String p) {
    return request.getParameter(p);
}

// This method is a 'safe' source.
public String getTheValue(String p) {
    return "bar";
}

A test case that calls getTheParameter is vulnerable and one that calls getTheValue is not, and the two test files look almost the same. Only the helper distinguishes them. These cases make up the 30% of the benchmark on which every model scores lower.

5. Defects observed in the tool

The benchmark also exposed three defects in how quodeq processes its findings.

The grade. Table 8 shows the grades each run produced for an application that is vulnerable by design.

Model	Violations	Confidentiality	Integrity	Authenticity	Overall
Claude Sonnet 5.5	1,752	3.7	2.7	4.0	3.5
GPT-6 Sol	2,436	3.4	4.4	5.3	4.4
GPT-6 Luna	1,334	3.8	4.9	5.4	4.7
Gemma 4 26B	2,655	2.2	2.3	2.8	4.4
Qwen 3.8 27B	4,026	1.9	2.0	3.7	4.8

Two mechanisms produce this result. First, a principle is scored on how many different rules are broken, not how often or in how many files. Qwen's 1,191 integrity violations come down to 10 rules before any arithmetic happens. Second, the overall grade is an unweighted mean of every principle that has a score. In the Gemma and Qwen runs, accountability and non-repudiation each had between 7 and 14 findings and scored between 6.0 and 8.8. Those two principles raised Qwen's overall grade from about 2.5 to 4.8. A principle with 7 findings therefore carries the same weight as one with 2,329.

Severity. Severity is assigned by the model, and the models disagree.

Model	Critical	Major	Minor
Claude Sonnet 5.5	786	847	118
GPT-6 Sol	139	191	2,050
GPT-6 Luna	12	18	1,304
Gemma 4 26B	1,686	954	15
Qwen 3.8 27B	1,817	2,043	166

Both GPT models rated most findings minor, including most SQL injections. The other three rated the same class of finding critical. In the Sol run, 56 findings carried a severity that was not one of the three allowed values, and quodeq accepted them. Nothing in the tool defines what a critical finding is independently of the model, and a user who writes their own standard cannot say that a rule is never minor.

Coverage. Two runs skipped files without reporting it. Luna left 265 of the 2,740 test cases unanalysed and Sol left 35, while the progress display read complete. Those cases are counted as misses in the tables above, since a user running the tool once would obtain the same result.

6. Limitations

• The benchmark is public and old. It dates from 2016, the files carry its name, and the models have very likely seen it. The scores should be read as an upper bound for real code.
• The cases are synthetic. Each is one servlet, sometimes with one helper. Real vulnerabilities often span more files.
• One run per model. No variance estimate across runs.
• One language, one dimension. Java, security only.
• The matching is by requirement family. Stricter than counting any finding in the file, looser than an exact CWE match.
• Local results are specific to these two models and the hardware used. Larger open models may behave differently.
• Local-model verification is untested here. A second pass that checks each finding against more context is expected to remove many of their false alarms. Related work supports this: Xiong and Zhang (2026) report that LLM agents cut a false-positive rate of over 92% on this same benchmark to as low as 6.3% in their best configuration, and warn that aggressive filtering can also remove real vulnerabilities. This was not tested with quodeq.

7. Reproducibility

• Benchmark: OWASP-Benchmark/BenchmarkJava at commit 8b67a88d, with its expectedresults-1.2.csv.
• Tool: quodeq 1.12.0, security dimension, default standards.
• Scorer: score_owasp.py. It reads a run's security report and the benchmark's expected results and prints the tables above. The mapping from benchmark category to quodeq requirement is a table at the top of the script.
• Data: one security report per run and a combined comparison.json with every count on this page.
• Not yet done: a repeat of any run, so there is no run-to-run variance.

References

• OWASP Benchmark Project. BenchmarkJava, version 1.2, and the scorecards it ships.
• Cognium. sast-benchmarks, OWASP Java comparison of 11 September 2026.
• Y. Xiong and T. Zhang. Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering, 2026.
• IOActive. The Security Gap in AI-Generated Code, May 2026.

Changes to this page

• October 2026: first version.