A first calibration

This is a line of Java from a test that was written to fool a machine.

int num = 106;
bar = (7 * 18) + num > 200 ? "This_should_always_happen" : param;

Seven times eighteen is 126, and with 106 more you get 232, which is always more than 200. So bar never contains what the user typed. It always holds the harmless string, and the database query built from it a few lines later cannot be attacked.

Last weekend I gave that file to two different models, inside the same tool and with the same instructions. One read it and reported nothing. The other reported a critical SQL injection. The tool they were running in is quodeq, which I built, so it was my problem to understand why.

I started quodeq because I don't type most of the code I ship anymore. An agent writes it, I read it, we go back and forth, I approve it, and in the end my name is on the commit. The amount of code I have to read keeps growing, and reading something is not the same as knowing it. I have also noticed that agents copy the style of the code around them. In a careful codebase they are careful, and in a sloppy one they carry on in the same way without hesitating. The code you already have shapes the code you get next, so I wanted a tool that keeps an eye on it, and one that I could inspect myself.

I am not the only one worried about this. In May, IOActive tested 27 models and coding tools on 730 programming tasks, and almost a third of the code they generated was fully exploitable.

A tool like quodeq is only useful if it is right, though, and I had never checked that properly. This article is about what I found when I did.

Marking my own homework

Someone who builds a tool and then tests it himself is not the most reliable source, especially when the result is good. That is why I made the whole test public, and why this kind of work now has its own place on the site.

From today there is a new section called Benchmarks. The blog stays the place for my opinions. Benchmarks is for measurements: how each test was done, all the tables, the raw data, and the script I used to calculate the numbers. Anybody can repeat the calculation and tell me if I made a mistake.

The first one is quodeq 1.12 on the OWASP Benchmark, a Java application that was built to be vulnerable on purpose. It has 2,740 small programs. A bit more than half contain a real security hole. The rest are almost the same code but safe, like the line at the top of this article. A tool scores well when it finds the real ones and stays quiet about the safe ones, which means that reporting everything earns the same score as reporting nothing.

I ran the test with five models. Three are cloud models. The other two are open models that I ran on my own computer, where one took an afternoon and the other most of a day.

Finding is the easy part

The first surprise was that almost all the models find most of the problems. The one that found the most was a local model running on my own machine. If finding problems were the whole job, I could use the cheapest model and stop here.

The difference is in the second column, which shows what each model does when the code is fine. Sonnet and Sol leave safe code alone about six times out of seven. The local models complain about two thirds of it. On SQL injection, for example, Gemma found 99% of the real cases and also flagged every one of the safe versions. It sees a query built from a variable and reports it, without checking where the variable comes from. In practice this decides whether people read the report or start ignoring it, and it is most of what you get from a more expensive model.

The squares on the chart are scanners that a lot of teams already use. I took their numbers from results that other people published for this same test. They end up on the right for the same reason as the local models: a rule cannot work out that 232 is always bigger than 200 either, so it reports the line to be safe. I did not run those tools myself, and they were scored a bit more strictly than quodeq, so the comparison is only approximate. The report explains where each number comes from.

None of this makes me want to remove local models from quodeq. For a team whose code is not allowed to leave the company they are the only option, and a model that misses almost nothing is very useful as a first step. What is missing is a second step that checks each finding before it reaches you. That is the next thing I want to try.

The big models have a weak point as well, and this one is my fault. In about a third of the programs the dangerous value passes through a helper in another file, and you can only tell whether the program is safe by reading that helper. With everything in one file, Sonnet was almost perfect. When the answer was in a second file it missed three out of ten and gave three times as many false alarms. Quodeq gives the model one file at a time and never asks it to open another, so Sonnet did not really have a chance with those.

The grade that would not move

I was happy with the detection. Then I looked at what quodeq does with the findings, and that part was not good.

Quodeq gives every codebase a grade from 0 to 10. This application was written by security people to be as broken as possible, and quodeq graded it between 3.5 and 4.8 depending on the model. The strangest part is that the run that found four thousand problems gave it the highest grade.

I know why, because it comes from a decision I made some months ago. I wanted the grade to depend on how many different rules a codebase breaks and not on how many times it breaks them. Otherwise a model that writes a lot could move the number too easily. That worked, but the side effect is that 260 SQL injections count the same as one, and the grade no longer says how serious things are.

There were other problems. Two of the five models marked most SQL injections as "minor" and quodeq accepted it. Two of the runs skipped some files without any warning, while the progress bar said everything was complete. And this benchmark has been public since 2016, so the models have almost certainly seen it before. With real code the results will be worse than these.

What comes next

So this is where quodeq stands, by its own test. With a strong model it finds most of the hidden problems and rarely raises a false alarm. Its grade is not reliable yet, its severity depends on the model you choose, and it only looks at one file at a time.

Those three things are what I am working on now. When they are ready I will run the same 2,740 programs again and publish the new numbers next to these, even if they are not better. The full report has the details that did not fit here, so if you care about the data, that is the place to look.

The labs that build these models carry part of the responsibility for the code their models write, but that does not take away ours. The code in your repository belongs to you or to your employer, and when something fails nobody will ask which agent wrote it. For me it starts with knowing which of my reviewers actually did the sum in that line of Java, and which one only looked as if it had.