GPT-3 as a Security Scanner: What the gpt3_security_vulnerability_scanner Repository Actually Contains

The problem is per-file LLM vulnerability triage, not a scanner you install

The README frames the work as an experiment: using OpenAI's GPT-3 to find security vulnerabilities in a code repository containing 129 vulnerable files. The author states that GPT-3 found 213 vulnerabilities in the repository, and that a commercial tool from a reputable cybersecurity company found 99 issues on the same code, while noting the commercial tool presents context in a more structured format. The author also states that both tools had many false negatives. So the subject here is an evaluation of an LLM as a static analyser, and the repository is the artifact of that evaluation: a set of folders, each named after a vulnerability class, each containing example code and a README.md holding GPT-3's analysis of the files in that folder. The intended reader is someone deciding whether an LLM can substitute for or supplement a rule-based scanner, not someone looking for a binary to point at a production codebase. The README is explicit that the analysed files are snippets, and that they therefore lack the context of a larger codebase.

Why the scan is file-by-file: the 4000-token window forces the split

The mechanism is simple and constrained. The README states the variant used, text-davinci-003, has a context window of 4000 tokens, roughly 3000 English words, so it can process at most a few hundred lines of code per request. GPT-3's architecture, as described, cannot take a whole repository at once. The workaround was to scan every file separately. That choice has a direct consequence the author names: vulnerabilities that arise from multiple files interacting may be missed, unless the imports and exports are clear enough for the model to guess what the imported functions do without seeing their code. The author reports that this often worked, particularly with common libraries such as express.js, Flask, the Python standard library and the C standard library, and suggests GPT-3 has those libraries partially or fully memorized. The author also concedes that many commercial scanners do not inspect imported library code during static analysis either, so the limitation is not unique to this approach. The data flow, then, is: source file in, prompt out, model response written into a per-folder README.md as the record of the analysis.

What the examples show about the quality of the analysis

Three examples in the README illustrate the range. A trivial C program that calls printf(argv[1]) receives an analysis listing unvalidated user input and a format string vulnerability; the README calls this correct. A C# controller that logs user input by concatenation, _logger.LogError("error!! " + userInfo), receives a single finding for log injection, again called correct. The third example, a C program that reads and manipulates an image with a struct holding a header, width, height and a fixed data buffer, is described as containing numerous vulnerabilities including out-of-bounds reads and writes. The pattern across the three is that the model produces short, plain-language findings rather than rule identifiers with line numbers. The README's own note that the commercial tool's output was more structured is the honest framing of the trade: the LLM output is readable and the rule-based output is machine-consumable. For a human triaging a snippet, the former may be enough. For a pipeline that needs to deduplicate, suppress, or track findings across commits, it is not.

The false-positive claim and how to read the 60-of-213 sample

The headline number in the README is that after manually reviewing a random sample of 60 of the 213 vulnerabilities GPT-3 detected, only 4 were false positives. That is roughly a 6.7 percent false-positive rate within the sample, and it is the strongest evidence in the repository for the approach. Two caveats follow from the material itself. First, the sample is 60 of 213, and the author does not describe the sampling procedure beyond calling it random, so the confidence interval around that rate is not stated. Second, the repository is a corpus of deliberately vulnerable files, each folder named after a vulnerability class. Precision measured on code that is known to contain vulnerabilities says nothing about precision on ordinary production code, where the base rate of vulnerable lines is far lower and a scanner's noise becomes the dominant cost. The README does not report a run over a non-vulnerable codebase, and neither does it report the false-negative rate for either tool beyond the statement that both had many. Anyone citing the 213-versus-99 comparison should note that the commercial tool is not named in the supplied material.

Running it: there is no install step in the material

The repository has no homepage, no releases retrieved, and a license field that is unknown in the supplied metadata. The README describes an experiment and links to a Medium article for the full text; it does not give a command to run, a configuration file, an API key environment variable, or a package manifest. The primary language is PHP, which is a mismatch worth flagging: the examples in the README are C, C# and other languages, and the PHP label appears to describe the repository's own contents rather than the language being analysed. In practice, reproducing the method means writing the harness yourself: walk the tree, read each file, send it to the model with a prompt asking for vulnerabilities, and write the response next to the file. The one hard constraint to design around is the 4000-token context of text-davinci-003 stated in the README, which sets the maximum chunk size. Because that model has been retired by OpenAI, a reproduction today would target a different model, and the README's precision figures would not transfer to it.

Where a per-file LLM pass is the wrong tool

The design fails at anything that requires whole-program reasoning. Taint that crosses a function boundary, a request handler in one file feeding a query builder in another, an authorization check that exists in a middleware file the model never sees: the README concedes these are exactly the cases the file-by-file split struggles with, and it relies on the model's prior knowledge of common libraries to paper over the gap. That reliance is a failure mode in itself. If the code imports an internal library the model has never seen, the guess about what the imported function does is unverified, and a wrong guess produces either a missed finding or a confident false one. Cost and determinism are the other two. Each file is a separate API call, so a large repository becomes a large number of calls, and the same file scanned twice can yield different findings because the model is not deterministic by default. A rule-based analyser gives the same output for the same input, which is what makes suppression files and CI gates workable. The README's own comparison makes the point: the commercial tool found fewer issues but presented them in a structured format, and structure is what a pipeline consumes.

The honest alternative: Semgrep, and where the two differ

The closest thing to a drop-in replacement for what this repository experiments with is Semgrep, an open source static analysis tool that matches code against declarative patterns written in YAML. The difference in approach is not a matter of degree. Semgrep evaluates rules against the syntax tree, so a finding is traceable to a rule and a line, and the same rule set produces the same findings on every run. It also supports inter-file and inter-procedural taint analysis through its dataflow engine, which addresses precisely the cross-file case the README says GPT-3 handles by guessing. What Semgrep does not do is explain a finding in prose or generalize to a vulnerability class it has no rule for; the rule has to be written first. That is the trade in one line: Semgrep is deterministic and rule-bound, the GPT-3 approach is generative and unverifiable per finding. The README's own data supports reading them as complementary rather than competing, since it reports that both tools had many false negatives. A reasonable use of the LLM pass is as a second opinion on files a rule-based scanner has already flagged, where a human is going to read the output anyway.

Maintenance, licence and what to verify before relying on it

The supplied metadata gives no license identifier, and the repository contains no LICENSE file in the material provided, so the terms under which the code samples and the GPT-3 analyses may be reused are unclear. Treat the snippets as reference reading rather than as material to copy into your own test suite until that is resolved; this is a factual observation about the repository, not legal advice. On maintenance, the last push is dated 2026-06-09 and there are no releases, which is consistent with an experiment that was written up and left as a record. The dependency that mattered most, text-davinci-003, is no longer the model anyone would call, so the reproduction path has already changed once and will change again. If you want to reuse the corpus, verify three things first: that the folder-level README analyses correspond to the files currently in that folder, that the licence permits the reuse you intend, and which commercial tool produced the 99-issue comparison, since the README does not name it and the comparison is the repository's most-quoted claim.

Editorial conclusion

Read this repository if you want a worked example of per-file LLM vulnerability triage and a labelled corpus of vulnerable snippets to reason against. Do not adopt it as a scanner: there is no CLI, no configuration, no licence file in the supplied material, and the scan depended on text-davinci-003, a model OpenAI has since retired. Before citing the 213-versus-99 comparison, check which commercial tool was used and on what file set, because the README does not name it.

Official sources

1. chris-koch-penn/gpt3_security_vulnerability_scanner on GitHub
2. Issues
3. README