Model or dataset
Tencent/AICGSecEval avatar
Tencent/AICGSecEval

A.S.E: evaluating generated code with model and agent workflows

A.S.E (AICGSecEval) is a repository-level AI-generated code security evaluation benchmark developed by Tencent Wukong Code Security Team.

667 stars113 forksPythonNOASSERTION

At a glance

What is it?
Tencent Wukong's repository-level benchmark generates code with a model or agent and evaluates it using static and dynamic checks. Its quickstart documents both paths, though some shell examples need syntax corrections.
Who is it for?
The design here is more considered than a leaderboard. Generating from real projects and CVE patches, extracting project-level context automatically, and then verifying with a dynamic pass built from test cases and proofs of concept answers the obvious objection that static scanners mostly produce false positives.
Can I use it commercially?
Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
Is it still maintained?
Yes. The repository last received commits 174 days ago.
What is it written in?
Mainly Python, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • Tencent/AICGSecEval667 ★ · +9
Star history
Repository2026-09-102026-11-15
Tencent/AICGSecEval658667

9 Stars in the last 66 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

The documented command loses its continuation and its comment syntax

There is one worked command on the page and it does not survive contact with a shell. The block opens with the general form, `python3 invoke.py [options...] {--llm | --agent} [llm_options... | agent_options...]`, then the help invocation, then an LLM example with line continuations on every line:

code
# Basic Usage
python3 invoke.py [options...] {--llm | --agent} [llm_options... | agent_options...]

# View all available options
python3 invoke.py -h

The full example continues with a model name, a base URL, an API key, a batch id, a dataset path and an output directory, each on its own continued line. Then the continuations stop: `--max_workers` and `--github_token` appear as bare lines, so the shell runs them as commands and fails, and the last line uses `//` as its comment marker, which is not a shell comment at all. Copying the block therefore runs an evaluation with fewer options than intended and then prints two errors.

The README also documents agent evaluation

The full upstream README describes agent evaluation and includes an example using `--agent`, `--agent_name`, a dataset and agent-specific model settings. It explains that unrecognized arguments are passed to the selected agent module. The earlier claim that the documentation stopped after one sentence came from a truncated excerpt.

The upstream example itself still needs care when copied: its line continuations and comment syntax should be checked, and credentials should be supplied securely using the configuration supported by the chosen agent.

A 1.0 batch id pointing at 2.0 data, and two validators

The example passes `--batch_id v1.0` together with `--dataset_path ./data/data_v2.json`, so the batch label and the data file are on different version lines. The repository keeps validation scripts for both, one named for the first version and one for the second, which suggests the two formats coexist rather than one replacing the other. The data directory at the root is where the file is expected to live, and there is no schema description in the quick start; the format is documented instead in the contribution guides, with separate documents for contributing a static dataset and a dynamic one. The retrieval side has two implementations as well, one built on BM25 through a search index library and one built on another vendor's coding agent, so the context you feed a generator is itself produced by a choice between two tools.

The benchmark clones real repositories, anonymously by default

One line of the example is the most operationally important thing on the page, and it is the line the shell cannot run. It passes a GitHub token with a note that if it is not provided, anonymous cloning will be used, which may be subject to clone rate limiting. So the evaluation fetches third-party source code from GitHub during a run, and without a token it does so against the anonymous rate limit, which is exactly the condition under which a long run with a configurable worker count will start failing. The token is also passed on the command line, where it is visible to anything reading the process list or the shell history. The notes confirm the run is long and that the worker count is the knob for total runtime.

Sixteen gigabytes, a hundred gigabytes, and a JVM in the dependency list

The system requirements table recommends at least 16GB of memory, at least 100GB of disk, Python 3.11 or newer and Docker 27 or newer. The container requirement is not incidental: the version 2 upgrade description says the dynamic half of the evaluation is based on test cases and vulnerability proofs of concept, which means executing generated code in a container. Twelve pinned dependencies follow, including a Docker client library, a search index library that pulls in a JVM, a MySQL driver, an OpenAI client, a transformers pin, a git library, a retry library and an encoding detector. A MySQL driver in a benchmark that documents only a pip install and one command means results are expected to land in a database that the quick start never mentions.

The dataset link is commented out in the markup

The badge block at the top of the page is worth reading as source. Four anchors are present with no text between them, pointing at the repository, the project's own site, an evaluation results page and an updates page, and a fifth is wrapped in an HTML comment. So is the HuggingFace dataset link, which appears as a commented-out anchor with a matching commented-out closing tag. The page describes the project as open, reproducible and continuously evolving, and asks for contributions through stars, forks, issues and pull requests, while the link that would publish the dataset to a public model hub is switched off in the file. Everything visual on this page is an image that did not survive the conversion, including the framework design figure.

A licence file at the root and no licence on the page

The repository root carries a licence file, and the table of contents at the top promises a licence section as its last entry. Neither is visible in the text. What is visible is the release history, which mixes two kinds of tag: version tags for the framework itself, and a tag whose name is a research report title in Chinese, published one day after the framework's own 2.0.1 tag. So the repository carries a report as a release artefact alongside its code. The branch was last pushed to on 25 May 2026, four months before the newest framework tag, which means the code on the default branch is ahead of what the tagged release contains.

Editorial conclusion

The design here is more considered than a leaderboard. Generating from real projects and CVE patches, extracting project-level context automatically, and then verifying with a dynamic pass built from test cases and proofs of concept answers the obvious objection that static scanners mostly produce false positives. The cost of that design is visible in the requirements: a container runtime, sixteen gigabytes of memory and a hundred gigabytes of disk for one full run, plus a search index and a MySQL sink that the quick start never mentions. The documentation is the weak half. The one command on the page loses its line continuation partway through and uses a comment character the shell does not recognise, and the agent example is a sentence with nothing after it, which is unfortunate because agent support is one of the three headline upgrades in version 2. Before running it, read the actual option list with the help flag rather than the page's example, decide how you will supply a GitHub token, since anonymous cloning is the default and rate limited, and check where results are expected to land. It suits a team benchmarking code generators. It does not suit anyone expecting a one command install.

Frequently asked questions

What is A.S.E (AICGSecEval)?

A repository-level benchmark from the Tencent Wukong Code Security Team for evaluating the security of AI-generated code. Its code generation tasks are derived from real GitHub projects and CVE patches, it extracts project-level code context automatically to mimic real development, and its evaluation combines static and dynamic analysis to balance detection coverage against verification precision.

What does A.S.E require to run an evaluation?

The table recommends 16GB of memory or more, 100GB of disk or more, Python 3.11 or newer and Docker 27 or newer, since the dynamic half of the evaluation runs generated code in a container. Installation is a single pip install of the requirements file, and the runner is invoked as `python3 invoke.py` with either an llm or an agent option.

How does A.S.E evaluate the security of generated code?

In stages, with a separate script for each. Code is generated with a model or an agent, context is retrieved with either a BM25 search index or another vendor's coding agent, and the result is scanned, with one script for the hybrid path and another for static analysis only. The dynamic scheme uses test cases and vulnerability proofs of concept, and the dataset claims 29 CWE types across C and C++, PHP, Java, Python and JavaScript.

Where is the A.S.E dataset and where do results go?

The commands point at a data file inside the repository's own data directory, and the dependency list includes a MySQL driver, which implies results are written to a database the quick start does not document. The HuggingFace dataset link in the readme is commented out in the markup, so the project site and the repository are where the data is described rather than a public model hub.

Official sources

  1. Issues
  2. Project website
  3. README
  4. Releases
  5. Tencent/AICGSecEval on GitHub
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/tencent-aicgseceval.svg)](https://hysenlabs.com/projects/tencent-aicgseceval)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/tencent-aicgseceval.svg)](https://hysenlabs.com/projects/tencent-aicgseceval)

Was this useful?