A.S.E (AICGSecEval): A Repository-Level Benchmark for AI-Generated Code Security
A.S.E (AICGSecEval) is a repository-level AI-generated code security evaluation benchmark developed by Tencent Wukong Code Security Team.
At a glance
- What is it?
- Tencent's Wukong Code Security Team publishes a project-level evaluation framework that asks whether AI coding tools write secure code inside a real repository, not in an isolated snippet. The design is more realistic than single-file benchmarks, but the hardware bar, the Docker dependency and the NOASSERTION licence are all things to settle before you commit a CI slot to it.
- Who is it for?
- Adopt A.S.E if you need to compare LLM or agent coding tools on security outcomes inside a real project context and you can provision the stated 16GB memory, 100GB disk, Python 3.11 and Docker 27 environment. Do not adopt it as a fast linting gate for a small team: a full run is documented as potentially long, the disk requirement is steep, and the repository does not declare a recognised licence.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 114 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap A.S.E is trying to close: snippet benchmarks versus repository reality
Most code security benchmarks hand a model a function and ask it to complete the function. That setup is cheap to grade and easy to reproduce, but it strips out everything that makes real code insecure: the surrounding modules, the existing helper the model should have reused, the project conventions that determine whether a string reaches a query. A.S.E, short for AICGSecEval, is built around the opposite premise. The README describes it as a project-level benchmark that simulates real-world development workflows, with generation tasks derived from real GitHub projects and authoritative CVE patches. The intended audience is therefore narrow and specific: teams evaluating LLMs or agentic coding tools who care about whether the generated patch introduces a vulnerability in the context of the repository it lands in. If your question is "can this model write a correct binary search," A.S.E is not aimed at you. If your question is "does this agent, given the project context, produce a patch that a security reviewer would reject," that is the question the framework is designed to answer.
What the 2.0 release actually changed, and what the numbers mean
The README lists three upgrades in 2.0. The dataset now covers 29 CWE vulnerability types across C/C++, PHP, Java, Python and JavaScript, drawing on risks from the OWASP Top 10 and CWE Top 25. The evaluation target was extended to agentic programming tools, not just raw model completions. The evaluation method moved to a static and dynamic hybrid: static analysis for breadth, plus dynamic evaluation based on test cases and vulnerability PoCs for verification precision. Those three changes pull in different directions and the third is the expensive one. Static-only scoring is cheap and reproducible but produces false positives; adding dynamic PoC execution raises confidence but requires the project to actually build and run inside a container. The repository's system requirements reflect that cost: Docker 27 or newer, at least 100GB of disk, and 16GB of memory recommended. The release history shows v2.0.0 in November 2025 and v2.0.1 in December 2025, with a separate report-v1.0 tag published the day after v2.0.1. The README does not state how many tasks are in data_v2.json, so treat any task count you see elsewhere as unverified.
How the evaluation pipeline is wired together
The entry point is a single script, invoke.py, which branches on a required mode flag: --llm for direct model evaluation, --agent for agentic tool evaluation. Both modes take a shared set of arguments: --batch_id to label the run, --dataset_path pointing at the dataset (the README example uses ./data/data_v2.json), --output_dir for results, --max_workers for concurrency, and --github_token. The README notes that without a GitHub token the tool falls back to anonymous cloning and may hit clone rate limits, which tells you the pipeline clones repositories at runtime rather than shipping them pre-vendored. The agent path is the more interesting design choice. The launcher forwards any argument it does not recognise in its own help output to the selected agent module for parsing. That is why the Claude Code example can pass --claude_api_url, --claude_api_key and --claude_model through invoke.py even though those flags belong to the agent integration. The README states that different agents may require distinct configurations such as model parameters, credentials or APIs, so the forwarding rule is the extension mechanism rather than a fixed plugin interface. The README also states that checkpoint recovery is supported: if a run is interrupted, rerunning the command resumes from the last checkpoint. For a benchmark whose full run is described as potentially long, that is a practical necessity rather than a feature.
Getting a first run off the ground
The documented path is short. Install dependencies with pip install -r requirements.txt, then run python3 invoke.py followed by either --llm or --agent and the relevant options. For a model run the README gives: python3 invoke.py --llm --model_name gpt-4o-2024-11-20 --base_url https://api.openai.com/v1/ --api_key sk-xxxxxx --batch_id v1.0 --dataset_path ./data/data_v2.json --output_dir ./outputs --max_workers 1 --github_token xxxxx. For an agent run it gives the claude_code example with --agent_name claude_code, --batch_id v1.0, --dataset_path ./data/data_v2.json, --claude_api_url, --claude_api_key and --claude_model claude-sonnet-4-20250514. Run python3 invoke.py -h to see the full option list, which is also the authoritative way to discover which flags belong to the launcher and which will be forwarded. Two operational notes from the README matter here. First, --max_workers controls concurrency, and the README says raising it reduces total runtime, so the default appears conservative. Second, the GitHub token is optional but recommended; anonymous cloning is subject to rate limiting. The README does not document an offline dataset format or a way to skip cloning, so plan for network access during evaluation.
Where this benchmark will frustrate you
The resource envelope is the first constraint. A 100GB disk floor and 16GB memory recommendation put A.S.E out of reach for a laptop-based workflow and into a dedicated machine or container host. The README's own note that a full evaluation may take a long time, with concurrency as the only documented lever, means you should not expect a fast feedback loop. The second constraint is scope. The benchmark covers 29 CWE types, which is a deliberate slice, not full coverage. Vulnerabilities outside that set are not measured, and the README does not describe how tasks are sampled or weighted, so a high score means the tool handled the covered CWEs, not that it writes secure code in general. The third issue is that the README does not document the dataset schema, the scoring formula, or how static and dynamic results are combined into a final number. The website's rank page is the place results are published, but the repository itself does not spell out the aggregation. If you need to explain a score to a reviewer, you will be reading the paper (arXiv 2508.18106) rather than the README. Finally, the licence field is NOASSERTION, which means GitHub could not identify a standard licence. The README has a License section in its table of contents, but no licence terms were visible in the material provided. That is a real blocker for commercial adoption and should be resolved before any internal rollout.
How it differs from running your own static analysis
The obvious alternative is to skip the benchmark and run Semgrep, CodeQL or a similar static analyser directly over the output of your coding assistant. The difference in approach is fundamental. A static analyser applied to a diff tells you whether that diff trips a rule. A.S.E instead defines a fixed task set derived from real projects and CVE patches, runs the model or agent against it, and grades the result with both static rules and dynamic PoC execution. That gives you a comparable number across tools and across model versions, which a local lint run does not. The trade-off is that you inherit the benchmark's CWE coverage, its task construction and its grading logic, and you cannot easily add your own project's idioms to it. A static analyser also runs in seconds on a diff and fits in CI; A.S.E needs Docker, a large disk and a long run. These are complementary rather than competing. If the question is "is this specific pull request safe," use the analyser. If the question is "should we switch our assistant from model A to model B, or from completion to an agent," A.S.E provides the controlled comparison that a local rule set cannot.
Maintenance, versioning and the licence question
The repository is active, not archived, with the latest push recorded in May 2026 and releases at a steady cadence through late 2025. Versioning is visible in the tags: v2.0.0, v2.0.1, plus a separate report-v1.0 tag for the published research report. The dataset file is versioned by name, data_v2.json, and the README's examples pass --batch_id v1.0, which suggests you should record both the dataset file and the batch identifier alongside any result you publish. The upgrade cost is mostly environmental. Because the pipeline clones repositories and runs dynamic PoCs inside Docker, moving between releases is less about code changes and more about re-validating that your container host still meets the Docker 27 and disk requirements. The licence situation is the open item. NOASSERTION is not a licence; it means the repository's terms were not machine-identifiable. The README's table of contents includes a License section, but the terms were not present in the material available here, and this is not legal advice. Anyone planning to use A.S.E in a commercial evaluation pipeline should confirm the terms directly with the maintainers before relying on it.
Editorial conclusion
Adopt A.S.E if you need to compare LLM or agent coding tools on security outcomes inside a real project context and you can provision the stated 16GB memory, 100GB disk, Python 3.11 and Docker 27 environment. Do not adopt it as a fast linting gate for a small team: a full run is documented as potentially long, the disk requirement is steep, and the repository does not declare a recognised licence. Before running anything, verify the licence terms for your intended use, confirm the dataset file path and schema in ./data/data_v2.json matches the version you pin, and check whether your target agent needs extra arguments that invoke.py will forward to its module rather than reject.
Community notes