ArkEval: an executable code repair benchmark for ArkTS issues

A repository-level repair benchmark, not a single-function one

The project positions itself as an executable repository-level benchmark for automated program repair, built on real ArkTS and OpenHarmony issues, and it frames the goal as turning scattered repair cases into an executable, comparable and continuously improvable LLM4Code research task.

The headline numbers define the scope. There are 502 real issue-fix instances drawn from 9 public source repositories, 149 official sample applications from OpenHarmony, 502 executable reproduction tests, and 8 models in the main experiment. The underlying motivation is stated as well: for languages with little training corpus, there has been a long-standing lack of a standard evaluation signal, and ArkEval is meant to supply a common experimental basis for model comparison, agent search, training feedback and failure attribution.

The paper has been accepted at ASE 2026 in the research track, with an arXiv entry and a reproduction package on figshare, so the artefact and the write-up are both published. The repository itself is Python, and the target language is ArkTS, the HarmonyOS development language, with the OpenHarmony SDK as the execution environment.

The scale is what makes it interesting. Repair benchmarks built from a single function are easy to score and easy to cheat, and a benchmark of whole repository issues with an end-to-end compile and test loop is a considerably harder thing to satisfy.

Compile@1 and Pass@1 are deliberately kept apart

The most important design decision in the benchmark is stated as an important note: ArkEval evaluates being able to compile separately from actually being fixed. `Compile@1` measures build validity, and `Pass@1` is decided automatically by the behaviour-level reproduction tests.

The main experiment ran 8 models over all 502 instances under the same localisation guidance, with RAG off and a single-patch protocol. The results show why the separation matters:

| Model | Compile@1 | Pass@1 | |---|---:|---:| | mimo-v2.5-pro | 268 (53.39%) | 30 (5.98%) | | deepseek-v4-pro | 208 (41.43%) | 26 (5.18%) | | kimi-k2.7-code | 62 (12.35%) | 16 (3.19%) | | minimax-m3 | 223 (44.42%) | 36 (7.17%) | | qwen3.7-max | 187 (37.25%) | 31 (6.18%) | | glm-5.2 | 273 (54.38%) | 42 (8.37%) | | gpt-5.6-sol | 172 (34.26%) | 97 (19.32%) | | openPangu-2.0-Flash | 170 (33.86%) | 20 (3.98%) |

The README draws the conclusion explicitly: the highest compile rate and the highest behavioural pass rate were obtained by different models. glm-5.2 compiles the most at 54.38 percent, and gpt-5.6-sol passes the most at 19.32 percent while compiling less than half the time. A benchmark that reported only compile success would have ranked these two the other way round.

The dataset is one JSONL file with two withheld fields

The main data file is `dataset/arkeval_dataset.jsonl`, where each line is one instance. The fields are listed in the README:

instance_id, org, repo, number, state, title, body, base,
resolved_issues, defect_files, fix_patch, test_patch,
fixed_tests, f2p_tests, run_result

Two of those fields carry the weight of the benchmark. `fix_patch` is the curated reference developer patch, and `test_patch` is the executable reproduction test. The README is explicit that neither is exposed to the repair model during model evaluation, which is what makes the benchmark a measurement rather than a copy of the answer.

The dataset statistics give a sense of its composition. There were more than 6,000 candidate PR or MR records, of which 502 became issue-resolution instances, and 149 are official sample applications, with 414 of the 502 coming from `applications_app_samples`. Most fixes are small, with 474 of 502 changing fewer than 300 lines, and 417 of 502 have an `.ets` file share of at least 70 percent. The resolution states are 498 merged, 2 closed and 2 open.

So this is a benchmark of many small, well-localised fixes in one language, drawn from real application repositories, with the repository state captured per instance so the fix can be applied and tested in context.

Tests are written around behaviour, then gated three ways

The test construction is where most of the effort went, and the README is specific about it. The tests are not mechanically extracted from the patch. They are built around the externally observable behaviour described in the issue.

A candidate test only enters the benchmark if it clears three gates. Three agents must agree on it, it must satisfy `base + test = FAIL` and `base + reference patch + test = PASS` in the same environment, and it must pass a final black-box semantic review by three experts.

The two-state run is the part that catches bad tests. A test that passes before the fix is broken, because it does not discriminate, and a test that fails even with the reference patch applied is broken in a different way. Requiring both halves means a test has to actually describe the bug.

The iteration numbers show what that costs. Across six full-benchmark iterations, the acceptance rate per iteration rose from `87/502 (17.33%)` in V1 to `28/34 (82.35%)` in V6, and the remaining 6 long-tail instances completed the loop after targeted feedback.

Read as a whole, the numbers mean most of the first pass of generated tests was rejected, and it took six rounds of tightening to reach a steady state. For a user of the benchmark that is good news, because the surviving tests are the ones that survived a filter designed to throw them away.

Localisation is a two-stage pipeline, and its ceiling is stated

The repair framework section describes five steps, and they are worth following because they explain where the difficulty sits.

First, the defective version of the repository is scanned for `.ets` and `.ts` files, which are split with an ArkTS-aware structural splitter. Second, the code blocks are encoded with Qwen3-Embedding-8B and the top-k candidate files are retrieved from Milvus. Third, the repair model performs core file selection and dependency completion, which forms a constrained repair scope. Fourth, the model outputs a unified diff only, and the optional RAG supplies nothing but Huawei and OpenHarmony official syntax, compile error analysis and example code. Fifth, patch applicability, Hvigor compilation, the reproduction tests and the usable regression tests are run in sequence, which yields `Compile@1` and `Pass@1`.

The shared embedding stage reached `304/502 (60.56%)` candidate file Hit@10 on the 502 issues. The README immediately qualifies that number: it measures whether the top ten contains at least one file related to the reference fix, and it is not the same as final localisation accuracy.

That qualification is the right one to hold on to. A retriever that puts a relevant file in the top ten out of hundreds has not solved the problem, and the gap between 60.56 percent Hit@10 and the 8.37 percent best Pass@1 is the size of the remaining problem.

Running it means a real OpenHarmony environment

The quick start is Windows-first and the requirements are heavy. The recommended environment is Windows 10 or 11, Python 3.11, Node.js 18 or later, Git, DevEco Studio, the OpenHarmony SDK, HDC, and an available emulator or device.

The installation is a virtual environment and the pinned dependencies:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
npm ci --prefix .\localization\localization_engine\ast\node
Copy-Item .env.example .env

You then fill in the model endpoints, the embedding service and the Milvus configuration in `.env`. The example file shows the shape of that configuration:

# OpenAI-compatible repair/localization endpoint
OPENAI_API_KEY=replace_with_your_key
OPENAI_API_BASE_URL=https://your-endpoint.example/v1
OPENAI_MODEL=replace_with_model_id

with a separate localization LLM block, `MILVUS_HOST` and `MILVUS_PORT`, and an embedding backend pointing at a DGX service running `Qwen/Qwen3-Embedding-8B`. The README is clear that the repository does not provide API keys, the SDK, third-party dependency caches, emulator images or the pool of repair repositories.

Once a repository pool exists under `depend/repair_repo/run01/<repo>`, a run is driven by dataset row index:

python .\run_arkts_pipeline.py `
  --dataset .\dataset\arkeval_dataset.jsonl `
  --rows 1 `
  --repo-pool .\depend\repair_repo `
  --localization-repo-pool .\depend\repair_repo\run01 `
  --model-name <model-id>

and patches are evaluated separately with `python .\evaluation\run_llm_patch_eval.py`.

What the repository does not ship, and how Pass@1 is decided

The reproducibility section is the most useful part of the README for anyone planning to use this, because it states plainly what is missing.

The GitHub directory provides the core source, the main dataset and the display assets, and it does not package roughly 20 GB of SDK, dependency caches, repository pools and run outputs. Reproducers have to install the DevEco and OpenHarmony toolchain themselves and restore the public source repositories, while the paper reproduction package provides audit material and experiment records.

The evaluation rules are stated as well, and they are strict. This is a full-chain evaluation that requires a real OpenHarmony project, the SDK and a device environment, and it does not support substituting static checks or patch text matching for the run results. `Pass@1` is decided entirely by an automatic oracle: where an existing regression suite exists, the run must compile and both the reproduction test and the regression test must pass, and where no existing regression suite exists, it must compile and the reproduction test must pass.

Two protocol details matter for comparison. The main eight-model results are all RAG-off, and the RAG results come from a separate four-model paired experiment, which the README says are not mixed with the main leaderboard. In that paired experiment, official-knowledge RAG gives a Compile@1 gain of 4.38 to 11.75 percentage points and a Pass@1 gain of 0.80 to 3.19 points, and the README is careful to call these descriptive comparisons that do not claim statistical significance.

Editorial conclusion

ArkEval fits a research team working on automated program repair for ArkTS or on a low-resource language, and that needs a benchmark whose tests are behaviour-based rather than patch-derived. It is the wrong tool for anyone without an OpenHarmony toolchain, SDK and device, since the evaluation cannot be substituted with static checks or patch text matching. Verify first that you can provision that environment and the repository pool, that the 20 GB of artefacts the repository does not ship are obtainable, and that the RAG-off protocol matches the experiment you intend to run.

Frequently asked questions

Official sources

1. Issues
2. README
3. SJTUjinmao/ArkEval on GitHub