# investment_data publishes a signed qlib archive, not a dataset

> This project turns a Dolt database of Chinese market data into a Qlib-ready tarball, and the interesting engineering is in the release discipline rather than the finance: a manifest with ten fields, digest pinning in CI, and a documented fail-closed rollback. Data freshness depends on a Tushare token someone donated.

**chenditc/investment_data** — Scripts and doc for https://www.dolthub.com/repositories/chenditc/investment_data

- Repository: https://github.com/chenditc/investment_data
- Stars: 1,557 · Forks: 191
- Language: Python
- License: Apache-2.0
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/chenditc-investment-data

## Consume it as a release, with verification

The README's usage section is four steps, and the middle two are the reason this project is interesting.

First, download the canonical archive and its provenance manifest from the same GitHub release. Second, validate the completed archive bytes and the date-bearing members against that manifest. Third, extract to the default qlib directory:

```bash
wget "https://github.com/chenditc/investment_data/releases/download/<release-tag>/qlib_bin.tar.gz"
wget "https://github.com/chenditc/investment_data/releases/download/<release-tag>/qlib_bin.manifest.json"
python3 qlib/validate_archive.py \
  --archive qlib_bin.tar.gz \
  --manifest qlib_bin.manifest.json \
  --expected-tag "<release-tag>" \
  --require-publishable
tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=1
```

The `--require-publishable` flag is the detail to notice. A standalone manifest produced locally has `image_digest:null`, and the README says such a manifest is suitable for local inspection but cannot be published as a release asset. So the validation step can distinguish an archive built by the project's own release workflow from one built on a laptop, which is a meaningful difference when you are downloading a dataset you intend to run experiments on.

Releases are named by date, with `2026-10-15`, `2026-10-14` and `2026-10-13` among the most recent, and the last push was on 2026-10-16.

## Dolt is the source of truth, and it is versioned like code

The raw data lives in a Dolt database hosted on DoltHub at `chenditc/investment_data`, and this repository holds the scripts and documentation that produce the qlib export from it. The README describes it as scripts and doc for that DoltHub repository.

Dolt is the load-bearing choice here. A MySQL-compatible versioned database means the dataset has history, so a bad daily update is a commit to revert rather than a dataset to restore from a backup nobody kept. Contributors clone it with `dolt clone chenditc/investment_data`.

The table naming carries the provenance. Every table name is prefixed with its data source, and the prefixes are decoded in the README:

- `w` for Wind, a high quality static source available only up to 2019.
- `c` for Caihui, also high quality and also only up to 2019.
- `ts` for Tushare.
- `ak` for Akshare.
- `yahoo` for data collected through qlib's own Yahoo collector.
- `baostock` for Baostock.
- `final` for the merged, validated and corrected data.

The distinction between the first two and the rest matters more than it looks. The two high-quality sources stopped in 2019, so any period after that rests entirely on the free and semi-free sources, and the project is explicit about merging and cross-validating rather than assuming one source is right.

The merging logic is documented for the price table too. Since each stock has a different first available date, the adjusted price uses the first date's price as a factor of 1.0, and when merging sources the factors are rescaled so that every source yields the same adjusted price.

## Running the daily update yourself

The update path needs a Tushare token, obtained from tushare.pro. The README credits a specific contributor with providing the token that makes the daily update possible, which is also a fair warning about the project's single point of failure.

Exporting to qlib format runs in the project's own container image:

```bash
docker run \
  -v /<some output directory>:/output \
  -it --rm chenditc/investment_data bash dump_qlib_bin.sh
```

You can mount an existing local Dolt directory into the container to avoid re-cloning. The export script accepts an optional working directory and qlib repository as arguments, and it always produces both `qlib_bin.tar.gz` and `qlib_bin.manifest.json` together.

Running the update and the export in one step is the shape the README uses for automation:

```bash
docker run -v /<some output directory>:/output -it --rm chenditc/investment_data \
  bash -lc 'bash daily_update.sh && bash dump_qlib_bin.sh'
```

The Python dependencies are short and pinned where it matters: `tushare==1.2.89`, `baostock`, `SQLAlchemy==2.0.29`, `pymysql` and `fire`.

## Release authority and the rollback runbook

The publication rules are the most disciplined part of this repository.

Release publication is authorised only by the digest-pinned workflow on `main`, and the `upload_release.sh` script is workflow-internal, rejecting direct, local or container publication. Publication is dispatched with:

```bash
gh workflow run upload_release.yml --repo chenditc/investment_data --ref main -f operation=publish
```

Before any release mutation, the workflow validates the ten-field manifest and the complete archive, then redownloads and validates both canonical assets. Redownloading and validating after publishing is the step most pipelines skip, and it is what turns a release into something a consumer can verify.

The rollback procedure is documented as ordered and fail-closed, and it is worth reading as a template for other data pipelines. Disable the upload workflow. Query its state and require the exact result `disabled_manually`, rather than assuming the disable worked. Query both `upload_release.yml` and `data_update.yml` runs and wait until every queued or in-progress job using the shared Dolt volume has drained, because a full revert may remove the shared lock and concurrency group. Then perform the full repository revert, and re-check the disabled state afterwards.

The README also states what a revert does and does not affect: it may move the convenience `latest` image and therefore affect data update, but it cannot publish while the upload workflow is disabled, and already accepted release assets are untouched. An interrupted historical repair completes only through the fixed `repair-2026-07-20` operation, and a stale backup is never auto-restored. The external monitor is separate state and rolls back through its own tracked script, never as part of the repository revert.

## The container is pinned harder than most

The Dockerfile shows how seriously reproducibility is taken here, and it starts with Python 3.9.

The Dolt version is a build argument, `DOLT_VERSION=1.88.0`, downloaded from the upstream releases as a linux amd64 tarball and configured with a fixed user identity. The qlib revision is an explicit 40-character commit hash, and the build asserts twice: once that the `INVESTMENT_DATA_REVISION` argument looks like a lowercase 40-hex commit, and once that the qlib revision equals the expected hash. Both checks fail the build rather than warn.

qlib itself is cloned from GitHub and checked out detached at that revision, with a verification that the resulting HEAD matches, then installed in editable mode with its dev extra. The image carries OCI labels recording both the repository revision and the qlib revision, so a container can be traced back to the exact code it was built from.

The cost of this approach is that Python 3.9 and numpy 1.23.5 are dated choices for a data project, and every upstream security fix requires a rebuild rather than a pull.

The repository also carries a `release_gate.sh`, a `tests/` directory, `infra/`, `ops/` and `one_time_db_scripts/` directories, and documentation per table such as `docs/final_a_stock_eod_price.md`. The licence is Apache 2.0.

One more practical note: the README asks for sponsorship to cover a VPS with 30G or more memory, four cores or more, and good connectivity to DoltHub and GitHub, to keep the CI/CD pipeline running.

## Conclusion

Adopt this if you run qlib research on Chinese A-share data and want a dataset someone else maintains rather than your own pipeline. Do not adopt it as a basis for live trading, since the daily update depends on a donated upstream token and the older Wind and Caihui sources stop at 2019. Verify first that a release tag has both qlib_bin.tar.gz and qlib_bin.manifest.json, that the manifest validates against your downloaded archive with `--require-publishable`, and that the data sources you depend on are present, since the final tables are merged from several upstreams.

## FAQ

### How do I get the investment_data dataset?

Download qlib_bin.tar.gz and qlib_bin.manifest.json from the same GitHub release, validate them with qlib/validate_archive.py using `--expected-tag` and `--require-publishable`, then extract the tarball into `~/.qlib/qlib_data/cn_data` with `--strip-components=1`.

### Which data sources does investment_data merge?

Table prefixes identify the source: Wind and Caihui for high quality static data that stops at 2019, Tushare, Akshare, Baostock and qlib's Yahoo collector, with `final` tables holding the merged, validated and corrected data.

### What does running a daily update require?

A Tushare token exported as TUSHARE, obtained from tushare.pro, plus either a cloned Dolt database or the project's container. The daily update and the qlib export are usually run together with `bash -lc 'bash daily_update.sh && bash dump_qlib_bin.sh'` in the container.

### Who can publish a release of investment_data?

Only the digest-pinned workflow on the main branch. The upload_release.sh script rejects direct, local and container publication, and the workflow validates the ten-field manifest and the archive before mutating the release and then revalidates the downloaded assets.

## Sources

- [chenditc/investment_data on GitHub](https://github.com/chenditc/investment_data)
- [Issues](https://github.com/chenditc/investment_data/issues)
- [License: Apache-2.0](https://github.com/chenditc/investment_data/blob/main/LICENSE)
- [README](https://github.com/chenditc/investment_data/blob/main/README.md)
- [Releases](https://github.com/chenditc/investment_data/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/chenditc-investment-data