MingLi-Bench: Scoring LLMs on Bazi and Ziwei Doushu Multiple-Choice Questions

The gap MingLi-Bench fills: chart reading versus chart construction

Most LLM evaluation sets measure either general knowledge or coding. Chinese metaphysics sits outside both, and the failure mode is specific: a model can compute a Bazi or Ziwei chart from a birth date and still give a wrong reading, or it can produce a plausible reading from a chart it derived incorrectly. Those two errors are indistinguishable in a single end-to-end score. MingLi-Bench separates them. The README describes data/fortune_api_results.json as pre-computed Bazi and Ziwei charts generated via iztro, keyed by case_id and joined to data.json at runtime when --astro is set, with the stated purpose of isolating pure reasoning from chart derivation. That is the project's actual contribution. The questions themselves come from the annual Global Fortune Teller Competition for 2022 through 2025, normalized into 160 multiple-choice items across twelve life aspects: career, health, appearance, marriage, children, education, legal trouble, family, personality, calamity, wealth, and fortune. Scoring is exact match against a ground-truth answer. The audience is narrow and identifiable: researchers and engineers who want a reproducible number for how well a model reasons over a structured Chinese chart, and who are willing to supply their own API keys to get it.

How a run is assembled: .env, provider routing, and the astro join

The CLI reads keys and defaults from a .env file, copied from .env.example. Empty or placeholder values are skipped, so you only fill in the providers you call. Supported keys include OPENROUTER_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, DEEPSEEK_API_KEY, and DOUBAO_API_KEY. Doubao is the odd one out: it needs both DOUBAO_API_KEY and DOUBAO_ENDPOINT_ID, plus DOUBAO_BASE_URL pointing at https://ark.cn-beijing.volces.com/api/v3. Defaults are TIMEOUT=60, MAX_WORKERS=5, MAX_TOKENS=8192, and TEMPERATURE=0.0. That last default matters. At temperature zero the benchmark is closer to deterministic, which is what you want for a leaderboard, but it also means a model that would answer correctly on a second sample gets scored once. Routing is inferred rather than configured in the common case. If --model contains a slash, the CLI treats it as an OpenRouter id. Otherwise it infers the provider from the name prefix: gpt-*, claude-*, gemini-*, deepseek-*, doubao-*. When inference fails, for example with a versioned Doubao endpoint id, you pass --platform explicitly, and it accepts openai, openrouter, anthropic, google, deepseek, or doubao. The OpenAI-compatible path is the escape hatch: set OPENAI_BASE_URL to any gateway and pass --platform openai with an arbitrary model name. The --astro flag is the piece that changes what is being measured. With it on, the pre-computed chart for that case_id is inserted into the prompt, so the model never converts a birth date into a chart. With it off, the model does that conversion itself and any error propagates into the final answer.

Getting a first score: the commands the README gives

Installation is three lines: git clone https://github.com/DestinyLinker/MingLi-Bench.git, cd MingLi-Bench, pip install -r requirements.txt. Python 3.9 or newer. Then cp .env.example .env and fill in one provider. Before spending money on a full run, the README offers two sanity checks that exit without calling a model: python -m mingli_bench.cli --list-models prints supported model names, and python -m mingli_bench.cli --stats prints dataset statistics, optionally narrowed by --year. A smoke test is available through --sample, for example -s 10 to score only the first ten questions. A typical scored run through OpenRouter looks like this: python -m mingli_bench.cli --model openai/gpt-4o --year 2025 --cot --astro --max-workers 8. The README recommends always passing --cot and --astro, on the reasoning that Chain-of-Thought gives the model room to work through the chart and --astro keeps the score about reasoning rather than date-to-chart conversion, and that they should be dropped only when you deliberately want to ablate one of those effects. Concurrency is controlled by --max-workers, default 5, with the README suggesting 8 to 16 if rate limits allow and lowering it under throttling. Two flags are worth knowing about even though they are not in the recommended defaults. --categories filters by Chinese category names, for example --categories 事业 婚姻, and --shuffle-options randomizes option order per question to guard against position bias. If you are comparing two models, running both with --shuffle-options is the cheaper way to find out whether one of them is just picking the second option more often.

What lands on disk, and why the raw responses matter more than the summary

Each run writes three artifacts under --output-dir, which defaults to logs/. The first is <model>_results.json, holding per-question predictions, scoring, and aggregates. The second is <model>_summary.txt with the headline numbers. The third is a <model>_responses/ directory containing raw model responses, one file per question. The third artifact is the one that makes this benchmark usable for diagnosis rather than just ranking. A summary score tells you a model got 60 percent; the per-question raw responses tell you whether the misses cluster in one category, whether the model refused to answer, or whether it produced a chart in its reasoning that contradicts the injected one. The --no-save flag suppresses all of this and prints to terminal only, which is fine for a smoke test and a bad idea for anything you intend to cite. Note that the output filenames are keyed on the model name, so two runs of the same model at different --year values will collide in the same output directory unless you separate them with -o. That is a small operational detail, but it is the kind of thing that silently overwrites a 2024 run with a 2025 run if you are not watching the directory.

The sample size is the first limitation you will hit

160 questions total, spread across four years and twelve categories. Filter to --year 2025 and you are scoring a fraction of that. Filter further by --categories 事业 婚姻 and the per-cell count gets small enough that a two-question swing moves the percentage by several points. The README does not publish per-year or per-category counts, so the only way to find out is to run --stats yourself before committing to a comparison. This is not a flaw in the implementation; it is a property of the source. The questions come from a single annual competition, and a competition has a fixed number of questions. The practical consequence is that MingLi-Bench is better suited to coarse comparisons than fine ones. Saying model A beats model B by 15 points is defensible. Saying model A beats model B by 2 points is not, and no amount of --max-workers will fix that. A second limitation is the exact-match scoring on multiple-choice items. That format is what makes the benchmark cheap and reproducible, but it also means a model that reasons correctly and then picks the wrong option is scored identically to a model that reasons nonsense and guesses right. The --cot flag gives the model room to reason, but the score ignores the reasoning. If your actual question is whether a model can write a useful consultation, this benchmark does not answer it.

Where --astro stops being a convenience and starts being a crutch

The recommended default of --cot --astro is the right choice for the stated goal, comparing reasoning over a fixed chart. It is the wrong choice if you want to know whether a model can do the whole task. The README is explicit that --astro injects pre-computed charts so the model does not have to derive them from the birth date. Turn it off and you are testing two skills at once, with no way to attribute a miss to either. That is a legitimate ablation, and the README frames it that way, but it means the headline number from a default run says nothing about a model's ability to build a chart. Anyone reading a MingLi-Bench score needs to know which flags produced it. A score reported without --astro is not comparable to one reported with it, and the benchmark's own documentation treats them as different measurements rather than a difficulty setting. The chart data itself comes from iztro, a separate project. MingLi-Bench does not validate that those charts are correct; it consumes them. If iztro's output for a given case is wrong, the --astro run will score the model against a wrong premise, and the results file will not flag it. That is an assumption baked into the design, and it is worth stating plainly rather than discovering later.

BaziQA and the difference between asking and scoring

The README points to BaziQA (Chen et al., 2025) as the reference for Bazi-specific benchmarking. The two projects answer different questions, and the difference is worth being precise about. BaziQA is a question-answering dataset: the model produces free-form answers about a Bazi chart, and evaluation is a harder problem because there is no single correct string to match against. MingLi-Bench is a scoring harness: fixed multiple-choice items, exact-match grading, a CLI that fans out concurrent API calls and writes per-question artifacts. The trade is real in both directions. Multiple-choice gives you a number that two people running the same command will agree on, at the cost of not measuring whether the model can express a reading. Free-form gives you something closer to the actual use case, at the cost of needing a judge or a rubric, which reintroduces the subjectivity the benchmark format was meant to remove. If your goal is a reproducible comparison across many models, MingLi-Bench's format is the right one. If your goal is to know whether a model's written readings are any good, a multiple-choice score is a proxy at best, and the README does not claim otherwise.

Maintenance, licence, and what to verify before citing a number

The repository is MIT licensed, which permits commercial use, modification, and redistribution provided the copyright notice and permission notice are retained. The benchmark data is drawn from the Global Fortune Teller Competition, and the README does not state what licence, if any, covers the underlying competition questions; the MIT badge covers the repository, not necessarily the provenance of every item inside data/raw/. If you plan to redistribute the dataset rather than just run the CLI against it, that is the question to resolve first. On maintenance: there are no retrieved releases, so versioning is whatever the main branch contains at the commit you clone, and the last push recorded is 2026-05-09. The data directory covers 2022 through 2025, which suggests an annual refresh cadence tied to the competition, but nothing in the material confirms a commitment to that schedule. Upgrade cost is therefore low in the mechanical sense, since there is no package to pin and no migration path, but it also means a clone today and a clone in six months may not produce identical numbers if the questions or the chart file change. For a benchmark, that is the risk that matters. If you intend to publish a score, record the commit hash alongside the flags you used, and run --stats at that commit so the question counts in your write-up match the data you actually scored against.

Editorial conclusion

Adopt MingLi-Bench if you are comparing models on structured chart reading and want the chart derivation step held constant: run --cot --astro and use --shuffle-options to check position bias. Do not adopt it if you need free-form consultation quality, a large sample per year, or a benchmark that tests date-to-chart conversion. Before trusting any number, run python -m mingli_bench.cli --stats for the year you intend to score, confirm the per-year question count is large enough for your comparison, and check whether the model you are testing appears in --list-models output.

Official sources

1. DestinyLinker/MingLi-Bench on GitHub
2. Issues
3. License: MIT
4. Project website
5. README