# PolyLedger: a resumable Polymarket indexer that lands in one DuckDB file

> PolyLedger pulls CLOB market metadata and Polygon OrderFilled logs into a single DuckDB database with SQL joins and Parquet export. It is built for analysts who want a local copy of Polymarket trade history, not a hosted data product.

**nahrek/polyledger** — Resumable Polymarket indexer: CLOB market metadata plus on-chain trades from Polygon, in one DuckDB file you can query with SQL

- Repository: https://github.com/nahrek/polyledger
- Stars: 600 · Forks: 100
- Language: Python
- License: MIT
- Published: 2026-09-17 · Updated: 2026-09-17 · Language: en
- Canonical page: https://hysenlabs.com/projects/nahrek-polyledger

## The gap PolyLedger fills: one queryable file instead of two APIs

Polymarket data lives in two places that do not join cleanly. Market questions, slugs, tick sizes and outcome tokens come from the CLOB and Gamma REST APIs. Executed trades live on Polygon as OrderFilled events emitted by the exchange contracts. Reconstructing a volume-by-market table means paging REST responses, streaming logs, matching token ids to markets, and doing the price arithmetic yourself. PolyLedger does that work once and writes the result into a DuckDB file.

The audience is narrow and specific: someone who wants to run SQL against Polymarket history locally, in pandas or polars or the DuckDB CLI, without standing up a warehouse or paying for a data vendor. The README frames the output plainly as a single DuckDB file you can query with SQL immediately. If you already have a hosted indexer you like, this project is not competing for your attention.

## How the pipeline moves from CLOB listings to a joined trades view

There are four stages, and the CLI exposes each one separately. markets pulls every market from the CLOB API and normalises responses through Pydantic models. chain streams OrderFilled logs from Polygon through Envio HyperSync. backfill resolves token ids that were missing from the CLOB listing by asking the Gamma API, and flags them rather than dropping them. trades materialises the join between raw fills and market metadata.

The price columns are the part worth reading closely. price, shares and usd_size are not on chain. The README states they are derived from makerAmountFilled and takerAmountFilled according to the maker's side: when the maker buys, their amount is collateral and the taker's is shares, and when the maker sells it is reversed. Both amounts are base units with 6 decimals, matching pUSD. Because the direction of that arithmetic depends on which side you are reading, PolyLedger stores maker_side and taker_side as explicit BUY/SELL columns instead of one ambiguous side field. That is a deliberate design choice, and it is the right one: any dataset that collapses the two sides forces you to guess whose perspective a row is written from.

Raw decoded logs stay available in order_fills with no interpretation applied. The README says you can build your own join from there if you disagree with the price math, which is the honest way to ship a derived column.

## Installing PolyLedger and running a first bounded sync

The README gives a clone-and-editable-install flow. The commands below are copied from the installation section, including the Windows activation line it shows; on macOS or Linux substitute your platform's activate script.

```bat
git clone https://github.com/nahrek/polyledger
cd polyledger
python -m venv .venv
.venv\Scripts\activate.bat
pip install -e .
```

Python 3.11 or newer is required. The install pulls hypersync, duckdb, httpx and pydantic. After installing, the polyledger executable is on your path; without installing, python -m polyledger and python -m polyledger.cli are equivalent, and the entry point is main() in polyledger/cli.py.

Before anything runs you need a HyperSync token. The README states a token has been required since November 2025 and that the free tier is sufficient, with registration at envio.dev. Copy .env.example to .env and set the value. PolyLedger reads .env from the working directory at startup, so there is no export step, and real environment variables take precedence if you prefer them.

```bash
HYPERSYNC_BEARER_TOKEN=
POLYLEDGER_DB=data/polyledger.duckdb
POLYLEDGER_CONTRACTS=v2
POLYLEDGER_HTTP_RPS=8
POLYLEDGER_HTTP_CONCURRENCY=8
```

Those keys come from .env.example. POLYLEDGER_CONTRACTS accepts v2, v1 or all, and the REST politeness keys control a shared token-bucket limiter used across the CLOB and Gamma sources.

The README recommends verifying the setup on a small slice before committing to a full backfill. The three commands below run metadata only, then roughly five days of Polygon history, then print what landed.

```bash
polyledger markets
polyledger chain --max-blocks 200000
polyledger stats
```

stats prints row counts, the block range, the time range and the checkpoints. The README's sample output shows markets, tokens, fills, unmatched_fills, first_block, last_block, first_time, last_time and a per-stream checkpoint line such as order_filled:v2. If those numbers are non-zero and the block range matches what you asked for, the pipeline is wired correctly.

Querying is a single statement through the CLI:

```bash
polyledger query "SELECT question, sum(usd_size) AS volume FROM trades GROUP BY 1 ORDER BY 2 DESC LIMIT 10"
```

Or open the file directly, which the README also demonstrates:

```python
import duckdb
con = duckdb.connect("data/polyledger.duckdb", read_only=True)
df = con.sql("SELECT * FROM trades LIMIT 10").df()
```

The full pipeline is polyledger sync, which runs markets, chain, backfill and trades in order. The README warns the first backfill is slow, anywhere from a few hours to a couple of days on the free HyperSync tier depending on how much history you want, and that it is safe to interrupt at any point.

## Why interruption is safe, and what that costs you

The resumability claim is not marketing. The README states that rows and the block cursor are committed in a single transaction, so an interrupted run resumes exactly where it stopped. Separately, order_fills is keyed on (transaction_hash, log_index) and inserted with ON CONFLICT DO NOTHING, which makes re-running a range a no-op rather than a duplicate-generating mistake.

Those two properties together are what let the first backfill survive a laptop sleeping, a dropped connection, or a Ctrl-C. The cost is write amplification: committing the cursor with the data means you cannot batch the cursor update cheaply and separately. That is why POLYLEDGER_FLUSH_EVERY exists, defaulting to 50000 in .env.example. Lower it and you get finer resume granularity at the price of more transactions. Raise it and each commit is heavier, so an interruption loses more progress. There is also POLYLEDGER_REORG_BUFFER, default 64, which is the project's acknowledgement that a chain cursor is not automatically safe against a reorg; the README does not explain the buffer's exact semantics beyond its name.

Two contract generations are handled, the V2 OrderFilled layout and the older V1 one, with separate checkpoints per stream. That matters because a single cursor across two contracts with different event shapes would be wrong after any partial run. The README does not document a migration path if you start with --contracts v2 and later want v1 history; you would be looking at a fresh stream and a fresh checkpoint.

## Where PolyLedger is the wrong tool

Freshness is the first boundary. This is a batch indexer with a block cursor, not a streaming feed. If your use case needs a fill in your system within seconds of confirmation, a DuckDB file being written by a periodic run is the wrong shape.

The second boundary is the upstream dependency. The README states a HyperSync API token is required, and that requirement is dated to November 2025. Without a token the chain stage does not run at all, which means no fills and nothing for the trades view to join. The CLOB and Gamma APIs are public and need no credentials, so markets alone can be synced without Envio, but that gives you metadata with no executions attached.

The third is the derived price columns. The README is explicit that price, shares and usd_size are computed from makerAmountFilled and takerAmountFilled according to the maker's side. If your definition of volume, or your treatment of fees, differs from that convention, the numbers in trades will not match yours. The escape hatch is order_fills, which the README describes as raw decoded logs with no interpretation applied. Anyone doing accounting-grade work should start there.

Finally, there are no releases. The repository has no tags or published artifacts, so installation is a git clone plus pip install -e ., and every upgrade is a pull from main. The README's Development section and tests/ directory suggest pytest is the intended test runner, with dev extras declared in pyproject.toml.

## Alternatives: Dune, subgraphs, or your own HyperSync client

Dune Analytics is the most common substitute. You write SQL against decoded Polymarket tables that Dune maintains, and you get a hosted query engine, dashboards and sharing. The difference in approach is ownership: Dune holds the data and the decode, you hold a query. PolyLedger inverts that. You hold a DuckDB file on your own disk and you can join it against anything else you have locally, but you also own the backfill, the disk usage and the upgrade.

A subgraph is the other route. It gives you a GraphQL endpoint over indexed events with the indexing hosted, and it is a good fit when the consumer is an application rather than an analyst. PolyLedger's output is a columnar file read by SQL, which suits ad hoc analysis and pandas or polars work far better than paginated GraphQL.

The third alternative is writing the HyperSync client yourself. The README credits Envio HyperSync as the transport, and the hypersync package is a declared dependency, so the streaming layer is not the hard part. What you would be rebuilding is the schema validation, the separate V1 and V2 decoders with independent checkpoints, the Gamma backfill for token ids missing from the CLOB listing, the shared rate limiter across REST sources, and the exactly-once write semantics. That is a real amount of work, and it is the actual product here.

## Licence, upgrade cost, and what the repository does not say

PolyLedger is MIT licensed, stated both in the repository licence file and in pyproject.toml as license = { text = "MIT" }. MIT is permissive: you can use, modify and redistribute it, including commercially, provided the copyright notice and permission notice travel with it. That is a description of the licence text, not legal advice; if you are redistributing it inside a product, read the LICENSE file yourself.

Upgrade cost is the thing to weigh. There are no versioned releases, so there is no changelog to read before pulling. The last push to the repository was on 2026-09-07, which is recent, but recency of commits is not a stability guarantee. The dependency floor is hypersync>=0.9, duckdb>=1.0, httpx>=0.27 and pydantic>=2.6, all lower bounds rather than pins, so a fresh pip install -e . can resolve to newer versions than the author tested against. If you need reproducibility, pin them in your own environment.

The README does not document rollback, and it does not document what happens to an existing database if a future version changes the schema. Given that the fills table is keyed on (transaction_hash, log_index) and the cursor is committed transactionally, a destructive schema change would be the scenario to watch for. There is no migration tooling described. If you are running this in anything long-lived, keep the DuckDB file and the Parquet export (polyledger export --out DIR) so you can rebuild from a known-good snapshot.

## Conclusion

Adopt PolyLedger if you want a local, SQL-queryable copy of Polymarket fills and metadata and you are willing to manage a DuckDB file and a HyperSync token. Skip it if you need a hosted API, sub-second freshness, or you cannot obtain a HyperSync token, since the README states the free tier is required and no release artifacts exist yet. Verify first that the derived price math matches how you define a trade: read order_fills and the maker_side/taker_side columns before trusting trades. Also confirm your disk budget for the initial backfill, which the README says can run from a few hours to a couple of days on the free HyperSync tier.

## FAQ

### What is PolyLedger and who is it for?

It is a resumable indexer that pulls Polymarket CLOB market metadata and Polygon OrderFilled events into a single DuckDB file. It is aimed at analysts who want to run SQL locally over Polymarket trade history rather than querying a hosted service.

### Does PolyLedger need an API key?

Yes for the chain stage. The README states a HyperSync API token has been required since November 2025 and that the free tier is sufficient, with registration at envio.dev. The CLOB and Gamma APIs are public and need no credentials.

### Is it safe to stop PolyLedger in the middle of a backfill?

The README states rows and the block cursor are committed in a single transaction, so an interrupted run resumes exactly where it stopped, and that it is safe to interrupt at any point. Re-running a range is a no-op because order_fills is keyed on (transaction_hash, log_index) and inserted with ON CONFLICT DO NOTHING.

### How long does the first PolyLedger backfill take?

The README says the first backfill is slow: anywhere from a few hours to a couple of days on the free HyperSync tier, depending on how much history you want. Subsequent runs are incremental and finish in seconds.

## Sources

- [Issues](https://github.com/nahrek/polyledger/issues)
- [License: MIT](https://github.com/nahrek/polyledger/blob/main/LICENSE)
- [nahrek/polyledger on GitHub](https://github.com/nahrek/polyledger)
- [README](https://github.com/nahrek/polyledger/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/nahrek-polyledger