poly_data rebuilds Polymarket trades from CTF Exchange V2 logs on Polygon

Polymarket switched contracts on 2026-04-28 and the old pipeline was deleted

Fetching prediction market fills used to mean polling a Goldsky subgraph over GraphQL, and that approach no longer works. Polymarket moved to a new set of CTF Exchange contracts on 2026-04-28 and stopped supporting its old subgraph indexer, so the previous pipeline in this repository was removed because it no longer returns complete data. The old code is not gone: it sits at the `v1-final` tag for anyone who still needs historical analysis on it.

The stated reason for not reviving it is cost and shape rather than correctness. Goldsky now serves data only through a turbo pipeline described as expensive and complex, so v2 takes the opposite route and reads on-chain events directly. That single switch is the design of the current version: one contract address, one log stream, no indexing service sitting in the middle.

What comes out the other end is plain files. Order events arrive from the chain, market metadata arrives from the Polymarket CLOB API, and the join between the two produces labeled trades carrying price, USD amount and BUY or SELL direction. Structured rows a notebook can load are the goal, not a query endpoint.

Three stages, one HyperSync connection, CSV files as the only handoff

`update.py` is a thin shim that calls `update_utils.pipeline.main`, and the packaged console script `poly-data` points at the same function, so `uv run poly-data` and `uv run python update.py` do identical work.

Stage one is Markets: pull every market from the Polymarket CLOB API endpoint `/markets`, 1000 per page, concurrently, into `data/markets.csv`. It runs to completion before anything else starts, so the full list already exists by the time trades are touched. Stage two is Chain: stream `OrderFilled` events from the CTF Exchange V2 contract `0xE111180000d2663C0091e4f400237545B87B996B` on Polygon through Envio HyperSync, resumable from the last scanned block. Stage three is Process: join those order events with market metadata and write labeled trades to `processed/trades.csv`.

The ordering is fixed, markets then chain then process, and the reason is that every scraped trade needs a market row to join against. Each stage also runs on its own:

uv run python -m update_utils.update_markets
uv run python -m update_utils.update_chain
uv run python -m update_utils.process_live

Decoding happens in-process rather than inside a hosted indexer, which is why the dependency list carries `eth-abi`, `eth-utils` and `eth-hash[pycryptodome]` next to `pandas` and `polars`. HyperSync streams logs with block timestamps inline across the full chain over a single connection, so ordinary RPC throttling never enters the picture.

Two environment variables are the entire configuration surface

There is no config file, no YAML block and no flag parsing anywhere in this project. Every knob is an environment variable, and there are exactly two.

`HYPERSYNC_API` is the Envio HyperSync bearer token and it has no default; a run without it stops immediately with a message telling you to set `HYPERSYNC_API`. `PROCESS_CHUNK_SIZE` defaults to 0, and any value above zero makes the process step read `data/orderFilled.csv` in chunks rather than loading the whole file, with `500000` given as the value to reach for when processing runs out of memory.

The documented install path is UV, and the installer ships separate commands per platform:

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

# Or with pip
pip install uv

Sync the environment afterwards, which reads the committed lockfile:

uv sync

`python-dotenv` loads a `.env` file from the project root at startup, so the token can live in a file rather than in your shell history:

HYPERSYNC_API="your-token-here"

After that, the first run is one command:

uv run poly-data        # or: uv run python update.py

Budget minutes rather than seconds for the first pass. The full market list from CLOB takes a couple of minutes, and the initial chain backfill from v2 genesis around April 2026 only begins once that list exists.

A token scoped to HyperRPC authenticates, then returns 403 on the data query

HyperSync has required a bearer token since 2025-11-03, and the free tier is enough for this pipeline. The trap is token scope. A token created for HyperRPC or HyperIndex authenticates successfully and then fails on data queries with `403 "Your token does not have access to this product"`, so a successful login is no evidence the token is the right one. Tokens carrying HyperSync product access are the ones that work, generated at envio.dev/app/api-tokens.

Speed is the second surprise, and here the project's own numbers pull in opposite directions. One section states that a single request scans hundreds of thousands of blocks and that the full backfill is only a handful of requests; another states that the free tier is rate-limited enough for the first full backfill to take several hours to a few days. Both can hold at once, since a handful of very large throttled responses is a slow day rather than a slow query.

The escape hatch is a paid Envio token, described as much faster with no throttling. Swapping it in needs no code change: set `HYPERSYNC_API` to the paid token and rerun. Because the backfill is resumable, letting the free tier run unattended is a workable substitute for paying, at the cost of wall clock time.

Resume state is two JSON cursors, and only later runs are quick

Two files under `data/` carry the state that makes an interrupted run restartable. `markets_state.json` holds the CLOB pagination cursor so the market stage can resume mid-listing rather than refetching from the first page, and `cursor_state.json` records the last scanned block so the chain stage picks up where the previous process stopped. Both sit inside a `data/` directory that is gitignored along with everything else the pipeline generates.

The lifecycle described for ordinary use is straightforward. The first run fetches the full market list in a couple of minutes, then backfills the chain from v2 genesis around April 2026. Every later run pulls deltas only and finishes in seconds. Stopping with Ctrl-C at any point is safe, because rerunning continues from the cursors instead of starting over.

That design carries a cost worth naming. History is held as CSV rather than in a database, so the same order events are re-joined from disk on every process stage run, and nothing prunes them. `markets.csv`, `orderFilled.csv` and `trades.csv` grow for as long as you keep appending, and nothing in the repository describes a compaction step, a retention policy or an index over any of the three.

No block range flag, whole-history default, and a memory ceiling

The one thing you cannot ask this pipeline for is a window of time. There is no start flag, no end flag, and no range variable in the environment table; the chain stage resumes from `cursor_state.json` and scans forward from wherever it stopped. If you only want last month's fills, you build the entire market list, run the chain stage across everything, and then filter afterwards yourself.

Memory is the second sharp edge, and the documentation concedes it. `PROCESS_CHUNK_SIZE` exists because processing the raw order event file can run out of memory, and the suggested value `500000` is a row count per chunk. The default is `0`, which means no chunking at all, so a fresh install on a small machine reaches the wall before anyone reads the note about it. Nothing states how much RAM the full file needs at that setting.

Reach for it when the whole history as files is the goal. Pass on it when you need one day rather than everything, when a hosted indexer already answers the question, or when your notebook expects a query engine instead of a directory of CSVs you have to load yourself.

HyperSync streaming against the v1-final subgraph, and what each one costs

Two real approaches to pulling this data out of Polymarket exist, and this repository has shipped both. The v1 code, kept at the `v1-final` tag, queried a Goldsky subgraph with GraphQL and polled for new activity, and its failure mode is the one that ended it: incomplete results once the old subgraph indexer lost support on 2026-04-28. The v2 code reads raw `OrderFilled` logs off Polygon and decodes them locally, so completeness no longer depends on somebody else indexing the chain on your behalf.

The cost profile inverts with the approach. GraphQL polling is cheap to start and stops being trustworthy. Direct log streaming over HyperSync returns complete history but hands you a rate-limited transport, which is why the free tier stretches a first backfill across several hours to a few days, and why a paid token is the documented fix that requires no code change.

Envio names HyperIndex and HyperRPC alongside HyperSync, and the setup steps here mention those tokens only to warn that they lack the HyperSync product grant. A hosted indexer would sit between you and the contract and answer queries rather than emit CSVs, which is a different shape of tool for a different job, and nothing in this repository compares the two.

The badges, the package manifest and the migration note disagree three times

Reading the metadata against the code surfaces three mismatches a skim would miss. The README badge advertises Python 3.10 and up, while `requires-python = ">=3.9"` in `pyproject.toml` sets the real floor one step lower. The title calls the project v2, which describes the Polymarket contract generation rather than the package, and the package version in the manifest is `0.1.0`.

The last push was on 2026-09-08 and the repository publishes no GitHub releases, so there is nothing versioned to pin, and upgrades arrive by moving the branch rather than by moving a pinned reference.

Repository contents are looser than the documented layout as well. Top level holds `.DS_Store`, `Explore and Backtest.ipynb`, `backtrader_plotting/`, `uv.lock` and `xmayeth.gif`, and neither the notebook nor the `backtrader_plotting/` directory appears in the structure listing or receives a description anywhere. `tests/` is called pure offline pytest, with `testpaths = ["tests"]` in the manifest and `pytest>=8.0.0` as the dev dependency group, so the suite touches neither the CLOB API nor HyperSync and will not warn you when either changes.

Editorial conclusion

Adopt poly_data when you want the whole Polymarket fill history as CSV files you own, and pass on it when one recent day, a hosted indexer or a query engine answers your question faster. Before vendoring any of it, settle the GPL-3.0 badge against the MIT string in pyproject.toml, confirm your machine survives the default PROCESS_CHUNK_SIZE of 0 over a raw orderFilled.csv, and budget a free tier backfill in hours to days rather than minutes. Start with a paid Envio token and PROCESS_CHUNK_SIZE=500000 before committing a whole night to the free tier.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. README
4. warproxxx/poly_data on GitHub