# Cloudflare Datamining: Tracking a Vendor's Public Surface, Commit by Commit > This repository scrapes Cloudflare's dashboards, APIs, NPM packages, and marketing site to record changes over time. It is a monitoring tool for researchers and developers who want early signals about undocumented endpoints, new products, or pricing shifts. **Cloudflare-Mining/Cloudflare-Datamining** — Public datamining for all things Cloudflare - Repository: https://github.com/Cloudflare-Mining/Cloudflare-Datamining - Website: https://cfdata.lol - Stars: 199 · Forks: 26 - Language: HTML - License: not declared - Published: 2026-08-22 · Updated: 2026-08-22 · Language: en - Canonical page: https://hysenlabs.com/projects/cloudflare-mining-cloudflare-datamining ## What This Repository Tracks and Who Needs It The repository solves a narrow but real problem: Cloudflare changes its public-facing systems without announcements. A developer relying on an undocumented dashboard subroute, a researcher watching for new product features, or a security analyst monitoring DNS-over-HTTPS schemas all face the same gap. The README says the project mines 'a lot of publicly available data' and makes automated commits to show how data changes over time. That is the core value. It is not a tool for end users. It is a surveillance layer for engineers who build on Cloudflare or study its behavior. The intended audience is clear from the structure: people who want diffs of API schemas, translation strings, entitlements, and package updates. ## The Data Pipeline: Scripts, Data, and Automated Commits The mechanism is straightforward. Scripts live in `scripts`, and output lands in `data`. Each subdirectory corresponds to a different source. The `api-schemas` folder extracts schemas from `api.cloudflare.com` and tracks changes. The `dashboard-subroutes` folder pulls subroutes from the customer dashboard, which the README says is useful for finding undocumented APIs. The `dashboard-translations` folder captures translation strings, which can reveal new features before they are documented. The `cdn-cgi` folder tracks `request.cf` keys and component changes across datacenters. The `doh` folder records the schema returned by 1.1.1.1. The README does not explain how often the scripts run, only that commits happen frequently. That is a gap. Without a schedule, you cannot know how stale the data is on any given day. ## How to Run It: Commands and Configuration The README does not include installation instructions or a list of dependencies. It only states the directory layout. To run the scripts, you would clone the repository and inspect the `scripts` folder for executable files. The README gives no command examples, no `npm install` step, and no environment variables. The contributing section asks for a JSON dump of entitlements, sent via Discord or Twitter, which suggests some scripts require an authenticated Cloudflare account. The `entitlements` data is not automated, per the README. That means the repository is not self-contained. You may need your own account credentials to reproduce the `account` or `entitlements` data. The lack of setup documentation is a real obstacle. A new user must read the script source to understand what is needed. ## What the Data Reveals: From Gates to Job Postings The repository covers more than API schemas. The `gates` folder tracks active A/B experiment gates on the dashboard, which can hint at unreleased features. The `jobs` folder lists Cloudflare job postings, which the README says can reveal upcoming projects. The `registrar` folder tracks TLDs and pricing, useful for cost planning. The `packages` and `github-repos` folders monitor the Cloudflare GitHub organization for new repositories and package releases. The `workerd` folder tracks capnp schemas, relevant to anyone working with Cloudflare's runtime. The `marketing` folder extracts raw JSON from cloudflare.com. Together, these sources form a broader picture than a single API endpoint. But the README is honest about limits: the `domains` data is 'not perfect'. That caveat applies to other datasets too, since scraping public pages can miss rate-limited or dynamic content. ## Limitations: Fragility, Gaps, and Unverified Claims The biggest limitation is that nothing here is guaranteed to work tomorrow. Cloudflare can change a dashboard route, alter a translation key, or block a scraper. The README gives no maintenance history or test suite. The repository has no release tags and no license listed, which complicates reuse. The primary language is HTML, which suggests the bulk of the content is data files, not code. The `entitlements` dataset is manually maintained and depends on community contributions, so it will always be incomplete. The `doh` schema and `cdn-cgi` keys may change without notice, and the scripts may break silently. The README does not mention error handling or fallback mechanisms. For a production deployment, you would need to add your own monitoring of the scripts themselves. This is a tool for observation, not a reliable data source. ## Alternatives: Official APIs and Vendor Monitoring Services The obvious alternative is Cloudflare's own API and documentation. The official API at `api.cloudflare.com` provides schemas and changelogs, but only for documented endpoints. The repository exists because official sources do not cover everything, especially dashboard internals and translation strings. Another alternative is a general web monitoring service like Changedetection.io or a custom cron job that fetches specific endpoints. Those tools give you control over frequency and storage, but you must build the selectors yourself and maintain them. The difference is in scope: this repository aggregates dozens of sources into one place, while a self-hosted monitor covers only what you configure. The trade-off is that the repository's scripts may be less transparent about their exact logic, since the README does not explain each script's implementation. ## Maintenance and Upgrade Cost: What to Expect The repository is archived: no. But the last push is unknown, and there are no recent releases. That is a warning sign. The README says automated commits will be made frequently, but without a visible commit history, you cannot confirm that. The license is unknown, which is a legal risk. You cannot assume you can use the code or data without permission. The maintenance cost is high because the project depends on external services that change unpredictably. Each Cloudflare dashboard update may break a script. The README asks for community help with entitlements, which shows the project relies on external contributions to stay current. If you adopt this, budget time to read the scripts, fix selectors, and handle API changes. The lack of releases means no versioned snapshots. You are always on the latest, possibly broken, state of the main branch. ## Who Should Adopt It and What to Check First Adopt this if you are a developer who uses undocumented Cloudflare endpoints, a researcher tracking feature rollouts, or a security analyst monitoring schema changes. Skip it if you need stable, documented data or if you cannot afford to maintain your own fork. Before relying on it, check the `scripts` folder for dependencies and read the script source to see what endpoints are called. Verify that the automated commits are actually recent by looking at the commit history on the main branch. Confirm that the data you care about, such as `dashboard-subroutes` or `doh`, is updated within a timeframe you can tolerate. The repository is a useful starting point, but it is not a finished product. Treat it as a reference implementation that you may need to adapt. ## Conclusion Adopt this repository if you work with Cloudflare's APIs, need early warnings about dashboard changes, or study the vendor's product direction. Skip it if you want official documentation or stable, versioned data. Verify first that the automated commits still run and that the scripts match the current Cloudflare endpoints, since the repository layout suggests frequent breakage. The project is useful as a signal source, not as a canonical reference. ## FAQ ### What does Cloudflare-Datamining actually collect? Twenty-four trackers covering the customer dashboard, the API schemas at api.cloudflare.com, the marketing site, the 1.1.1.1 DNS-over-HTTPS schema, the Coveo search indexes, Cloudflare Registrar TLDs and pricing, the `@cloudflare/types` and `@cloudflare/component-icon` npm packages, and the workerd capnp schemas. Output goes into the data directory and is committed on a schedule. ### Why would someone want to track undocumented Cloudflare APIs? The `dashboard-subroutes` tracker extracts subroutes from the dashboard, which the documentation says can be useful for determining undocumented APIs. Set against the `api-schemas` tracker, which only sees what api.cloudflare.com publishes, the difference is the surface Cloudflare has not announced. ### Is the Cloudflare-Datamining entitlements list complete? No, and the documentation says so. The `entitlements` tracker is explicitly not automated, because entitlements are per-account properties. The list is grown by contributors who send a JSON dump of their own account's entitlements, so it is a self-selected sample rather than a full inventory. ### How current is the data in Cloudflare-Datamining? Automated commits are made frequently, and each one shows how the data changed since the previous one, so the git history itself is the record. A Discord channel is offered for notifications of changes. The last push was on 2026-09-22. ### What licence applies to Cloudflare-Datamining? None is stated. There is no licence file at the top of the repository and no licence recorded, so the terms for reusing the extracted data are not given by the project. Check the repository root before shipping anything derived from it. ## Sources - [Official documentation](https://cfdata.lol) - [Official README](https://github.com/Cloudflare-Mining/Cloudflare-Datamining#readme) - [Project repository](https://github.com/Cloudflare-Mining/Cloudflare-Datamining) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/cloudflare-mining-cloudflare-datamining