the S&P 500 companies dataset and the problem of a scraped index list

A constituents list, not a financials dataset

The repository holds a list of the companies in the S&P 500 together with associated information, and it is hosted as a core dataset on datahub.io. The index itself is described as a free-float, capitalization-weighted index of the top 500 publicly listed stocks in the US, by market capitalisation.

The scope is narrower than the name suggests, and the repository is upfront about it. The dataset includes the list of all the stocks contained in the index. The financial figures are elsewhere: the README points to a separate Standard and Poor's 500 dataset on GitHub for aggregate information on the S&P, naming dividends and earnings as examples.

That split matters for anyone planning an analysis. This repository gives you entities, the metadata that identifies them, and a record of index membership. It does not give you price series, valuations or fundamentals, and joining against the other dataset is a step the documentation expects you to take yourself.

The repository layout is what a data project should look like. There is a data directory holding the output, a scripts directory holding the scraper and its requirements, a Makefile, a datapackage.json describing the dataset, a devcontainer configuration, and a file called UPDATE_SCRIPT_MAINTENANCE_REPORT.md whose name says more about the state of the pipeline than any README paragraph would.

The last push was on 2026-10-22, the default branch is main, and the repository has no GitHub releases. The primary language is Python because the pipeline is Python.

The upstream is a wiki, because the official list stopped being free

The reason this dataset exists at all is a licensing change upstream. Information on the S&P 500 index used to be available on the official Standard and Poor's webpage, but until they publish it back, Wikipedia's list of S&P 500 companies is described as the best up-to-date and open data source.

The Sources section tells the fuller story. Detailed information on the index, primarily in XLS format, used to be obtained from the official webpage, and it was free, but registration was required. So the data did not disappear because of a technical reason. It became gated behind a registration wall, and the community response was to scrape the wiki table instead.

That history determines the character of the data. The output inherits the shape of a Wikipedia table, which means the fields are whatever the wiki page happens to expose, and the provenance is one community-maintained page rather than the index provider. For a teaching dataset, a fixture in a test suite, or a join key against another dataset, that is a reasonable trade. For anything that must be defensible to an auditor, it is a weak foundation, and the README does not pretend otherwise.

The metadata edge case is documented, which is a good sign. The Founded field contains mixed-format values for some companies, with an example given as 2013 (1888), where the first year is the current legal entity's founding date and the parenthetical year is the predecessor organisation's founding date. A field with two meanings in one string is a parsing problem for the consumer, and this one is at least described rather than left to be discovered.

Running the scraper, or letting the Makefile do it

The pipeline relies on Python, so the development section starts with installing Python and then creating an isolated environment. The four documented steps are to create a virtual environment with the venv module, activate it, install the dependencies from the requirements file, and run the scrape script.

The Makefile is the tidier route, and it is short enough to read in full:

VENV = .env
PYTHON = $(VENV)/bin/python3
PIP = $(VENV)/bin/pip

.PHONY: data clean

all: data

data: $(VENV)/bin/activate
	$(PYTHON) scripts/scrape.py

The structure is worth noting because it is a common and sound pattern. The data target depends on the virtual environment marker, and that marker depends on the requirements file, so the environment is created only when it does not exist and only reinstalls when the dependencies change. The pip and python paths are derived from the environment rather than taken from whatever happens to be on your PATH, which removes a class of version mismatch bugs.

There is also a validate target that runs the frictionless validator against data/constituents.csv, and a clean target that removes the pycache directory and the virtual environment. The frictionless check is the one a user of the data should care about, because it is the mechanism that tells you the CSV on disk still conforms to the schema described in datapackage.json.

The alternative documented path is to run the steps by hand, which the README spells out one at a time, and the Makefile route is described as creating the environment, installing the dependencies and running the script for you.

The maintenance report tells you what a dataset's real state is

UPDATE_SCRIPT_MAINTENANCE_REPORT.md is an unusual file to find in a data repository, and it is arguably the most valuable thing here.

A scraped dataset has a specific failure mode. The script keeps working, the workflow stays green, and the file on disk quietly stops matching reality. Nothing breaks, because nothing is failing. The only symptom is that a company that left the index is still listed, or a company that joined is missing, and the person who notices is usually the person whose analysis looks wrong.

A report that tracks the state of the update script is the acknowledgement of that failure mode. The file name alone does not tell you its contents, but its presence tells you the maintainers know the scraper is the fragile part of the project rather than the data.

The devcontainer configuration points the same way. A developer container pins the toolchain, so the person regenerating the file runs the same Python, the same dependency versions and the same environment as the person who wrote the scraper. For a pipeline whose output other people depend on, that reproducibility is the difference between a dataset that can be regenerated and one that can only be copied.

The repository also keeps a.github directory, which is where the periodic update runs. Since the default branch is main and the last push was on 2026-10-22, the file on main reflects a recent regeneration rather than a snapshot from years ago.

The financial notes point at the SEC, not at this repository

The General Financial Notes section does something slightly surprising: it tells you that the repository does not contain the financial data, and explains where that data actually lives.

Publicly listed US companies are obliged to file various reports on a regular basis with the SEC, and the section names two of them as being of especial interest to investors and others interested in their finances and business. The 10-K is the annual report, and the 10-Q is the quarterly report.

So the repository is pointing you at the primary source for company financials rather than trying to be a mirror of it. For a dataset that describes itself as a list of companies together with associated financials, that is worth noticing: the index membership and the company identifiers are here, the filings are at the SEC, and the aggregate index data is in the sibling repository.

This is a reasonable division of labour. Financial statement data changes quarterly, is filed in a standard format, and is available from a primary source with a stable interface. A scraped Wikipedia table will never be that. Trying to maintain earnings and dividend figures in a community-scraped file would produce numbers that look authoritative and are not.

The practical guidance for a user is then simple. Take entity names, identifiers and index membership from this dataset, take filings from the SEC, and take index-level aggregates from the separate Standard and Poor's 500 dataset. Anything that mixes them into one table needs a clear note about which source each column came from.

Licensing is the one place to read the fine print

The licence is split by content type, and the split is explicit. All data is licensed under the Open Data Commons Public Domain Dedication and License, and all code is licensed under the MIT/BSD licence.

The project metadata records the overall licence as unknown, which is a consequence of that split: a single identifier cannot describe a repository where the data and the code carry different terms. If your tooling looks for a licence string, it will not find a satisfying answer here.

The PDDL is a dedication rather than a permissive licence in the usual sense, so before you use the data commercially, put the actual text in front of whoever handles licensing on your side. The README also asks for something that is not legally required: while no credit is formally required, a link back or credit to Rufus Pollock and the Open Knowledge Foundation is much appreciated. That request is the human context of the project, and honouring it is cheap.

The MIT/BSD terms on the code side are the familiar permissive ones, so the scraper can be reused, modified and relicensed in a derived pipeline without much friction.

One point of tension is worth naming. The upstream source is Wikipedia, whose own terms govern reuse of the text, and the pipeline scrapes a table from it. Whether that creates any additional obligation on the redistributor is a question for your own legal review rather than something the repository answers. In practice most consumers of a company name and an index membership are using facts rather than protected expression, but that is your call to make with your own counsel, not ours.

Where this dataset is the wrong tool

The first wrong case is anyone who needs financials. Despite the repository description, there is no revenue, no earnings, no dividend history and no price data in it. If your notebook needs a market cap or a P/E ratio, this is the wrong file and the sibling repository is where the aggregate index data lives.

The second wrong case is anything that needs a current, authoritative index list. A scrape of a wiki table is a snapshot of what that page said when the script last ran. Companies enter and leave the index through corporate events, and a wiki table can lag. The README itself frames the wiki as the best available open source rather than as the authority, and it says the official information will presumably be published back at some point. When it is, the pipeline's premise changes, and a consumer who built a dependency on the wiki-sourced snapshot will need to revisit it.

The third wrong case is a task where the identity of the entity matters more than the list does. The dataset is a membership list, and its records carry the fields the wiki exposes. Anything that needs a canonical identifier for a company, or a legal entity that survives a merger, is asking for something this repository does not promise. The Founded field is the clearest example of the limit, since it deliberately holds two dates in one string when the legal entity was created by a predecessor.

The alternative worth naming is a commercial data provider or the exchange's own reference data, and the difference in approach is exactly the trade this project makes. A provider gives you a maintained, licensed, timestamped feed with a support contract and a schema you can rely on, at a price and with terms you must accept. This dataset gives you a free, transparent, reproducible scrape under an open licence, with the accuracy and freshness resting on Wikipedia and on how often the maintainers rerun the script. Which one is right depends on whether your use can tolerate a snapshot that is only as good as its most recent successful run, and for a tutorial, a fixture, or a prototype the answer is usually yes.

Editorial conclusion

The dataset fits a project that needs a clean list of index constituents with company metadata, and that can accept Wikipedia as the upstream source. It does not fit anything that needs financial figures or a legally authoritative index membership list, since the data here is constituent metadata scraped from a wiki and the aggregate numbers live in a separate repository. Before you rely on it, read the Founded field and handle the mixed format values yourself, run the frictionless validation target on the CSV, and check the UPDATE_SCRIPT_MAINTENANCE_REPORT.md for the state of the last regeneration.

Frequently asked questions

Official sources

1. datasets/s-and-p-500-companies on GitHub
2. Issues
3. Project website
4. README