All the Places: thousands of location spiders feeding one point of interest dataset

One spider per site, one standard output format

All the Places is a project to generate point of interest data sourced from websites with store location pages. The mechanism is Scrapy, the Python web scraping framework, with one spider per site retrieving the point of interest data, and the results published in a standard format described in DATA_FORMAT.md.

The output side is what makes the project worth running at scale. The README describes the data being used in many places, including some of the most popular maps, and argues that having high quality, fresh, first party data helps brands be listed accurately across applications. The flow diagram in the README makes the downstream picture explicit: thousands of first party datasets feed All the Places, and All the Places feeds OpenStreetMap, TomTom and Overture, with Overture in turn feeding Foursquare OS Places.

That chain is the real product. No single spider is valuable, but the aggregate is a large corpus of locations that came from the places themselves rather than from a crowd of users, and the value of the corpus is what justifies the effort of writing thousands of small spiders.

The repository layout reflects that the spiders are the bulk of the work. There is a locations directory holding the spiders themselves, a templates directory for spider scaffolding, a tests directory, a ci directory with the run scripts, a contrib directory, and a docs directory carrying the guides. The top-level DATA_FORMAT.md, API.md and pyproject.toml describe the contract every spider has to satisfy.

Setting up with uv, Docker or Codespaces

The project uses uv for dependency management and pins Python tightly, with requires-python set to ">=3.11,<3.12". The Ubuntu instructions, tested with Ubuntu 24.04 LTS, start with installing uv:

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

Then the clone and the dependency install:

git clone [email protected]:alltheplaces/alltheplaces.git

cd alltheplaces
uv sync

And the installation check is running scrapy itself:

uv run scrapy

If that runs without complaint, the documentation says you have a functional installation and are ready to run and write spiders. The macOS path is the same except that uv comes from Homebrew with brew install uv, and the Windows note points at the Scrapy documentation for the extra steps that platform needs.

Two other routes are offered. GitHub Codespaces gives a cloud development environment created from the repository with everything configured. And Docker builds a self-contained image:

cd alltheplaces
docker build -t alltheplaces .

docker run --rm -it alltheplaces

The Dockerfile shows what the container carries: Python 3.11 on bookworm, uv copied in from a pinned uv image, tippecanoe 2.78.0 compiled from source, the uv dependencies installed with a frozen lockfile, Playwright's system dependencies plus the Firefox browser, and a camoufox browser fetched at build time. The container's default command runs the whole spider set through ci/run_all_spiders.sh.

A dependency list that explains what the spiders actually do

The pyproject.toml dependency list reads like a map of the problems a location scraper runs into, and it is worth reading as a set of requirements rather than as a list.

For extracting data from pages, there is scrapy itself at 2.17.0 or newer, scrapy-playwright, scrapy-camoufox, scrapy-zyte-api, chompjs for parsing JavaScript structures, json5, ijson for streaming JSON, xmltodict, and phpserialize for sites that hand out PHP serialised payloads. There is also playwright_captcha and brotli for compressed responses.

For turning extracted values into geography, there is geopandas, shapely, pyproj, pygeohash, reverse-geocoder and geonamescache. For addresses there is phonenumbers and pycountry, and unidecode for transliteration. For files and tabular data there is openpyxl, pdfplumber for reading PDFs, and duckdb with pyarrow for working over the results. boto3 and requests-cache sit there for storage and for not re-fetching the same page during a run.

One override in the config file is worth noting because it is the kind of detail that would otherwise cost an afternoon. The comment explains that scrapy-camoufox pins camoufox below 0.5.0, and that camoufox 0.4.x installs whatever browser build is newest, which breaks on a specific browser version, so the project overrides the dependency to camoufox 0.5.7 or newer, where the fetch command installs the browser build paired with the library version.

The dev group adds the usual tooling, with black configured for a line length of 120 targeting Python 3.11, flake8, isort, pylint, pytest, autoflake8, pep8-naming, ty, pre-commit and awscli.

Writing a spider without writing code

The contribution guides are unusually practical, and the most interesting one is about structured data. Many of the sites provide their data in a standard format, and the guide on structured data explains that data from many point of interest pages can be extracted without writing code at all. That is a large share of the easy wins: a chain or a supermarket publishes location data as JSON-LD or similar, and a spider becomes a declaration rather than a parser.

The other guides cover the parts that are not mechanical. There is a guide to what a spider should be called, and a guide to using Wikidata and the Name Suggestion Index, which is how a new place gets matched to an existing identity rather than added as a duplicate. There is a guide on sitemaps, described as making it easier to find point of interest pages in the first place. There is a guide for sites that export their data through simple APIs rather than through HTML pages.

Then there are the process guides: what is expected in a pull request, and what the project does behind the scenes in its pipelines. That last one is the one to read before assuming how a spider runs, since the weekly run is orchestrated by machinery the contributor never sees.

The project also says who the contributors are. Spiders are typically written by map makers who want the data, though brands are welcome to be involved, and there is a guide for brands on that point. Communication happens through tickets on the GitHub issue tracker, and many contributors are also present on the OSM US Slack workspace, in an alltheplaces channel.

The weekly run and the policy of restraint

The output from running the project is published on a regular cadence to alltheplaces.xyz, and the documentation includes an explicit warning against running every spider locally to pick up the output. The reasoning given is worth quoting in spirit: the less the project bothers a website, the more the project will be tolerated.

That is a rate limiting policy stated as a community principle, and it shapes what the project is for. A project that scraped every site as fast as possible would be brittle, since most sites would block it and the data would go stale faster than the publication cycle anyway. Running on a weekly cadence with the load spread out is what makes the data fresh enough to be useful and light enough to be tolerated.

The licence structure follows the same separation of concerns. The data generated by the spiders is provided on the website and released under the Creative Commons CC-0 waiver, meaning it can be used without attribution requirements. The spider software that produces the data, which is the repository itself, is licensed under the MIT licence. The licence field on the repository is classified as a custom licence that GitHub cannot classify, with a LICENSE file in the tree, so the two documents should be read separately rather than assumed.

The repository is not archived, and the last push was on 2026-10-22, so the spiders are actively maintained.

Where All the Places is the wrong choice

The first limitation is cadence. The data is published on a regular schedule, so this is not a source for anything that needs current information. A store that opened yesterday or closed last week will not be reflected until the next publication, and any consumer needing real-time presence data has to look elsewhere.

The second is that a spider is a liability that ages badly. Every site it depends on can change its markup, its structured data or its rate limiting, and the fix belongs to whoever wrote that spider. A project with thousands of spiders therefore accumulates thousands of things that can break, and the weekly run exists precisely to detect them. For an organisation that needs one specific dataset from one specific chain, running a single spider on a schedule you control is a smaller commitment than joining a shared project.

The third is the output format's rigidity. Every spider has to produce the same standard format so the corpus can be merged, which is right for a shared dataset and wrong for a site with unusual data. The guides do accommodate different sources, from structured data to simple APIs, but the merge target is fixed.

The alternative to weigh is a paid place data API, which offers contractual reliability, freshness and support in exchange for money and for accepting the provider's coverage. The comparison that actually matters depends on the question: All the Places gives you breadth that no single vendor matches, including the long tail of independent businesses, and gives it away under CC-0. A vendor gives you a guarantee. If the coverage is the hard part of your problem, this project is the answer; if the guarantee is, it is not.

Editorial conclusion

All the Places fits a map data consumer that needs place data the major APIs do not cover, and a contributor who wants to write one site spider inside a project with a fixed output format, weekly publication and a policy of restraint. It does not fit a team that needs real-time data, since the output is published on a regular cadence rather than continuously, and it does not fit a project that must scrape aggressively. Before adding a spider, read the structured data and pipeline documentation, follow the naming guide, and plan around the stated policy of not running every spider locally, since the project treats the load it places on other websites as the constraint that keeps it tolerated.

Frequently asked questions

Official sources

1. alltheplaces/alltheplaces on GitHub
2. Issues
3. Project website
4. README