opensdmx

Fetching official statistics the safe way

opensdmx is a Python CLI and library for any SDMX 2.1 REST API, with Eurostat as the default provider and built in support for ISTAT, OECD, ECB, World Bank, and others. The README makes a pointed argument about using AI with statistics: large language models are good at understanding questions but they fabricate numerical data, and the project cites an IMF paper estimating GenAI returns inaccurate statistics up to two thirds of the time. The correct pattern, the docs say, is to use AI to generate structured API queries, not to generate the numbers themselves. opensdmx is the execution layer for that pattern: the AI decides what to fetch, and opensdmx fetches the exact published figure. The README says the tool works fine on its own but shines when driven by an AI agent, and it ships a sdmx-explorer Agent Skill for guided dataset discovery, schema exploration, filter selection, and data retrieval. The design philosophy is that the CLI is meant to be composed, queried, and orchestrated step by step. This positioning is unusual for a statistics library because it treats the command line as the natural interface for an agent rather than only for a human analyst. The note that the tool is experimental and asking for issue reports sets an honest expectation about maturity.

CLI workflow

The command line quick start shows the shape of everyday use. You search with a keyword such as unemployment, inspect a dataset with info, list the values a dimension actually contains with constraints, and then get the data with filters and an output file. The README stresses a golden rule: check constraints, not the dataset title, because a dataflow's dimensions change over time and only codes returned by constraints are safe to use as filters. Codes from the general SDMX codelist may not exist in a specific dataset and will return a 400 error. The examples walk through youth unemployment for Italy and Spain with readable labels, GDP per capita for EU countries saved as a reusable query file, and inflation plotted as a chart. Three time saving rules are given: include single value dimensions explicitly, turn on labels early, and save every query you will rerun. The labels flag appends human readable names next to codes so the output is ready for a report. A query file captures the full query, including provider, dataset, filters with descriptions, and time range, and you rerun it with opensdmx run. The CLI also has commands for values, tree to browse the thematic category structure, siblings to find related dataflows, plot to render a chart, and blacklist to manage unavailable datasets. Providers switch with a flag or an environment variable.

Semantic search

Beyond keyword search, opensdmx offers a semantic mode that uses embeddings through a local Ollama server. The README contrasts the two: keyword search ranks with BM25 over title, ID, and category name, while semantic search uses embedding similarity and helps when you do not know the exact wording. Setup requires Ollama with the nomic-embed-text-v2-moe model and one index build per provider, stored locally as an embeddings parquet file; there is no incremental update, so rebuilding scans the whole catalog. For providers that expose a thematic catalog, running tree once before embed enriches each embedding with category names, which improves recall on short or generic queries. The docs report a measured result that semantic retrieval finds the right dataflow in the top ten for 57 percent of queries against 17 percent for the keyword path, and that on English queries against Italian metadata keyword search finds essentially nothing. A worked example shows a colloquial phrase with zero word overlap against the catalog returning no keyword results but finding the right dataset semantically. The guidance is to use multi word descriptive queries and that English queries on non English catalogs work well because the model is multilingual. This mode is optional and needs the extra Ollama dependency, but it addresses the real problem of finding data when you only know the concept.

Python API

The library API mirrors the CLI for use inside Python projects. You call all_available to list datasets as a Polars DataFrame, search_dataset by keyword, load_dataset to build an object, set_filters to narrow dimensions, and get_data or the one liner fetch to retrieve results. The providers function switches the active backend among eurostat, istat, oecd, ecb, and worldbank, or to a custom URL with an optional agency id and rate limit. The README notes that Eurostat returns the compact SDMX-CSV format while other providers return a more verbose text/csv with extra metadata columns, which is expected and handled by filtering. A worked Python example loads the unemployment dataset, sets filters, gets sixty observations, casts the value column, and plots with plotnine, saving a PNG. The function table documents set_provider, get_provider, dimensions_info, get_dimension_values, get_codelist_hierarchy, get_available_values, run_query for saved YAML, semantic_search, build_embeddings, set_timeout, and parse_time_period. Start and end periods accept formats like a year, a quarter, or a month, and you can request the last or first N observations. The examples include plotting straight from a CSV. Because the data comes back as Polars frames, it drops straight into the usual Python analysis tooling without conversion steps. The source lives in the ondata-opensdmx repository on GitHub and is implemented in Python, with the license recorded as MIT in the project metadata.

Editorial conclusion

The library is written in Python, released under the MIT license, and installs as a CLI with uv or as a dependency with pip.