# Magda: a federated data catalog where every record is a bag of JSON aspects

> Magda is a Kubernetes-deployed catalog that federates metadata from open data portals and internal sources into one searchable place. The design is deliberately unopinionated, with the Postgres-backed Registry storing records as dynamically declared JSON aspects, and the README is candid that the internal catalog use case still has rough edges.

**magda-io/magda** — A federated, open-source data catalog for all your big data and small data

- Repository: https://github.com/magda-io/magda
- Website: https://magda.io
- Stars: 607 · Forks: 105
- Language: JavaScript
- License: Apache-2.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/magda-io-magda

## Federation is the idea, not a feature bolted on later

Magda describes itself as a data catalog system that provides a single place where all of an organisation's data can be catalogued, enriched, searched, tracked and prioritized, whether big or small, internally or externally sourced, and available as files, databases or APIs. The system is designed specifically around the concept of federation, providing a single view across all data of interest to a user regardless of where the data is stored or where it was sourced from. The mechanism is that Magda can quickly crawl external data sources, track changes, make automatic enhancements, and raise notifications when changes occur. The audience is a data platform team or a national open data portal operator, and the first production user named in the README is data.gov.au, which has used Magda in production for over a year and is described as relatively mature for that use case.

## The Registry stores records as dynamically declared aspects

Everything in Magda revolves around the Registry, an unopinionated datastore built on top of Postgres. The Registry stores records as a set of JSON documents called aspects. A dataset is a record with a number of aspects, a basic one recording name and description, and more esoteric ones that might not be present for every dataset, such as temporal coverage or determined data quality. Distributions, the actual data files or URLs linking to them, are also modelled as records with their own aspects, including whether the URL to the file worked when last tested. The flexible part is that aspects can be declared dynamically by other services simply by making a call with a name, description and JSON schema. Because the system is not opinionated about what a record is beyond a set of aspects, the same mechanism adds new entity types that link together, and the README notes this has been used to store projects that link to a number of datasets.

## Connectors are jobs, minions are listeners

The two extension patterns explain most of the architecture. Connectors go out to external datasources and copy their metadata into the Registry so it can be searched and have other aspects attached. A connector is a Docker based microservice invoked as a Kubernetes job, so it scans the target datasource, usually an open data portal, then completes and shuts down. There are connectors for a number of existing open data formats, and you can write and run your own. Minions are the second pattern. A minion is a service that listens for new records or changes to existing records, performs some operation, and writes the result back to the registry. The broken link minion is the example given: it listens for changes to distributions, retrieves the URLs described, records whether they could be accessed, and writes that back in its own aspect. Several minions can write the same aspect, as with a quality aspect that holds ratings from different sources, which are averaged and used by search.

## Search runs on OpenSearch, and it replaced ElasticSearch in v4

Datasets and distributions in the Registry are ingested into an OpenSearch cluster, which indexes a few core aspects of each and exposes an API. The README notes that since v4.0.0 the internal ElasticSearch cluster was replaced with a latest OpenSearch cluster, and the repository has separate `magda-elastic-search` and `magda-opensearch` directories alongside the search API services, which tells you the migration left both names in the tree. On top of that sit the search features the README lists: powerful and scalable search based on OpenSearch, and quick and reliable aggregation of external sources of datasets. The federation claim is a search claim in practice. A user does not care that one record came from a connector and another from a minion; they care that one box finds both, and the OpenSearch index is where that convergence happens.

## Extensions are Docker images, which is the real extension story

Magda is built around a collection of microservices distributed as Docker containers, and the README gives the reason, easy extensibility. Magda can be customised by simply adding new services using any technology as Docker images and integrating them with the rest of the system via stable HTTP APIs. That is a real design choice with a real cost. Your extension can be written in Go, Python, Scala, or anything else that produces a container, and it plugs in through HTTP rather than through a plugin API you have to learn. But it also means every extension is a separately deployed service with its own health, scaling, and version story, and the HTTP boundary is where your latency and failure modes live. Orchestration uses Helm and Kubernetes, and the payoff the README names is that configuration of a customised instance can be stored and tracked as plain text, so instances with identical configuration can be reproduced quickly.

## Authentication is federated, and policy based authorization is still being built

Sign-in is handled with passport.js, and the README lists the providers: Google, Facebook, WSFed, AAF, and CKAN, with new providers easy to add. CKAN in that list is a good clue about the intended government data audience, since CKAN is what many public data portals are built on. Authorization is a different story. The README lists a robust, policy-based authorization system built on Open Policy Agent as currently under development, where you write flexible policies to restrict access to datasets and have them work across the system, including restricting search results to what a user is allowed to see. The policy code is visible in the repository layout, with a `magda-opa` service, a `deploy/helm/internal-charts/opa/policies` directory, and scripts to run and test the policies locally:

```json
    "start-opa": "opa run deploy/helm/internal-charts/opa/policies",
    "test-opa": "opa test deploy/helm/internal-charts/opa/policies -v",
```

So the OPA work is real and testable today, while the README is honest that the wider authorization story is unfinished. A heavily automated data cataloguing process intended to produce high quality metadata is also listed as under development, as is storage of datasets.

## The maintainers say plainly where the rough edges are

The Current Status section is unusual in its candour and worth reading before you deploy. Magda is under active development by a small team, and the team often has to prioritise between making the open source side steadier and adding features to their own deployments, which can mean newer features are not documented well or require specific configuration to work. The distinction between the two use cases is the point. As an open data search engine, with data.gov.au in production for over a year, the project is relatively mature. As a data catalogue for use within organisations, the focus for the past 18 months has been development, and the README says to use it if you want to but to expect some rough edges. The releases tell the same story from a different angle, with v7.0.0 and v7.0.1 published in October 2026, preceded by a v7.0.0 alpha, and the last push on 2026-10-20. The licence is Apache 2.0, which keeps commercial use straightforward, and a demo instance is published at demo.dev.magda.io for trying the interface before you install anything.

## Conclusion

Magda fits a government data portal or an organisation that wants one search box over many open data sources, and it fits teams already running Kubernetes who want their own extensions written as plain Docker images. It is a poor fit for an organisation that needs its internal catalog to be polished on day one, because the README says to expect rough edges in that role, and it is a poor fit if you cannot run a Kubernetes cluster. Before you commit, read the architecture guide and the registry docs, plan for the aspect model rather than a fixed schema, and check the roadmap for the features still marked as under development, including policy based authorization and dataset storage.

## FAQ

### What is the Registry in Magda and how does it store data?

The Registry is an unopinionated datastore on top of Postgres that stores records as a set of JSON documents called aspects. A dataset is a record with a basic aspect plus optional ones such as temporal coverage or data quality, and aspects can be declared dynamically by other services with a name, description and JSON schema.

### What is the difference between a connector and a minion in Magda?

A connector copies metadata from an external datasource into the Registry and runs as a Kubernetes job that scans the target and then shuts down. A minion is a long running service that listens for new records or changes, performs an operation such as checking whether a distribution URL still works, and writes the result back into the registry as its own aspect.

### How do I extend Magda with my own code?

Extensions are added as new Docker images to the cluster, and they can be written in any language, integrating with the rest of the system through stable HTTP APIs. Orchestration is handled with Helm and Kubernetes so that configuration stays as plain text and identical instances can be reproduced.

### Which search engine does Magda use?

Magda indexes datasets and distributions from the Registry into an OpenSearch cluster and exposes a search API. The README notes that since v4.0.0 the internal ElasticSearch cluster was replaced with a latest OpenSearch cluster.

### How mature is Magda for internal organisational use?

The README says Magda has been used in production for over a year by data.gov.au as an open data search engine and is relatively mature for that case. For use as an internal data catalogue, the focus has been development for the past 18 months and the README advises expecting some rough edges.

### Which authentication providers does Magda support?

Authentication is federated through passport.js, with Google, Facebook, WSFed, AAF, and CKAN listed in the README, and new providers described as easy to add. A policy-based authorization system built on Open Policy Agent is listed as currently under development.

## Sources

- [License: Apache-2.0](https://github.com/magda-io/magda/blob/main/LICENSE)
- [magda-io/magda on GitHub](https://github.com/magda-io/magda)
- [Project website](https://magda.io)
- [README](https://github.com/magda-io/magda/blob/main/README.md)
- [Releases](https://github.com/magda-io/magda/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/magda-io-magda