# PipelineWise runs Singer ELT pipelines with per-connector isolation

> PipelineWise is a Python 3.12 framework for configuring, running and operating Singer ELT pipelines, with a small supported connector set and isolated virtual environments per connector. It suits a data team that has standardised on Singer taps and targets, and it is a poor fit if you expect a broad catalogue of production-ready connectors.

**transferwise/pipelinewise** — Data Pipeline Framework using the singer.io spec

- Repository: https://github.com/transferwise/pipelinewise
- Website: https://transferwise.github.io/pipelinewise
- Stars: 661 · Forks: 140
- Language: Python
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/transferwise-pipelinewise

## The connector catalogue has been cut back on purpose

PipelineWise started as a broad data movement framework and is now a much narrower one, and the readme is direct about it. The project scope section states that available sources are MariaDB and PostgreSQL, that available targets are PostgreSQL and Snowflake, and that other packaged connectors, including the Snowflake source, are experimental. The supported table is four rows. The source for MariaDB is tap-mysql, the source for PostgreSQL is tap-postgres, the target for PostgreSQL is target-postgres and the target for Snowflake is target-snowflake. MariaDB and MySQL share tap-mysql, and the readme adds that MariaDB is available while MySQL remains experimental. The experimental list is long, with packaged experimental sources including GitHub, Jira, Kafka, Mixpanel, MongoDB, MySQL, S3 CSV, Salesforce, Slack, Snowflake, Twilio and Zendesk, and target-s3-csv as an experimental target. The definition of what packaging means is given explicitly. Packaging means the component is included by make all_connectors, and it does not imply production support. A reader who skims a long connector list and assumes it is a catalogue of supported integrations will get this wrong, so the readme points at the connector support and route matrix before you create a pipeline.

## Singer as the contract, PipelineWise as the operator

The framework sits on the Singer specification, which is what makes the shape of the system understandable. PipelineWise is a framework for configuring, running and operating Singer ELT pipelines, and it supports log-based, incremental and full-table replication, plus FastSync transfers for selected database routes. That list matters because it is the actual replication vocabulary, and the differences are the ones that bite. Log-based replication reads a change log, incremental replication resumes from a bookmark, and full-table replication copies the whole table each time. FastSync is a separate mechanism for selected database routes, and it is a different performance and consistency profile rather than a variation on the same path. The point of adopting a framework rather than running taps directly is the operation around the transfer, scheduling, state, status and error handling, and the readme describes the project as being for configuring, running and operating these pipelines rather than merely moving data. The README also names a historical reference point. Release v0.64.1 is the last release from before the connector set was reduced, and the readme is careful to say it is a historical reference and not a recommendation to deploy an older release. That distinction is worth holding on to, because the older release does have the wider connector set.

## Docker is the recommended runtime, and the alias hides the plumbing

Docker is the recommended runtime because it isolates connector and system dependencies, and the install path is a fixed sequence of commands. You clone the repository, change into the directory, pull the published image, tag it locally, create an alias that points at the wrapper script, and check the status:

```bash
git clone https://github.com/transferwise/pipelinewise.git
cd pipelinewise
docker pull transferwiseworkspace/pipelinewise:latest
docker tag transferwiseworkspace/pipelinewise:latest pipelinewise:latest
alias pipelinewise="$(pwd)/bin/pipelinewise-docker"
pipelinewise status
```

The alias is the clever bit, because the wrapper script is what turns a normal pipelinewise command into a run inside the container. The wrapper persists generated configuration, state and logs below ~/.pipelinewise on the host, so the container can be thrown away between runs without losing the pipeline definitions. The readme's production advice is to pin a release tag instead of latest. To customise the image you build it locally with docker build, and the Dockerfile in the repository shows how heavy that image is. The base is python:3.12-slim-trixie, it installs a set of system packages including mbuffer, wget and tzdata, and it separately fetches the MongoDB database tools as a Debian package with a pinned version and a SHA-256 checksum verified during the build, with a comment explaining that MongoDB does not publish those tools for ARM64.

## A source install creates one virtualenv per connector

The source route is different in kind, because the dependency isolation happens inside the machine rather than inside a container. Source installations own Python, system-library and connector compatibility, and the documented sequence installs the CLI and only the connectors you asked for:

```bash
make pipelinewise
make connectors -e pw_connector=tap-postgres,target-snowflake
export PIPELINEWISE_HOME="$(pwd)"
source .virtualenvs/pipelinewise/bin/activate
pipelinewise status
```

The second command is the one that matters, because the pw_connector variable selects which Singer connectors get built, and each one is installed into its own environment under the .virtualenvs directory. The Makefile shows the isolation explicitly, with a VENV_DIR derived from PIPELINEWISE_HOME and a per-connector environment created inside it. It also prints a warning during installation, that the license of some connectors are different than the default PipelineWise license, and that you should abort the installation if you do not wish to accept. The readme's warning is just as direct. Do not use a root pip install as a replacement for the Makefile workflow, because it does not create isolated connector environments. The reason is the same one that makes the Docker route recommended, and it is a real operational property rather than a stylistic preference. Connector environments are the mechanism that lets a pipeline keep a Kafka tap and a Postgres tap side by side without their dependency sets colliding.

## Tests are credentialed, and the readme says so loudly

The development section is a warning about running the test suite. Use the dev-project Docker environment for development, tests and verification wherever possible, because it provides Linux, source databases, targets, and the runtime layout closest to production. Then the instruction that saves an afternoon: do not run bare pytest tests/, because it collects credentialed end-to-end tests. That distinction between unit tests and integration tests is easy to miss and expensive to learn the hard way, and the same file makes the point with a comment in the ruff configuration, where connector suites outside GitHub connector CI remain out of scope, and a long extend-exclude list naming individual integration test files under the Singer connector directories. The authoritative commands live in the repository and the scoped AGENTS.md files, for the lint, unit, connector, E2E and documentation commands. The contribution rule is worth quoting because it explains the whole shape of the project. New connectors begin as experimental until their ownership, compatibility, recovery, CI and operated route are documented. That is a maintenance policy expressed as a gate, and it explains why the experimental list is long while the supported list is short.

## Licence obligations follow the connectors you ship

The licence position is a genuine constraint and the readme handles it with more care than most projects would. PipelineWise core is licensed under the Apache License 2.0. Packaged connectors can use different licences, including AGPL 3.0, and the obligations of a distributed image depend on every included component. There is a license inventory page and a LICENSE file for the detail. This is the second place where the connector selection has consequences beyond function, since the Makefile prints the licence warning during the build and the Docker build passes a pw_acceptlicenses flag to accept on your behalf. Building the image with connectors=all therefore accepts the licence terms of every packaged connector at once, which is worth remembering before you build a distributed image. The project itself is run by Wise, given the transferwise organisation and the author field in setup.py, and the version there is 0.96.3 with a strict python_requires of ==3.12.*, so the framework targets one interpreter version exactly rather than a range. The dependency list in setup.py is pinned to exact versions throughout, with a comment on the SQLAlchemy pin explaining that it is Alembic's engine and is pinned rather than implicit.

## Conclusion

PipelineWise is worth adopting if your data movement already speaks Singer and you want the operational parts, isolated connector environments, scheduled runs and status reporting handled for you rather than by hand. It is the wrong choice if you need an unsupported source such as a SaaS API as a production dependency, because everything outside MariaDB and PostgreSQL sources and PostgreSQL and Snowflake targets is labelled experimental, and the readme is explicit that packaging does not imply production support. Verify first that the route you need appears in the connector support and route matrix, that the connectors you select are the ones with documented ownership, compatibility, recovery and CI, and that the licence obligations of every component you ship in the image are acceptable. The last push was on 2026-10-18 and the current release is 0.96.3 from 2026-10-14.

## FAQ

### Which connectors are fully supported in PipelineWise?

Four. The sources are MariaDB through tap-mysql and PostgreSQL through tap-postgres, and the targets are PostgreSQL through target-postgres and Snowflake through target-snowflake. Everything else packaged in the repository is labelled experimental, which the readme says does not imply production support.

### How do I install PipelineWise with Docker?

Clone the repository, pull the transferwiseworkspace/pipelinewise image, tag it locally, alias pipelinewise to the bin/pipelinewise-docker wrapper and run pipelinewise status. The wrapper keeps generated configuration, state and logs under ~/.pipelinewise on the host, and the readme advises pinning a release tag rather than latest in production.

### How do I install only the connectors I need from source?

Run make pipelinewise and then make connectors with a pw_connector value listing them, for example tap-postgres,target-snowflake, then set PIPELINEWISE_HOME and activate the virtualenv. The readme warns against a root pip install as a substitute, because it does not create the isolated connector environments.

### Why should I not run pytest tests/ in the PipelineWise repository?

Because it collects credentialed end-to-end tests. The readme directs you to the dev-project Docker environment for development, tests and verification, and to the repository and scoped AGENTS.md files for the authoritative lint, unit, connector, E2E and documentation commands.

### What licence applies to PipelineWise and its connectors?

The core is Apache License 2.0, while packaged connectors can use different licences including AGPL 3.0, so the obligations of a distributed image depend on every included component. The Makefile prints a licence warning during connector installation and the project publishes a license inventory.

## Sources

- [License: Apache-2.0](https://github.com/transferwise/pipelinewise/blob/master/LICENSE)
- [Project website](https://transferwise.github.io/pipelinewise)
- [README](https://github.com/transferwise/pipelinewise/blob/master/README.md)
- [Releases](https://github.com/transferwise/pipelinewise/releases)
- [transferwise/pipelinewise on GitHub](https://github.com/transferwise/pipelinewise)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/transferwise-pipelinewise