pdpipe turns pandas preprocessing into a chain of named stages

A method chain over DataFrames instead of a fit and transform object

The problem is that pandas preprocessing written inline in a notebook is hard to reuse and harder to review. pdpipe addresses that with a small vocabulary of named stages. The project describes itself as easy pipelines for pandas DataFrames, and the README's opening example is short enough to see the whole idea. You build a DataFrame, chain two stages on it, and call the pipeline with the frame to get a new frame. The stages are named for what they do, ColDrop and OneHotEncode here, and the chain is read left to right as a description of the transformation. The install is one command:

  pip install pdpipe

The intended user is a data scientist working with tabular data who wants preprocessing steps that can be reused, named and documented rather than copy-pasted, and the topics on the repository are pandas, dataframe, pipeline and data science, which says the audience plainly.

Ready-made stages cover the boring half of preprocessing

The README lists what ships in the box, and it is a familiar set of tasks: column dropping, value filtering, one-hot encoding, column mapping, row-wise feature generation, dataframe-wide feature generation, and scikit-learn-style transformations. The last of those is what bridges to the wider Python data stack, and the package is explicit that some stages need extra dependencies. The optional stages are grouped into two extras in the packaging metadata, one for scikit-learn and one for nltk, and the README says the same thing in prose. The dependency list in pyproject.toml is short and mostly explains what each piece is for: birch for reading config from files and environment variables, pandas at version 2 or newer because pdpipe is a pandas extension, skutil because the Scale stage uses a scaler from it, sortedcontainers because the Bin stage needs a sorted list, strct because ColReorder uses a reverse dict helper from it, and tqdm for progress bars on some stages.

The example in the README shows a two-stage transform

The README example is the fastest way to understand the shape of the API. A DataFrame is built from a small list of rows with medals, height and country of birth columns, then two stages are chained, one dropping the Medals column and one one-hot encoding the Born column, and the pipeline is applied to the frame. The result shows the Height column alongside two new indicator columns, one for the United Kingdom and one for the United States, with the third row having zeros in both because Greece is not among the encoded categories. That is worth pausing on, because it is the design decision in one glance. One-hot encoding drops the reference category, so a frame with three countries produces two columns rather than three, which is a sensible default for modelling but a surprise if you expect the original cardinality to be preserved.

Optional dependencies disappear silently, with a warning

The failure mode to design around is the optional dependency. The README states that some pipeline stages require scikit-learn, that they will simply not be loaded if scikit-learn is not found on the system, and that pdpipe will issue a warning, and it says the same about nltk. So the package imports, the stages that need the missing library are absent, and you find out when you reach for a stage and it is not there. For a library meant to be dropped into a notebook, that is a defensible choice, since scikit-learn is a heavy dependency and most of the built-in stages do not need it. It does mean a missing dependency shows up as an attribute error at the point of use rather than an install error, which is the kind of thing to catch in a CI environment rather than at the call site.

Recent APIs add diagrams, traces and parallel execution

The README lists the recent APIs, and they are the ones to look at if you are deciding now. PdPipeline.to_dot() produces dependency-free Graphviz DOT pipeline diagrams, which means you can produce a picture of a pipeline without installing a graphing library. PdPipeline.trace() gives structured per-stage dry-run diagnostics. Diff applies pandas.Series.diff to selected columns. SklearnColumnTransform wraps arbitrary matrix-to-matrix scikit-learn transformers while preserving DataFrame column context, which is the part that matters, because a plain sklearn transformer would hand you an array and lose the column names. There is also optional n_jobs thread-based parallel execution in ApplyByCols and ApplyToRows, with serial execution remaining the default, so the parallel path is opt-in rather than a change in behaviour.

The chainer pattern is a config switch, not a compiler feature

There is a second calling convention that the README describes in passing, and it explains what LOAD_STAGE_ATTRIBUTES does. Stage attributes enable the chainer construction pattern, so a call like pdp.ColDrop('b').Bin('f') works, and the configuration option LOAD_STAGE_ATTRIBUTES, true by default, controls whether those attributes are loaded at all. The stated use is sensible documentation generation, and setting it to false in cfg.json or exporting PDPIPE__LOAD_STAGE_ATTRIBUTES=False stops the attributes from being loaded. That is the whole of the configuration surface, and the README says the options are only relevant for development at the moment. The file lives at $XDG_CONFIG_HOME/pdpipe/cfg.json, or at ~/.pdpipe/cfg.json if XDG_CONFIG_HOME is not set, and birch is the dependency that reads it.

A 100 percent coverage requirement shapes what contributions look like

The contribution guide is unusually strict, and it tells you what the maintainer values. pdpipe is maintained with a test coverage of 100 percent, and the README says that although challenging, the author hopes to maintain this status. Adding code means testing it thoroughly, and Codecov reports coverage changes on each pull request, with the note that a pull request reducing coverage will not be examined before that is fixed. The test layout is prescribed too, with tests under a tests directory where each module has its own folder and each class, usually a pipeline stage, has a dedicated file whose name starts with test, containing global functions whose names also start with test. A development install clones the repository and installs it in editable mode with the test extras:

  cd pdpipe
  pip install -e ".[test]"

Style is enforced by flake8, with a Travis job checking for a successful run of the flake8 CLI in the repository root, and documentation follows the numpy docstring conventions.

An MIT package with a beta classifier and Python 3.10 floor

The licensing is the easy part. The package is under the MIT Licence, with the license file referenced from the packaging metadata and a badge in the project header, and the classifier set includes the MIT licence entry. The maturity signal is more mixed. The distribution is classified as Development Status 4, Beta, and the README describes the package as very much in its infancy with many other pipeline stages still to be added, and the maintainer invites contributors with the offer of help. The supported Python versions run from 3.10 to 3.14, and the version is dynamic, read from an attribute in the package, with the recent releases being v1.1.1 for smarter warnings on missing optional dependencies, v1.0.1 for bugfixes, and v1.0.0 which made pandas 2.0 an explicit dependency. The last push to the repository was on 2026-10-05.

Editorial conclusion

pdpipe suits a data scientist who writes the same column-dropping, filtering and encoding steps across many notebooks and wants them named, reusable and readable, and it suits a team that wants a pipeline diagram for documentation. It does not suit a project that needs a scikit-learn compatible estimator to slot into an existing ColumnTransformer pipeline, since the package deliberately uses a method-chaining API rather than the fit and transform convention. Before adopting, check which optional stages you actually need and install scikit-learn or nltk up front, since the package only warns when they are missing, and note that the last push to the repository was on 2026-10-05 with release v1.1.1 dated 2026-03-06.

Frequently asked questions

Official sources

1. License: MIT
2. pdpipe/pdpipe on GitHub
3. Project website
4. README
5. Releases