# polars-ds: statistics and models as Polars expressions

> PDS, published as polars-ds, is a Rust extension for Polars that exposes regression, evaluation metrics, string and array distances and statistical tests as expressions you can put inside a select or a group_by. It installs from PyPI as a wheel, but its dependency stack reaches into forked crates and pinned Polars versions, which is where the operational risk sits.

**abstractqqq/polars_ds_extension** — Polars extension for general data science use cases

- Repository: https://github.com/abstractqqq/polars_ds_extension
- Stars: 656 · Forks: 44
- Language: Rust
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/abstractqqq-polars-ds-extension

## A Polars extension rather than a separate data science stack

PDS is a data science package that sits on top of the Polars dataframe library. The README lists its qualities as being fast, small with minimal dependencies, having an intuitive API for people who already know Polars, and being dataframe friendly. The coverage is broad: simple statistics, linear regression, string edit distances, tabular data transforms, feature extraction, modelling pipelines and model evaluation metrics.

What makes it different from a typical data science package is where the computation lives. Instead of returning fitted estimator objects that you then call predict on, the functions here are expressions. The README example evaluates several metrics inside a group_by aggregation, so the metric computation happens as part of the query rather than in a separate pass over extracted rows.

The packaging follows from that. The project is written in Rust and built with maturin, exposing a native extension module named polars_ds._polars_ds, and the Python import name is polars_ds while the distribution on PyPI is polars-ds. The README gives the install command as pip install polars-ds. Runtime dependencies are kept to two entries: polars itself and, on Python 3.11 and older, typing-extensions. The heavier libraries for plotting, model evaluation and scikit-learn compatibility are optional extras, so a plain install stays small.

The repository supports this with a normal Python project layout: a python directory holding the Python side of the package, a src directory for the Rust crate, maths for mathematical notes, benchmarks, docs built with mkdocs and served through readthedocs, examples in notebook form, and a SKLEARN_COMPATIBILITY.md file that maps the API against scikit-learn.

## Metric functions that run inside a group_by aggregation

The headline example computes several evaluation metrics per segment in one lazy query:

```python
import polars as pl
import polars_ds as pds
```

```python
df.lazy().group_by("segments").agg( 
    # any other metrics you want in here
    pds.query_roc_auc("actual", "predicted").alias("roc_auc"),
    pds.query_log_loss("actual", "predicted").alias("log_loss"),
).collect()
```

Two things are worth pulling out of that snippet. The functions are named with a query_ prefix because they are called as Polars query context functions, taking column names as strings and being usable inside an aggregation, which is why the README describes it as parallel evaluation of multiple metrics on different segments of the data. And the metrics are aggregated the same way any other Polars aggregate would be, so the result is a dataframe with one row per segment rather than a set of Python objects you have to reassemble yourself.

The Rust side of that plumbing is visible in Cargo.toml. The crate depends on pyo3 with the abi3-py39 feature and the extension-module feature, and on pyo3-polars with the derive and lazy features, which is the bridge that lets a Rust function be used as a Polars expression. The polars dependency itself is pinned to version 0.55.2 with an explicit feature list, default features off, and it includes streaming, rank, cross_join and diff among others. The comment in Cargo.toml explains the pinning policy, saying that dependencies found in polars should be kept at the same versions to minimise duplicates, and the same reasoning is applied to rand, rand_distr and unicode-normalization.

## Regression inside a select, with a statsmodels-style report

The modelling side follows the same expression-first design. A quick linear regression with predictions and residuals is a single select:

```python
df.select(pds.lin_reg(pl.col("x1"), pl.col("x2"), target=pl.col("y"), add_bias=False, return_pred=True))
```

The README calls these models non-persistent, which is the key difference from a scikit-learn workflow: nothing is saved, and the fitted state lives in the expression rather than in an object you can store and reuse.

The variant worth looking at is lin_reg_report, which returns a regression table in the shape of a statsmodels summary, with columns for beta, std_err, t, p>|t|, the 0.025 and 0.975 bounds, r2 and adj_r2. The README shows it used with polynomial features generated from polars_ds.pipeline.transforms, so you get a design matrix and its summary in one expression. The same call also shows the precision switch: a comment explains that if you want the underlying computation done in f32 rather than f64, you set pds.config.LIN_REG_EXPR_F64 = False.

The model list covers ordinary linear regression, Lasso, Ridge and Elastic Net through the l1_reg and l2_reg arguments, rolling linear regression with skipping, recursive linear regression, and non-negative linear regression with positive = True. One caveat is documented rather than hidden: binary logistic regression with L1 and L2 parameters exists as pds.logistic_reg, but the README states it does not work with F32 yet. Precision control is therefore not uniform across the model set, which is the sort of detail that matters if you are chasing reproducibility between runs.

## String and array distances, and in-dataframe statistical tests

The distance functions cover the usual edit distance family and work against either a literal or another column:

```python
df.select( # Column "word", compared to string in pl.lit(). It also supports column vs column comparison
    pds.str_leven("word", pl.lit("asasasa"), return_sim=True).alias("Levenshtein"),
    pds.str_osa("word", pl.lit("apples"), return_sim=True).alias("Optimal String Alignment"),
    pds.str_jw("word", pl.lit("apples")).alias("Jaro-Winkler"),
)
```

The return_sim argument adds a similarity score alongside the distance, which is what you usually want for fuzzy matching pipelines. The Rust dependency for these is rapidfuzz at version 0.5.0, so the heavy lifting is done by a dedicated fuzzy matching library rather than a hand-written implementation.

Array and list distances work on fixed-shape arrays:

```python
df = pl.DataFrame({
    "x": [[1,2,3], [4,5,6]]
    , "y": [[0.5, 0.2, 0.3], [4.0, 5.0, 6.1]]
})
```

The README explains that arr_sql2_dist is the squared L2 distance, the same as squared Euclidean distance, and that list_sql2_dist is the drop-in replacement for list columns.

Statistical tests take the same expression form, grouped by a key in the README example:

```python
df.group_by("market_id").agg(
    pds.ttest_ind("var1", "var2", equal_var=False).alias("t-test"),
```

The equal_var argument selects between the equal-variance and unequal-variance variants of the independent two-sample t-test.

## Installing from PyPI and building the extension locally

For users, the install is a single pip command, and the wheels are built with maturin with the pyo3 extension-module feature, so the binary is a compiled extension module rather than pure Python.

For contributors, the Makefile drives the same toolchain. The requirements target creates a virtual environment, installs uv into it and then uses uv to install the development, test and documentation requirements:

```bash
dev-release: .venv
	unset CONDA_PREFIX && \
	source .venv/bin/activate && maturin develop --release -m Cargo.toml
```

The dev-release target unsets CONDA_PREFIX before running maturin develop, so a conda environment in the shell does not interfere with the build. The pre-commit target runs cargo fmt and then pre-commit on all files, which matches the .pre-commit-config.yaml in the repository root.

The local requirements list is small and tells you what the development loop needs: maturin with patchelf on Linux, polars, pre-commit, ipykernel, numpy, matplotlib and jupytext. The last one explains the jupytext-formatted examples, and the ipykernel entry explains how the notebooks in examples are run. The examples directory includes a basics notebook, an exploratory data analysis notebook, a pipeline notebook and a sample and split notebook, plus a CSV and a Parquet file to work against.

## Git dependencies on forked crates are the sharpest edge

Two of the Rust dependencies are not taken from a registry. The linear algebra library faer is pulled from a Codeberg repository at a specific revision, and argmin, the optimisation crate, is pulled from a fork under the project author's own GitHub account on the main branch:

```toml
faer = { git = "https://codeberg.org/sarah-quinones/faer", rev = "8377404e78", default-features = false, features = [
    "nightly",
    "rayon",
    "sparse-linalg",
] }
argmin = { git = "https://github.com/abstractqqq/argmin.git", branch = "main" }
```

The faer dependency enables a nightly feature, which means the build expects a recent Rust toolchain, and the repository pins one through rust-toolchain.toml. The argmin dependency tracks a branch rather than a commit hash, so a rebuild at a later date can pick up different code than the build that produced the released wheel. There is a commented-out alternative in Cargo.toml that would take faer from crates.io at version 0.24 instead, with the same feature set.

This matters for reproducibility rather than for everyday use. If you install the wheel from PyPI, none of this reaches you. If you build from source, or vendor the crate, you are depending on a moving branch and on a nightly feature of a third-party library.

The Python side has a similar caution. The dependency on polars is written as >= 1.4.0 with four versions explicitly excluded, 1.25, 1.26, 1.32.1 and 1.32.2, which tells you the project has hit incompatibilities with specific Polars releases. The Rust crate pins polars 0.55.2 separately, and those two version lines are different numbering schemes, so a mismatch between the Python polars you have installed and the Rust polars the extension was built against is a plausible source of trouble.

## Versioning, licence and when to stay with the standard stack

The release cadence is quick. Version 0.12.0 shipped on 2026-06-01, 0.12.1 on 2026-08-16 and 0.13.0 on 2026-09-20, and the last push was on 2026-10-16. The version in pyproject.toml and Cargo.toml is already 0.13.1, which is the development version ahead of the 0.13.0 tag. The project classifiers call it Production/Stable, which is a reasonable claim for an API that has had three minor releases in four months with a compatibility document tracking scikit-learn.

Python 3.9 is the floor, and the wheel is built for CPython and PyPy. The licence is MIT, stored in LICENSE.txt and referenced from pyproject.toml, so there is no copyleft obligation attached to the extension itself.

The honest alternative to consider is the standard Python stack. scikit-learn and statsmodels give you estimators with fit and predict, a model persistence story, and a much wider catalogue of algorithms, and they remain the right answer when you need to save a trained model or deploy it as a service. PDS gives you something different: computation fused into the dataframe engine, so a metric or a regression summary is produced in the same pass as the aggregation, with no round trip to Python objects and no separate model object to persist. That is a genuine win for exploration and for batch scoring inside an existing Polars pipeline, and a genuine loss when the trained model is the asset.

Two further limits are worth stating plainly. PDS covers classical models, not the deep learning side of data science, so if your task list includes gradient boosted trees from this package alone you will be disappointed. And the results are computed inside the query engine with optional f32 arithmetic, so a pipeline that must reproduce a reference implementation bit for bit is a case where the default f64 path, and a direct comparison against your existing numbers, is the first thing to check.

## Conclusion

PDS suits a Polars shop that wants model outputs and evaluation metrics to live inside the dataframe query rather than in a separate Python pipeline, and it suits a scikit-learn and statsmodels workflow that is already working. Before adopting it, check that your installed polars version is not one of the excluded releases, since the project pins out 1.25, 1.26, 1.32.1 and 1.32.2, and expect the install to pull binary wheels for a Rust extension rather than a pure Python package. If you need logistic regression, note that the README states pds.logistic_reg does not work with F32 yet.

## FAQ

### How do I install polars-ds?

The README gives the install command as pip install polars-ds. The package is built with maturin and installs a native extension module, polars_ds._polars_ds, rather than pure Python.

### What Python and Polars versions does polars-ds support?

Python 3.9 or newer is required, with wheels for CPython and PyPy. The polars dependency is >= 1.4.0 with versions 1.25, 1.26, 1.32.1 and 1.32.2 excluded, and the Rust crate pins polars 0.55.2.

### How do I compute ROC AUC and log loss with polars-ds?

Use the query_ functions inside an aggregation, for example pds.query_roc_auc("actual", "predicted") and pds.query_log_loss("actual", "predicted") within a group_by agg, then collect the result.

### Does polars-ds support scikit-learn models?

No. It provides its own implementations of linear models, metrics, distances and statistical tests as Polars expressions, and the repository includes a SKLEARN_COMPATIBILITY.md that maps the API against scikit-learn rather than shipping those estimators.

### Can I use f32 precision with polars-ds regression?

Yes, by setting pds.config.LIN_REG_EXPR_F64 = False so the computation is done in f32. The README notes that logistic regression does not work with F32 yet.

## Sources

- [abstractqqq/polars_ds_extension on GitHub](https://github.com/abstractqqq/polars_ds_extension)
- [Issues](https://github.com/abstractqqq/polars_ds_extension/issues)
- [License: MIT](https://github.com/abstractqqq/polars_ds_extension/blob/main/LICENSE)
- [README](https://github.com/abstractqqq/polars_ds_extension/blob/main/README.md)
- [Releases](https://github.com/abstractqqq/polars_ds_extension/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/abstractqqq-polars-ds-extension