Fairlearn: group fairness metrics and mitigation algorithms for scikit-learn models
A Python package to assess and improve fairness of machine learning models.
At a glance
- What is it?
- Fairlearn is an MIT-licensed Python package that splits into two jobs: assessing a model's behaviour across protected groups, and running mitigation algorithms when that assessment shows a gap. The hard part is not the API, it is choosing which groups and which constraint you are willing to defend.
- Who is it for?
- Adopt Fairlearn if you already have a fitted scikit-learn style estimator, a defensible definition of the groups you care about, and a stakeholder who can accept a trade-off between a fairness constraint and accuracy. Do not adopt it if you are looking for a metric that tells you the model is fair; the README states plainly that quantitative fairness metrics cannot all be satisfied simultaneously and that justice and due process are not captured by them.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem Fairlearn addresses, and the framing it imposes
A model can be accurate on average and still extend or withhold opportunities unevenly. Fairlearn's README names two harm categories. Allocation harms occur when a system extends or withholds opportunities, resources, or information, with hiring, school admissions, and lending given as the examples. Quality-of-service harms occur when a system simply works less well for one person than another, even with nothing allocated or withheld. That split is the package's central framing, and it is worth taking seriously before writing any code, because the two categories lead to different metrics and different mitigation choices.
The package follows group fairness, which the README describes as asking which groups of individuals are at risk of experiencing harms. The groups are not inferred. They must be specified by the data scientist and are application specific. That single sentence is the most important constraint in the whole project: Fairlearn does not decide who is protected, and it cannot tell you whether the groups you chose are the right ones. It formalizes group fairness as a set of constraints requiring some aspect of the system's behaviour to be comparable across groups, then gives you the machinery to measure and to push on those constraints.
Two components: metrics for assessment, algorithms for mitigation
The package has exactly two components according to the README. Metrics assess which groups are negatively impacted by a model and let you compare multiple models across fairness and accuracy metrics. Algorithms mitigate unfairness across a variety of AI tasks and fairness definitions. The separation matters operationally. Assessment is a read-only pass over predictions and sensitive features; mitigation changes the estimator or its output. You can ship the first without the second, and many teams probably should, because the assessment result is what tells you whether mitigation is warranted at all.
The README does not enumerate the individual metric functions or the individual algorithm classes in the text supplied here. It points instead to an in-depth guide on the Fairlearn metrics and to a mitigation overview on the project website. If you need the exact call signatures, the API reference is the place to look, and I am not going to guess at names that are not in the material.
Install and the first working loop
Installation is a single command: pip install fairlearn. The README then sends you to the quickstart guide, the user guide, and the example notebooks rather than reproducing a code sample inline. That is a deliberate documentation strategy, and it has a consequence worth flagging: the README alone is not enough to get a first result, because the metric and algorithm names live on the website.
The README does state one compatibility caveat that will bite people. Example notebooks downloaded from the main branch may not be compatible with pip-installed versions. If you follow a notebook from main against a pip release, you can hit import errors or changed signatures that have nothing to do with your data. Pin the release and read the matching documentation, or install from source if you want the notebooks as written.
For help, the project routes questions to Stack Overflow under the fairlearn tag and to a Discord community, with a separate #development channel for contributors. Security issues go to an email address rather than the public issue tracker, which is the right default for a library that touches model behaviour.
Where Fairlearn stops, stated by the project itself
The README carries a note that is unusually direct for a library of this kind. Fairness is described as fundamentally a sociotechnical challenge. Many aspects of fairness, such as justice and due process, are not captured by quantitative fairness metrics. And many quantitative fairness metrics cannot all be satisfied simultaneously. The stated goal is to let humans assess different mitigation strategies and then make trade-offs appropriate to their scenario.
Read that as a boundary, not a disclaimer. If your requirement is a single number that certifies a model as fair, Fairlearn will not produce it, and no configuration of it will. If your requirement is to see how a candidate model distributes outcomes across the groups you named, and to see what a mitigation algorithm costs you in accuracy, the package is aimed exactly at that. The wrong-tool case is a team that wants an audit result to hand to a regulator without having made a judgement call about which constraint they are willing to defend. The library will give them the numbers and leave the judgement where it was.
How it compares with error-analysis tooling
The obvious adjacent tool is the What-If Tool, which ships with TensorFlow's tooling and presents a browser interface for slicing a model's behaviour across features, including sensitive ones. The difference in approach is structural rather than cosmetic. What-If Tool is an interactive inspection surface: you look at slices and form a judgement visually. Fairlearn is a Python library inside your training and evaluation code, so its metrics can be asserted in a test, logged per release, or used as an input to a mitigation step in the same script. That makes Fairlearn better suited to a pipeline that runs repeatedly, and What-If Tool better suited to a one-off investigation where you do not yet know which slices matter.
The trade-off is real in both directions. Fairlearn gives you no visual explorer, and What-If Tool gives you no mitigation algorithm that alters the fitted model. If you need both, you are running two tools, not choosing between them.
Release cadence and what an upgrade actually costs
The release history in the repository shows v0.12.0 in December 2024, v0.13.0 in October 2025, and v0.14.0 in June 2026. That is roughly two releases a year, and the version numbers have not reached 1.0, which is a statement about API stability expectations rather than about quality. For a library that sits in an evaluation script, a minor bump twice a year is manageable. For a library that sits inside a production scoring path, it means you should pin the version and read the release notes before moving.
The upgrade cost is concentrated in two places. Metric and algorithm signatures can change between minor versions, and the README's warning about main-branch notebooks versus pip-installed versions is the visible symptom of that. The second cost is conceptual rather than technical: if you change which groups you pass as sensitive features, every number you previously reported becomes incomparable, because group fairness is defined relative to the groups specified by the data scientist. A version bump is cheap. A change in the group definition invalidates your history.
The licence is MIT, which is permissive and imposes no copyleft obligation on your own code. That is a statement about the licence text, not legal advice; if Fairlearn ends up inside a distributed product, your own legal review is the thing that decides.
Who should pick this up, and what to check first
Fairlearn fits teams already working in the scikit-learn style, with a fitted estimator, a labelled dataset, and at least one column that identifies the groups they care about. It fits them because the assessment step is cheap to add and produces evidence rather than opinion, and because the mitigation algorithms are there when the evidence shows a gap the team is willing to trade accuracy against.
It does not fit teams without a group definition they can defend, teams expecting a fairness verdict, or teams whose sensitive attributes are missing or unreliable for a meaningful fraction of the scored population. In that last case the metrics will run and the numbers will be misleading, which is worse than not running them.
The concrete first check is the version alignment: install with pip install fairlearn, note the version, and confirm the example notebooks you intend to follow match it, since the README warns that notebooks from main may not work against a pip install. The second check is data-side: confirm the sensitive-feature column is populated for every row you will score, because the package will not tell you when a group is too small or too incomplete for its metrics to mean anything.
Editorial conclusion
Adopt Fairlearn if you already have a fitted scikit-learn style estimator, a defensible definition of the groups you care about, and a stakeholder who can accept a trade-off between a fairness constraint and accuracy. Do not adopt it if you are looking for a metric that tells you the model is fair; the README states plainly that quantitative fairness metrics cannot all be satisfied simultaneously and that justice and due process are not captured by them. Before committing, verify three things in your own environment: that the pip-installed version matches the version the example notebooks were written against, since the README warns notebooks downloaded from main may not be compatible with pip-installed versions; that the sensitive-feature column you plan to pass exists and is populated for every row you score, because group fairness is defined over groups the data scientist specifies; and that your team has written down which of the two harm categories, allocation or quality of service, your use case falls into, because that choice determines which metric and which mitigation algorithm are even meaningful.
Community notes