# FL-bench puts forty federated learning methods on one comparison table

> FL-bench is a Python benchmarking harness for federated learning that implements traditional and personalized aggregation methods side by side so their behaviour can be compared under the same conditions. It suits researchers who want to reproduce published results or try a new aggregation rule without building a simulator first, and it is a research codebase rather than a production framework.

**KarhouTam/FL-bench** — Benchmark of federated learning. Dedicated to the community. 🤗

- Repository: https://github.com/KarhouTam/FL-bench
- Stars: 694 · Forks: 116
- Language: Python
- License: GPL-3.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/karhoutam-fl-bench

## The point of the project is comparison, not a single algorithm

FL-bench is described as a benchmark of federated learning, dedicated to the community, and the tagline the project puts at the top is Benchmarking Federated Learning Methods. That framing is the whole design. Federated learning research produces a steady stream of new aggregation rules, and each paper arrives with its own experimental setup, its own data partition, its own client sampling and its own hyperparameter choices. The result is that two papers reporting the same accuracy improvement are rarely comparable. A harness that runs many methods through one code path removes that variability. The project structure supports the claim. There is a single main.py entry point, a src/ directory holding the implementations, a config/ directory for experiment configuration, a data/ directory, and a generate_data.py script. One runner, one configuration format, one partitioning scheme, many methods. That is the infrastructure a comparison needs.

## The traditional methods list spans fourteen aggregation rules

The traditional half of the catalogue starts with FedAvg, the Communication-Efficient Learning of Deep Networks from Decentralized Data paper, and works forward through the field. FedAvgM addresses the effects of non-identical data distribution for federated visual classification. FedProx targets federated optimization in heterogeneous networks, and SCAFFOLD introduces stochastic controlled averaging. MOON adds model-contrastive federated learning, while FedDyn applies dynamic regularization and FedLC handles label distribution skew via logits calibration. FedGen does data-free knowledge distillation for heterogeneous settings, and CCVR adds classifier calibration for non-IID data. FedOpt is adaptive federated optimization, FedADMM is a framework with adaptivity to system heterogeneity, Elastic Aggregation is a CVPR 2023 contribution, FedFed uses feature distillation against data heterogeneity, and DTFL is a dynamic tiering approach for heterogeneous environments from IEEE IoT-J in 2024. The span matters: it covers optimisation, regularisation, calibration, distillation and systems aware tiering, so the harness is not limited to one family of ideas.

## The personalized catalogue is larger and covers several distinct strategies

The personalized side of the benchmark is the larger of the two lists, and its size is the clearest signal of where the field has moved. It includes Local-Only, which is local training with no communication at all, and that baseline is worth pausing on, because it is the honest floor for any personalized method. A method that cannot beat training entirely locally has not demonstrated that communication helps. From there the list covers model distillation with FedMD, adaptive personalization with APFL, local and global representations with LG-FedAvg, local batch normalisation with FedBN, personalization layers with FedPer, shared representations with FedRep, meta-learning with Per-FedAvg, Moreau envelopes with pFedMe, multi-task mixtures with FedEM, fair and resilient personalization with Ditto, hypernetworks with pFedHN, layer-wise aggregation with pFedLA, clustered learning under privacy constraints with CFL, first order optimisation with FedFomo, representation learning with FedBabu, adaptive batchnorm for healthcare with FedAP, local memorization with kNN-Per, federations with cyclic knowledge distillation with MetaFed, bridging generic and personalized learning with FedRoD, prototype learning with FedProto, feature alignment and classifier collaboration with FedPAC, adaptive local aggregation with FedALA, and learning to learn with PeFLL. Several private methods are included, and the project explicitly flags pFedSim as the author's own work.

## The repository layout is a data generator, a runner and a config directory

The top level of the repository is small and tells you how a run is assembled. There is main.py as the entry point, a src/ directory for the method implementations, a config/ directory for experiment configuration, a data/ directory, and generate_data.py for producing the data. There is also a .env/ directory at the top, and the topics list includes ray and parallelism, which is the hint about how the client simulation is distributed. The .env directory is the one item to look at carefully before a run, because environment files in a machine learning project usually hold paths to datasets and possibly credentials, and the project does not publish a reference configuration in the README. The topics also list differential-privacy, pytorch-implementation, parallelized and personalized-federated-learning, so the harness is built on PyTorch and has some privacy-aware components in scope. A CITATION.cff file is present, which is the file a project uses to declare how its results should be cited, and for a benchmark that is a detail worth respecting.

## A benchmark compares methods, it does not certify a deployment

It is worth being blunt about what a harness like this can and cannot tell you, because the name invites the wrong reading. FL-bench gives you a shared implementation of many methods under one partitioning and sampling regime, which lets you answer whether method A beats method B when you control everything else. It cannot tell you whether method A will beat method B in your deployment, where the client population is heterogeneous, the devices are slow and intermittently offline, the data is skewed in ways the simulation does not model, and the communication budget is a phone plan rather than a simulation parameter. The benchmark also inherits the assumptions of the papers it implements. Each method carries the assumptions of its authors, and reimplementing twenty of them in one framework means the harness's author made choices about what each paper actually does. A result from FL-bench is a result about FL-bench. The value is in the controlled comparison, and the risk is in reading that comparison as a deployment verdict.

## Where FL-bench is the wrong tool

Three situations call for something else. The first is a production system with a real federated deployment. A benchmark harness simulates clients, it does not host them, so it cannot tell you whether your aggregation rule survives a network with 5% packet loss, or how the server behaves when half the clients have a different PyTorch version. The second is a team that needs the algorithm in production rather than the comparison. FL-bench is a research codebase, its licence is GPL-3.0 rather than the permissive Apache or MIT licences that most commercial teams prefer for a component they ship inside a product, and it publishes no GitHub releases, so there is no version to pin a dependency to. The third is a privacy audit. The differential-privacy topic in the repository suggests the harness includes privacy-aware components, but a benchmark does not constitute a privacy evaluation, and a real deployment needs an analysis of what your data leaves the device. The honest alternative for a production need is to implement the aggregation rule inside your own training stack, where you can also own the client behaviour, and to use FL-bench as a sanity check on the algorithm's mechanics rather than as the source of truth for its behaviour.

## What to check before you trust a number from this harness

Three checks will tell you most of what you need. The first is the data. The generate_data.py script and the data/ directory tell you what partition the methods are compared on, and non-IID partitioning, the distribution skew that most of the catalogued papers exist to address, is a parameter rather than a constant. Two results from different partition schemes are not comparable, so check that the scheme matches the paper you are trying to reproduce. The second is the client sampling and communication budget in config/, since a method that needs more rounds is a different method in practice, and the same accuracy after fifty rounds and after five hundred is not the same result. The third is the implementation itself, which means reading src/ for the method you care about, particularly if it is one of the less common entries. The repository is not archived and the last push was on 2026-10-20, so the code is current, and the project states that it welcomes pull requests on everything that can make the project better, which is a reasonable expectation for a community benchmark.

## Conclusion

Adopt FL-bench when you need to compare federated aggregation methods under one harness, to reproduce a paper's protocol, or to plug a new method into an existing set, and build your own simulation stack when your real deployment has data partitions and client behaviour that a generic benchmark cannot represent. Verify first which datasets and configurations ship in the config and data directories, because the repository layout points to a data generation script and a main entry point rather than a documented configuration reference, and read the CITATION.cff file before citing results you produced with it.

## FAQ

### What is FL-bench used for?

It is a benchmark of federated learning methods, meaning implementations of many traditional and personalized aggregation algorithms that run under one harness so their results can be compared under the same conditions. The repository layout is a main entry point, a src directory of methods, a config directory and a data generation script.

### Which federated learning methods does FL-bench implement?

The traditional list includes FedAvg, FedAvgM, FedProx, SCAFFOLD, MOON, FedDyn, FedLC, FedGen, CCVR, FedOpt, FedADMM, Elastic Aggregation, FedFed and DTFL. The personalized list is larger and includes Local-Only, FedMD, APFL, LG-FedAvg, FedBN, FedPer, FedRep, Per-FedAvg, pFedMe, Ditto, FedProto, FedPAC, FedALA, PeFLL and the author's own pFedSim.

### Is FL-bench a production-ready federated learning framework?

No. It is a research benchmarking codebase for comparing methods, not a framework for running a real federated deployment. It publishes no GitHub releases, and its licence is GPL-3.0 rather than a permissive one.

### How is FL-bench configured and run?

The repository provides main.py as the entry point, a config/ directory for experiment configuration, a data/ directory and a generate_data.py script for producing data, with a .env/ directory also at the top level. The topics list includes ray and parallelism, and the project is a PyTorch implementation with differential-privacy topics in scope.

### Is FL-bench still being worked on?

The repository is not archived and the last push was on 2026-10-20. The project states that FL-bench welcomes pull requests on everything that can make the project better, and it carries a CITATION.cff file for citing results produced with it.

## Sources

- [Issues](https://github.com/KarhouTam/FL-bench/issues)
- [KarhouTam/FL-bench on GitHub](https://github.com/KarhouTam/FL-bench)
- [License: GPL-3.0](https://github.com/KarhouTam/FL-bench/blob/master/LICENSE)
- [README](https://github.com/KarhouTam/FL-bench/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/karhoutam-fl-bench