contextualbandits is a research toolbox that assumes binary rewards

The problem as the package states it

The README opens with a definition that is more precise than the marketing phrase, and it is worth reading closely before choosing this library. Contextual bandits, also known as multi-armed bandits with covariates or associative reinforcement learning, is a problem similar to multi-armed bandits, with the difference that side information or covariates are available at each iteration and can be used to select an arm, whose rewards are also dependent on the covariates. The data generating process is described as iterative. At each round, the world creates an observation consisting of a set of covariates of fixed dimension, and a reward for each arm, which is stochastic but dependent on the covariates. The agent then chooses an arm, and the world reveals the reward for the chosen arm but not for the others. The aim is a policy that maximises the rewards obtained, and the same exploration-exploitation dilemma applies, with arms that may expire and new arms appearing over time. The package also notes the resemblance to multi-class or multi-label classification, with one crucial difference: the right label is not known for each observation, only whether the label the agent chose was correct.

Binary rewards and identical covariates, stated up front

Two restrictions define the package and are easy to skim past. The README says that while algorithms for the contextual bandits problem generally assume continuous rewards in the range [0,1], this package deals only with the case of discrete rewards {0,1}, and only with the case of arms that all see the same covariates. Some methods might still work with continuous rewards, but they are not meant to be used with them. That second restriction has an important consequence. If your arms carry their own feature vectors, which is the usual setup for a multi-armed bandit library where each item has its own attributes, this package is the wrong shape. The two together also mean that the binary reward setting is the natural fit for click-through style problems, where the reward is whether the chosen label was correct, and a poor fit for anything where the payoff is a measured value. The package is explicit about what it leaves out as well, and the list of exclusions is short and useful. It does not deal with incorporating social networks and similarity information between observations and arms, for which it points to BanditLib, and it does not cover bandits with expert advise, such as Exp4 and OnlineCover, or streaming clustering used for defining similarity buckets.

Three modules for the three hard problems

The package structure maps directly onto the three problematics the README names. Building a policy or strategy for making choices in an online setting, managing exploration of new arms under different features against exploitation of arms known to be good for some features, is covered in contextualbandits.online. Building a policy or algorithm with data collected from a different policy is covered in contextualbandits.offpolicy, also known as counter-factual risk minimization. Evaluating the performance of different strategies and algorithms based on partially-labelled data is covered in contextualbandits.evaluation. That structure is the main thing this library offers over assembling algorithms yourself, since every method in a module is constructed and called the same way, which is what makes comparison meaningful. The README also points out that most of the algorithms are meta-heuristics that take a binary classifier, such as Logistic Regression or XGBoost, as a black-box oracle, so the learning model is yours to choose and the bandit machinery wraps it.

Installing it means having a compiler

The installation section leads with the constraint that catches people out. The package requires C and C++ compilers configured for Python, with a link to a guide for setting that up. The package is on PyPI and installs with:

pip install contextualbandits

If that fails, the README gives a fallback that skips the PEP 517 build path:

pip install --no-use-pep517 contextualbandits

Fedora users have a distribution package:

dnf install python3-contextualbandits

The reason a compiler is needed is stated in a note. As of version 0.2.0 the package contains Cython code which needs to be compiled, which means it requires a C compiler and Python's setuptools configured to use it. The note also offers an escape hatch for people whose environment fights the build. An earlier version that was pure Python can still be installed with pip install contextualbandits==0.1.8.5, and the same code is available in the GitHub repository under the branch no_cython. The build requirements in pyproject.toml match the story, with cython at 3.1.0 or newer, setuptools, wheel, numpy at 2.0.0 or newer and scipy in the build system list, while requirements.txt adds pandas, scikit-learn and joblib.

The -march=native flag will surprise you

There is a warning block in the README that deserves to be read before any pip install, and it is the kind of detail that causes production incidents. The setup script will try to add the compilation flag -march=native, which instructs the compiler to tune the package for the CPU in which it is being installed, for example by using AVX instructions if available, but the result might not be usable in other computers. If you are building a binary wheel or putting the package into a docker image that will run on different machines, that is a problem. There are two documented ways to override it, and both are environment variables. You can define DONT_SET_MARCH=1, or you can supply compilation CFLAGS yourself:

export CFLAGS="-march=x86-64"
export CXXFLAGS="-march=x86-64"
pip install contextualbandits

The README notes that for maximum compatibility, at the cost of the slowest speed, the same can be done by setting DONT_SET_MARCH=1 before installing. The setup.py source confirms the mechanism, with a check for architecture flags already present in the environment and an add_march_native call when neither a march native flag nor a CFLAGS architecture flag is found, alongside -O3 optimisation and OpenMP linkage for the C++ extensions, or /O2 and /openmp for MSVC.

What has been added to the online policies lately

The Recent additions section is a good map of the current capabilities and, incidentally, of the direction of travel. There is an output_all_scores parameter on the predict() method of some online policies, which returns the full score matrix used for the choices of arms during the same predict() call, so you can inspect what a policy considered rather than only what it chose. Per-arm smoothing and beta_prior hyperparameters can now be passed, which matters when arms have very different update behaviour. Sparse matrices in CSR format are supported, so large feature spaces no longer require densifying. There is functionality for ranking the top-N arms instead of always picking the single best one. A tree-based partitioned UCB and TS were added, along with a new online method called ParametricTS. ExploreFirst gained an option to make choices using active learning, which is a different exploration strategy from random or optimistic initialisation. The example notebook for the online case was also updated for better results with both full-refit and stream settings. Between them, these changes move the package from a paper reproduction set towards something you can build a prototype service on.

Where the learning happens: notebooks, docs and a paper

The getting started material is in three IPython notebooks that use public datasets, one each for online contextual bandits, off-policy learning in contextual bandits, and policy evaluation in contextual bandits. That is a deliberate choice for a research package, since the interesting comparisons need data rather than toy loops. The package documentation is published on readthedocs at contextual-bandits.readthedocs.io, and the README notes that the same information is available internally through docstrings, with examples such as help(contextualbandits.online.BootstrappedUCB) and help(contextualbandits.online.BootstrappedUCB.fit). The citation path matters if this ends up in a paper, because the README points to the arXiv paper Adapting multi-armed bandits policies to contextual bandits scenarios for details about the implementations and for how to cite them. The licence is BSD 2-clause, with the LICENSE file at the root of the repository. The repository layout follows the usual research package shape, with the contextualbandits/ package directory, docs/, example/, plots/, tests/, a setup.py alongside pyproject.toml and setup.cfg, and a MANIFEST file.

Editorial conclusion

contextualbandits suits a researcher who needs to prototype a contextual bandit idea quickly, reproduce a paper that shipped no usable code, or compare several policies on partially labelled data with a consistent API. It is a poor fit for a production recommender with continuous rewards, different covariates per arm, or a deployment requirement, since the package states it deals only with the case of discrete rewards in the set {0, 1} and only with arms that all see the same covariates. Verify first that your problem matches that scope, install with DONT_SET_MARCH=1 if the artifact has to run on more than the build machine, and read the paper linked from the README before citing the implementation.

Frequently asked questions

Official sources

1. david-cortes/contextualbandits on GitHub
2. Issues
3. License: BSD-2-Clause
4. Project website
5. README