BGSLibrary: 43 Background Subtraction Algorithms Behind One C++ Factory

The problem BGSLibrary actually solves

Background subtraction is a crowded research area. Papers publish a method, release a fragment of code, and move on. If you need to compare five of those methods on your own footage, you normally spend a week gluing together incompatible implementations, each with its own frame format, its own parameter names, and its own idea of what a background model is.

BGSLibrary exists to remove that glue work. It is a C++ framework, started in 2012 by Andrews Cordolino Sobral, that registers background subtraction algorithms behind one interface and exposes that interface through wrappers for Python, Java and MATLAB. The intended user is a researcher or engineer who wants to run several detectors over the same video and compare the resulting foreground masks, not someone who has already decided on a method and wants the smallest possible dependency.

The README frames it plainly: the library is "designed for background subtraction in computer vision applications, particularly for detecting moving objects in video streams" and provides "an easy-to-use and extensible platform for researchers and developers to experiment with." That word, experiment, sets the expectation correctly. This is a comparison bench, not a production detector.

How the algorithm factory and the OpenCV version constraint work

The architectural center of the library is a factory. The C++ example in the README calls BGS_Factory::Instance()->GetRegisteredAlgorithmsName(), which returns a list of algorithm names as strings, and the example prints the count and then the names. Every algorithm is registered with that factory, and the wrappers call into the same registry. In Python, that registry is what makes bgs.FrameDifference() resolvable as an attribute on the pybgs module.

This design has a direct consequence that the README treats as important enough to flag in a callout. The set of algorithms available to you is fixed at compile time and depends on the OpenCV version the pybgs binding was compiled against, not on the version of opencv-python you have installed in the same environment. On OpenCV 4 and above, the DP and T2F families are unavailable, which the README quantifies as roughly 26 algorithms instead of about 39 on OpenCV 2 or 3. The README's instruction is explicit: gate on the binding with a check like if hasattr(bgs, "DPAdaptiveMedian"), never on cv2.__version__.

That is a real design trade-off rather than an oversight. Several classic background subtraction methods depend on OpenCV APIs that were removed or relocated in the 4.x line. Rather than reimplement them, the library drops them and tells you to check at runtime. The cost is that two developers on the same machine with different OpenCV builds can get different algorithm lists from the same pip package name.

Installing pybgs and running your first foreground mask

The README offers two routes: a pre-built binary package from the releases page, or a source build. For Python users it calls the pip route the quickest.

The published package installs under two names, pybgs and bgslibrary, but the importable module is always pybgs.

pip install pybgs

The README is direct about what that command does: it builds from source, so you need a C++ compiler and an OpenCV development install on the machine. On Ubuntu that is libopencv-dev, on macOS it is brew install opencv. If you do not have OpenCV, the README points you at the Pixi build instead, which it labels recommended and which provides OpenCV for you. The pyproject.toml explains why numpy sits in the build requirements: the CMake build detects the NumPy include directory by running python -c "import numpy; print(numpy.get_include())", and under pip's build isolation the build environment contains only the packages listed there.

If you clone the repository instead, the README warns that pybind11 is bundled as a git submodule, so the clone has to be recursive.

git clone --recursive https://github.com/andrewssobral/bgslibrary.git

If you already cloned without that flag, the recovery command is git submodule update --init --recursive. The README notes that this does not affect the pip path, since the published PyPI package bundles pybind11 already.

A first real run looks like the README's minimal example. It opens a video, applies the algorithm frame by frame, and displays three windows: the input frame, the foreground mask, and the current background model.

import cv2
import pybgs as bgs

algorithm = bgs.FrameDifference()
capture = cv2.VideoCapture("dataset/video.avi")
while True:
    ok, frame = capture.read()
    if not ok:
        break
    fg_mask = algorithm.apply(frame)
    bg_model = algorithm.getBackgroundModel()
    cv2.imshow("foreground", fg_mask)
    if cv2.waitKey(10) & 0xFF == 27:
        break
cv2.destroyAllWindows()

Two details matter here. apply() returns a single-channel CV_8UC1 mask, and getBackgroundModel() returns the model the algorithm currently holds, which is what makes side-by-side comparison of methods practical. The README keeps the 0xFF mask on the waitKey result because the return value is wider than 8 bits on 64-bit platforms. The repository also ships demo.py and demo2.py, plus a separate Python examples repository, which are more useful starting points than the snippet once you want to swap algorithms.

Where BGSLibrary stops being the right tool

The library is a collection, and collections do not make decisions for you. The README links a wiki page titled "Which algorithms really matter?", which is a telling admission: if the answer were obvious from the code, the page would not need to exist. You should expect to spend real time on the algorithms benchmark wiki page and on your own footage before you have a detector you trust.

The OpenCV 4 gap is the sharpest limitation. If your pipeline is built on a modern OpenCV, you are working with the smaller set, and any tutorial or paper that names a DP or T2F method will not map onto your install. The README's hasattr check is the correct guard, but it also means algorithm selection becomes a runtime concern in code that would otherwise be static.

The build story is the second limitation. pip install pybgs compiling from source with a system OpenCV dependency is a heavier requirement than most Python packages, and it is the kind of thing that breaks in slim containers, on CI images without a compiler, and on machines where the OpenCV development headers are missing or mismatched. The Pixi route exists precisely because that friction is common.

Finally, consider what the library does not include. The README describes background subtraction and foreground detection. It does not describe tracking, object identity, or classification of what the mask contains. If your goal is "tell me which car this is", BGSLibrary gets you a binary mask and stops there.

BGSLibrary against OpenCV's own background subtractors

The obvious alternative is OpenCV itself. OpenCV ships a small set of background subtractors, most prominently the MOG2 and KNN families, exposed through cv2.createBackgroundSubtractorMOG2 and its relatives. If you already depend on OpenCV, that route adds nothing to your dependency graph, and the maintainers of OpenCV keep those classes aligned with the rest of the library.

The difference is breadth against integration. OpenCV gives you a handful of methods that are maintained alongside the core library. BGSLibrary gives you 43 registered algorithms, including the classic statistical and fuzzy approaches that OpenCV never shipped, and it normalizes all of them behind apply() and getBackgroundModel(). If your question is "which of these methods handles my scene best", OpenCV cannot answer it and BGSLibrary can. If your question is "what is the least code I can write to get a mask", OpenCV wins on every axis.

There is a second, subtler difference. Because BGSLibrary delegates to OpenCV for the algorithms that OpenCV provides, its OpenCV 4 build is not simply a smaller library, it is a library whose contents shifted with the dependency. OpenCV's own subtractors do not have that property.

Licence, maintenance and the cost of upgrading

The source is MIT licensed, which the README describes as free for both academic and commercial use. That is permissive and imposes no copyleft obligation on your own code. The repository also bundles pybind11 as a submodule, and pybind11 carries its own licence, so a redistribution that includes the bundled copy should account for it. This is a description of what the repository states, not legal advice.

The repository is not archived, and the last push was on 2026-05-28. The most recent tagged release listed is v3.3.0 from 2024-07-13, while the README advertises library version 3.3.1 and links to a build status wiki page, so the README's version string and the release list do not line up exactly. Worth checking the wiki release notes before you pin a version.

Upgrade cost is dominated by the OpenCV coupling rather than by the library's own API. The C++ surface shown in the README is small: a factory call and an algorithm object. The Python surface is two methods, apply and getBackgroundModel, plus the algorithm constructor. That is a stable-looking interface. What changes underneath is which algorithms exist, and that is governed by the OpenCV you build against. Moving from OpenCV 3 to 4 can remove algorithms from your available set without any change to BGSLibrary's own version number. Pin both, and treat an OpenCV bump as an algorithm-availability change, not just a dependency refresh.

Editorial conclusion

Adopt BGSLibrary if you are evaluating background subtraction methods and want to switch between them without rewriting your pipeline, or if you need a C++ core with Python and MATLAB bindings. Do not adopt it if you expect a single tuned detector: the library hands you a menu, not a recommendation, and the wiki page on which algorithms matter is the closest thing to guidance. Before committing, verify three things: that your target algorithm is present in the binding you installed, that the algorithm you pick is not in the DP or T2F families if you are on OpenCV 4, and that you can build the Python wrapper from source with an OpenCV development install, because pip install pybgs compiles rather than downloads a wheel.

Frequently asked questions

Official sources

1. andrewssobral/bgslibrary on GitHub
2. Issues
3. License: MIT
4. README
5. Releases