Squirrel: A C++ Framework for Action Recognition and Temporal Detection on Linux

What Squirrel Actually Solves

Squirrel addresses a specific research niche: training and evaluating neural networks for action recognition and temporal action detection on video data. The framework is not a general-purpose deep learning library. It is a C++ codebase that implements the methods from four papers by Alexander Richard and colleagues, covering weakly supervised action learning, bag-of-words equivalent RNNs, and statistical language models for temporal detection. The intended user is a graduate student or researcher who wants to reproduce these results or extend them. The README makes this clear by asking users to cite the papers. This is not a tool for practitioners who want a quick deployment path. It is a research artifact, and the design choices reflect that origin.

Architecture and Data Flow as Seen in the Repository

The framework is built around C++ modules, with a visible separation between network types, math routines, and data handling. The README mentions feed-forward networks, CNNs, and recurrent neural networks as supported architectures. The actual data flow is not documented in the README, but the structure suggests that video frames are read through OpenCV, processed through network layers defined in C++, and trained using CUDA for GPU acceleration. The presence of Modules.hh and definitions.make implies a modular build system where you can enable or disable CUDA, OpenMP, and OpenCV. The math layer includes Blas.hh and Lapack.hh, indicating that linear algebra operations are central to the implementation. The framework borrows matrix and vector classes from the RWTH speech recognition toolkit RASR/NN, which shows a lineage from speech processing. The temporal action detection method likely uses a statistical language model over frame-level predictions, but the README does not specify the exact flow. That gap is a real limitation for anyone trying to understand the system before compiling it.

Installation: A Manual, Legacy-Aware Process

Getting Squirrel running requires manual configuration. The README states that you must modify definitions.make in the src directory to point to your MKL and CUDA libraries. CUDA 8.0 is the minimum version, which is a hard constraint: modern GPUs and drivers may not support it. OpenCV 2.x is required for reading images, and Intel MKL is used for parallelization. If you want to disable CUDA, OpenMP, or OpenCV, you comment out the corresponding MODULE lines in definitions.make and the matching lines in Modules.hh. For a non-MKL BLAS implementation, you would need to edit includes in src/Math/Blas.hh and src/Math/Lapack.hh, plus the library paths in definitions.make. The compile command is simply make from the src directory. This is a build process that assumes you know your system's library paths and are comfortable editing makefiles. There is no CMake, no package manager integration, and no prebuilt binaries. The README does not mention any test suite or smoke test after compilation, so you are on your own to verify that the build works.

The Data Format Problem

The README says, and I quote, "Please inspect the examples or see the manual for information on how to use the framework and to determine the correct data format." That is the only guidance on data. There is no explicit description of the input format, such as whether frames are stored as image sequences, video files, or pre-extracted features. This is a serious barrier for adoption. A researcher who wants to run the framework on their own dataset must reverse-engineer the format from example code, which the README does not show. The manual is mentioned but not linked or described. This lack of documentation is a genuine failure mode: you could spend days just figuring out how to feed data into the network. If you are evaluating Squirrel, plan to read the source code and examples carefully. The framework is not suitable for anyone who wants a plug-and-play data loader.

Constraints and Failure Modes

Squirrel has several hard constraints that make it the wrong tool in many scenarios. First, it is Linux-only, which excludes Windows and macOS developers unless they use virtual machines or containers. Second, the dependency on CUDA 8.0 is a time capsule: that version is over a decade old, and current NVIDIA drivers often drop support for it. If you have a modern GPU, you may not be able to use CUDA at all. Third, OpenCV 2.x is ancient, and most distributions now ship OpenCV 4.x, so you will likely need to compile OpenCV 2.x from source. Fourth, the license is listed as NOASSERTION, which means the repository does not declare a standard open source license. This is a legal red flag for any commercial or even academic use beyond personal research. The README asks for citation, but does not grant explicit permissions. You should contact the author before using it in a project with legal requirements. Finally, the framework is not maintained: the last push is unknown, and there are no recent releases, so bugs and compatibility issues will not be fixed.

Alternatives: What Else Exists

The obvious alternative is a modern deep learning framework like PyTorch or TensorFlow, combined with a video understanding library such as MMAction2 or SlowFast. These tools offer prebuilt models for action recognition and temporal detection, active maintenance, and extensive documentation. The key difference is approach: Squirrel is a monolithic C++ framework with a custom build system and a focus on reproducing specific papers, while PyTorch-based tools are modular, Python-first, and support rapid prototyping. For a researcher who wants to implement a new idea, PyTorch offers autograd, a huge model zoo, and community support. For reproducing the exact methods in Richard's papers, Squirrel may be closer to the original code, but the effort to set it up could outweigh the benefit. Another alternative is to implement the methods directly in a modern framework using the papers as reference, which gives you control over dependencies and data pipelines. That is often more practical than wrestling with a legacy C++ codebase.

Maintenance and Upgrade Cost

The repository shows no signs of active maintenance: no recent releases, no pushed changes, and no issue tracker mentioned in the README. The last push is unknown, which suggests the project may be dormant. The dependencies are frozen in time: CUDA 8.0, OpenCV 2.x, and MKL are all legacy versions. Upgrading any of these would require modifying the build system and likely the source code, since the framework may use APIs that have changed. The license being NOASSERTION adds uncertainty: you cannot assume you have the right to modify and redistribute the code. The README asks for citation, which is a scholarly norm, not a legal grant. If you adopt Squirrel, you should treat it as a reference implementation, not a maintained library. The upgrade cost for a new user is high: you must build the entire dependency stack from old versions, and any bug you find is yours to fix. There is no community forum or support channel beyond the author's email, which may or may not be monitored.

Editorial conclusion

Adopt Squirrel if you are a researcher needing to reproduce the cited papers on temporal action detection and weakly supervised learning, and you are comfortable with legacy C++ builds, CUDA 8.0, OpenCV 2.x, and MKL. Do not adopt it for production deployment or if you expect modern Python tooling or active maintenance. Before committing, verify that your GPU and driver support CUDA 8.0, that you can locate compatible OpenCV 2.x and MKL versions, and that the examples match your data format, since the README gives no explicit data schema. The project appears frozen, so budget for porting effort if you need newer libraries.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. README
4. Releases