CNNGestureRecognizer: a five-class hand gesture CNN that ships with its own training images

The problem it solves: a webcam gesture pipeline you can read in two files

Most gesture recognition material online is either a notebook that classifies a static dataset or a closed demo with no training path. This repository sits in between. It captures frames from a webcam with OpenCV, preprocesses them into a shape the network can consume, runs a small convolutional network, and prints or writes the predicted class. The two files that matter are trackgesture.py, which holds the UI options and the camera capture code, and gestureCNN.py, which holds the model definition, the weight loading, the training loop and the feature-map visualisation. The README describes trackgesture.py as calling into gestureCNN.py, so the split is deliberate: camera and UI on one side, network on the other.

The audience is narrow and worth naming. It is for someone learning how a CNN is wired to a live camera, or for someone who wants a five-class gesture switch for a demo and is willing to retrain it. It ships with 4015 gesture images in imgfolder_b, taken by the author, which means you can train without collecting data first. The five pretrained classes are OK, PEACE, STOP, PUNCH and NOTHING, where NOTHING covers any input that is not one of the other four. That fifth class is the part most hobby projects skip, and it is the reason the demo does not simply return a confident wrong answer for an empty frame.

Two capture modes and the preprocessing that decides whether it works

Recognition quality here is mostly a preprocessing question, and the README is explicit that there are two modes with different assumptions. Binary Mode converts the region of interest to grayscale, applies a Gaussian blur with a 5x5 kernel and sigma 2, then an adaptive threshold using ADAPTIVE_THRESH_GAUSSIAN_C with THRESH_BINARY_INV, block size 11 and constant 2, and finally an Otsu threshold pass. The README states this mode is useful when you have an empty background such as a wall or a whiteboard. The code sample shows the exact call chain, so you can see that the Otsu step is applied on top of the adaptive result rather than instead of it.

SkinMask Mode takes a different route. It converts to HSV, applies cv2.inRange with a low and upper skin range, erodes once and dilates once with a skin kernel, blurs with a 15x15 Gaussian and sigma 1, then uses the mask in cv2.bitwise_and against the original frame and converts the result to grayscale. The README says this mode is useful when there is good light and no empty background. The trade-off is direct: SkinMask depends on a fixed HSV range, so it will behave differently under different lighting and for different skin tones, and the README does not say how the range was chosen or whether it was tuned across users. Binary Mode depends on contrast between hand and background, which is why the empty-background requirement is stated. The release notes for 2.0.0 call the new background subtraction filter the best performing filter for this project, which tells you the author kept iterating on this stage rather than on the network.

The network is a small MNIST-style stack, and that is the honest description

The README does not oversell the architecture. It says the CNN is a common model found across tutorials, mostly seen in digit classification on MNIST. The visible definition starts with a Sequential model, a Conv2D layer with nb_filters filters and an nb_conv by nb_conv kernel, padding set to valid, and an input shape of (img_channels, img_rows, img_cols). A ReLU activation follows, then a second Conv2D with the same filter count, another ReLU, a MaxPooling2D with an nb_pool by nb_pool pool, and Dropout at 0.5 before a Flatten. The README excerpt ends there, so the dense layers are not shown in the material available.

What matters for adoption is that all the dimensional parameters are named variables rather than literals. That is what makes the New Training feature meaningful: you can change the architecture without rewriting the graph. The cost is that the model is small and shallow by current standards, and there is no augmentation, no transfer learning and no mention of normalisation beyond the thresholding and masking done in OpenCV. The network sees whatever the preprocessing produces, so a change to the threshold block size or the HSV range is effectively a change to the model's input distribution. If you retrain, retrain with the same capture mode you will run at inference.

Getting it running: pinned versions, a backend variable and a 150 MB download

The README lists Python 3.6.1, OpenCV 3.4.1, Keras 2.0.2, TensorFlow 1.2.1 and Theano 0.9.0, and labels Theano obsolete and not supported any further. It recommends Anaconda for managing these versions and for keeping separate virtual workspaces, which is sensible given how far the pinned versions are from current releases. The launch command differs by platform. On Mac the README gives:

KERAS_BACKEND=tensorflow python trackgesture.py

On Windows it gives two lines:

set "KERAS_BACKEND=tensorflow" python trackgesture.py

The README notes that setting KERAS_BACKEND switches the backend, and that if you have already set it in keras.json you do not need to do this. Note the inconsistency in the README text: the example comment says it is setting the backend to Theano while the command sets tensorflow. Trust the command.

The weights are not in the repository. The README says the two pretrained files, _pretrained_weights_MacOS.hdf5 and _pretrained_weights_WinOS.hdf5, are each about 150 MB and are hosted on separate Google Drive links. It also notes that ori_4015imgs_weights.hdf5 was replaced by these two OS-specific files, which suggests the weight format or the saved graph differs between platforms. Download the one matching your OS before expecting Prediction to work, and check the links are still live, because a Drive link is not a package registry.

Prediction, retraining and feature-map visualisation are three separate workflows

The application exposes three modes. Prediction classifies the current gesture against the pretrained classes and can dump the result to the console or to a JSON file. The README points at a separate repository, asingh33/LivePlot, for turning that JSON into a real-time bar chart, so the plotting is not part of this project. New Training retrains the network and, per the README, includes inbuilt options to capture new image samples for user-defined gestures, which is the path to adding a sixth class or replacing the five. Visualisation renders feature maps from different layers of a pretrained model for an image in the imgs folder.

The visualisation mode is the most under-documented of the three. The README says gestureCNN.py can visualise feature maps at different layers for a given input image present in ./imgs, and that imgs holds a few samples taken from imgfolder_b, but it does not describe the output format, whether it writes files or opens windows, or which layers are exposed. If you are evaluating this project for teaching, that gap is worth checking in the source before you build a lesson around it. The 2.0.0 release notes also mention an in-app graph plotting feature for observing prediction probabilities, which overlaps in purpose with the JSON plus LivePlot route and is not described further.

Where it breaks: static poses, pinned dependencies and a self-collected dataset

The clearest limitation is scope. The model recognises five gestures and the README does not claim temporal modelling, so there is no evidence it handles motion, gesture sequences or transitions. If your gesture is defined by movement rather than a held pose, this is the wrong tool and no amount of retraining fixes it, because the input to the network is a single preprocessed frame.

The second limitation is the environment. Python 3.6.1, Keras 2.0.2 and TensorFlow 1.2.1 are all old, and TensorFlow 1.x code does not run unchanged on TensorFlow 2.x. The README's own suggestion to use Anaconda and separate virtual workspaces is an admission that this does not install cleanly alongside modern stacks. Theano is listed as obsolete in the requirements, so the multi-backend claim in the description is historical rather than current.

The third is the dataset. The 4015 images were taken by one person, and the README does not report how many subjects, what lighting conditions or what camera were used. The accuracy and loss plots, ori_4015imgs_acc.png and ori_4015imgs_loss.png, are described as plots of training against validation, but the README gives no numeric results and no held-out test protocol. Validation drawn from the same capture session as training will look better than performance on a new user, so treat the plots as a sanity check, not as a generalisation claim.

The alternative to compare against: MediaPipe Hands

The obvious alternative is MediaPipe Hands, which is the reference point most people reach for now. The difference in approach is fundamental. This project learns a classifier end to end from raw preprocessed pixels, so the network has to discover hand shape from thresholded or skin-masked images, and it only knows the five classes it was trained on. MediaPipe Hands instead predicts 21 hand landmarks per frame with a pretrained model, and you classify gestures by writing rules over those landmark coordinates or by training a small classifier on the landmark vectors rather than on pixels.

That shift changes what you maintain. With landmarks, a new gesture is often a few geometric conditions on finger positions, and it works across backgrounds because the detector was trained on far more varied data than 4015 images. With this project, a new gesture means capturing samples, retraining and revalidating. The trade-off runs the other way too: landmark pipelines give you coordinates, not a probability distribution over your own classes, and if your gesture is a shape the landmark model does not localise well, a pixel classifier trained on your exact setup can beat it. This project also keeps everything in Keras and OpenCV with no extra runtime, which matters if you are reading the code to learn.

Licence, maintenance and what the commit history implies

The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the most permissive common option and it removes the licence question from the adoption decision. Note that the pretrained weights are hosted on Google Drive rather than distributed with the code, so if you redistribute a build that depends on them, check the terms attached to those files separately. This is a description of the licence text, not legal advice.

On maintenance, the signals are mixed and worth reading carefully. The last release is v2.0.0 from November 2019, roughly six and a half years before the last push recorded on the default branch in May 2026. The release history shows 1.2.9 and 1.3.0 on the same day in November 2017, then a two-year gap to 2.0.0. So the pattern is long quiet periods punctuated by a large update that adds a filter, a backend and performance work. The 2.0.0 notes claim no FPS drop when prediction mode is enabled, but the README gives no measurement method or hardware, so that claim is not verifiable from the material here.

Upgrade cost is dominated by the TensorFlow 1.x to 2.x gap and the Keras API changes that came with it. A port means rewriting the model construction, the weight loading and any layer-level calls used by the visualisation mode, and the two OS-specific weight files suggest the saved format is tied to the environment that produced it. Budget for retraining rather than for loading the existing weights into a ported model.

Editorial conclusion

Adopt it if you want a small, readable Keras and OpenCV pipeline to study or to fork for a fixed five-class demo, and if you can pin Python 3.6.1, Keras 2.0.2 and TensorFlow 1.2.1. Do not adopt it if you need hands that move, more than five classes out of the box, or an install that survives a modern pip resolver. Before you commit, confirm the pretrained weights download from the two Google Drive links in the README, since the repository itself only carries the 4015 training images and the scripts.

Official sources

1. asingh33/CNNGestureRecognizer on GitHub
2. Issues
3. License: MIT
4. README
5. Releases