cnn_for_captcha: A Python Toolkit That Treats CAPTCHA Recognition as Five Separate Problems

The problem it solves, and the two checks the README demands first

Most CAPTCHA recognition code on GitHub is one script for one puzzle type, with the assumptions left implicit. This repository does the opposite: it names five distinct families (fixed-length numeric or text, slider gap, click-text, rotation, and same-object) and gives each one its own file, so the reader can see that these are not variations of a single task. A fixed-length text CAPTCHA is a sequence labelling problem. A slider gap is a localisation problem. Rotation is either regression or a nearest-neighbour lookup. Treating them as one pipeline would be a mistake, and the layout makes that visible before you write any code.

The intended audience is someone with a labelled dataset and a GPU budget, not someone looking for a drop-in service. The README opens with two preconditions that are unusual to see stated so plainly. The first is to confirm whether the CAPTCHA can simply be avoided; if most flows can bypass it, the project is not worth the effort. The second is to check whether the CAPTCHA space is small enough to enumerate by brute force. Both are engineering judgements about whether to build a model at all, and both are cheaper than training one.

The repository also carries an unresolved question in its own header about what CAPTCHAs are still for, given that multimodal models both solve and generate them. That framing is honest about scope. Nothing in the code claims to be a general solution.

fixed_length_captcha.py: naming rules and a size constraint you cannot ignore

The fixed-length path is the most conventional piece. Training data goes into directories named in a config file, defaulting to fixed_length_captcha.json. Two input requirements are stated and both are hard constraints rather than suggestions: every image in a directory must be the same size, and filenames must follow the pattern captcha_text_serial.ext, with abce_012312.jpg given as the example. The label is therefore encoded in the filename, which means a mislabelled file silently becomes a mislabelled training example. There is no separate manifest to validate against.

Training is a single command, python fixed_length_captcha.py, with no arguments; everything is read from the JSON config. Inference is exposed through a Predictor class with three entry points. predict('xxx.jpg') reads a local file, predict_single_image_content(b'PNGxxxxx') takes raw bytes, and predict_remote_image('http://xxxxxx/xx.jpg', save_image_to_file='remote.jpg') fetches over HTTP and can optionally persist the download. That third method is the one to think about carefully: it puts network I/O inside a prediction call, so error handling for timeouts and non-image responses is your responsibility, not the library's.

The only accuracy figure in the README is conditional and should be read that way. It states that for the sample image shown, roughly 20,000 training images reach above 90 percent accuracy, and that results depend on training set size. There is no held-out benchmark, no per-character breakdown, and no statement about how accuracy degrades as the character set grows.

Two ways to find a slider gap, and why the OpenCV one is the honest default

slide_captcha.py offers two detection strategies with different cost profiles. The first is OpenCV template matching, invoked as slide_captcha.detect_displacement('image_slider.jpg', 'image_background.jpg'), taking the puzzle piece and the background as separate inputs. The README describes this as simple and easy to verify, and says it reaches a satisfactory result when combined with some rules. Those rules are not specified in the material, which means the practical accuracy depends on work you do yourself: thresholding the correlation surface, rejecting ambiguous peaks, and handling backgrounds where the gap region has low contrast.

The second strategy is a YOLOv5 detector. The repository ships 100 annotated images for this purpose and points at the upstream YOLOv5 custom-data training guide. The documented training invocation is python train.py --batch-size 4 --epochs 200 --img 344 --data displacement.yaml --weights '' --cfg yolov5s.yaml, with the README noting that batch size should follow available memory and epochs should follow observed results. At inference you load a checkpoint and call detector.detect_displacement('image.jpg', 344), where the second argument is the image resize basis.

That 344 value is worth pausing on. The README advises using the image width or height and lowering it if images are large. The training and inference resize values must agree, and nothing in the described API enforces that. A mismatch produces a detector that runs without error and predicts badly.

Click-text and rotation: the two places where the README is a design document

The click-text section does not ship a trained model. It lays out a two-stage approach: localise candidate characters with a YOLO-style detector, then match them against the target, which may be given as text or as an image. The matching stage is where the README admits difficulty. If the characters survive localisation well enough for OCR, the listed options are PaddleOCR, tesseract, or cnocr, and string comparison finishes the job. If the characters are distorted or bolded, OCR fails and three fallbacks are offered: a Siamese network trained to decide whether two crops are the same character, direct classification through YOLO when the candidate set is only a few hundred or few thousand classes, or rendering the target text to an image and comparing that way. This is a menu of options, not an implementation. Budget accordingly.

Rotation is more concrete. rotate_captcha.py frames the task as regression and uses ResNet50 for feature extraction. The README reports trying a 0/1 classifier (upright versus not) and finding it performed poorly because only one orientation is positive while N rotations are negative, producing a badly imbalanced training set. Regression is stated to work better. Angle classification by binning, for example twelve classes at 30-degree steps, is listed as untried, so there is no evidence either way in the material.

The brute-force alternative in section 4.1 deserves attention because it may be the better choice. If the site draws from a fixed image pool of around a hundred originals, you can label the upright versions by hand, generate rotated copies yourself, and match a target image by cosine distance over CNN features. The README suggests imagededup's find_duplicates for grouping the same source image across rotations. For a small pool, this avoids training entirely, and the README does not pretend the deep learning route is always superior.

Same-object detection, and the failure modes the author reports

sameobject_captcha.py uses YOLOv5 to detect objects in the puzzle image, then compares detected classes. The README asks for at least 200 annotated images and says more is better. If you annotate with labelme, labelme_json_to_yolov5_format.py converts the output to YOLOv5 format, which is a small but genuinely useful utility since that conversion is otherwise a recurring annoyance.

The reported result is specific and unflattering, which makes it credible. With 300 annotated images and 100 training epochs, the author states that visually similar classes are confused: h with r, C with G, U with a cylinder. The conclusion drawn is that more training data should reduce the error rate. That is a hypothesis, not a measured outcome, and the material contains no follow-up run confirming it. If your object set contains visually close pairs, expect this class of error and plan your annotation volume around it rather than assuming 200 images is sufficient.

Multimodal models as an alternative, with dates attached

Section 6 is the closest thing to a competing approach inside the repository itself, and it is dated. As of March 2026, the README states that multimodal models handle simple CAPTCHAs acceptably but still make mistakes on distorted text. The prompt given is a strict JSON-output OCR instruction that asks the model to return {"result": "result"} and an empty string when no character is found, with no explanation permitted.

Results are attributed to specific model versions and dates: Gemini 3.1 Pro on 2026-03-09 for simple and complex CAPTCHAs, Nano Banana 2 on 2026-03-19 for annotated standard fonts, and GPT 5.4 on 2026-03-09 for complex CAPTCHAs. Two papers are cited for background on general-purpose vision models as OCR tools. The README notes that an agent loop can retry after a failed attempt, which is a real architectural difference from a single-shot CNN: a classifier gives you one answer with a confidence score, while a model plus retry logic gives you a sequence of attempts whose cost scales with failure rate.

What is missing is any cost or latency comparison. Running a hosted multimodal model per CAPTCHA has a per-request price and a network round trip; a local ResNet50 or YOLOv5 checkpoint does not. The README does not quantify that trade-off, so you have to.

Getting it running: split_data.py, dependencies, and what is not packaged

There is no setup.py, no pyproject.toml, and no released version in the supplied material. You clone the repository and work inside it. The README points at requirements.txt and describes it as fairly complete, installable as needed, which is a warning that it likely includes more than any single script requires. TensorFlow, Keras, and PyTorch all appear in the topic list, and the code references keras.application models, OpenCV, and YOLOv5, so the dependency surface is wide and version-sensitive.

One utility is fully documented with arguments. To split a prepared image directory into training and validation sets at a 90/10 ratio, the command is python split_data.py all_image_dir train_image_dir validation_image_dir 0.9. The README states that the target directories must already exist, so create them first; the script will not do it for you. The fourth argument is the training proportion.

Beyond that, the operational guidance is about hardware. The README recommends pay-as-you-go GPU instances from cloud providers so that a short training run can use a large machine without a long-term commitment. That is consistent with the project's shape: you run a training job, keep a checkpoint, and run inference against it. There is no serving layer, no queue, and no batching API described.

On licensing, the repository is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant, with the usual requirements to retain notices and state changes. The YOLOv5 training instructions point at a separate repository with its own licence, and any pretrained weights you download carry terms set by their publisher. Confirm those separately; nothing here is legal advice.

Where this is the wrong tool

The clearest boundary is the one the README sets itself. If the CAPTCHA can be avoided in most flows, or if the answer space is small enough to enumerate, this repository is the wrong choice and the author says so before any code. Take that at face value.

The second boundary is maintenance. The repository has no releases, and the fixed-length, slider, and rotation scripts are single files driven by config and filenames rather than a versioned interface. Upgrading TensorFlow or PyTorch can break the fixed-length and rotation paths, and the YOLOv5 dependency is pinned to an external project whose own API has moved over time. If you need a library you can pin and upgrade on a schedule, this is a starting point to copy from, not a dependency to import.

The third boundary is data shape. The fixed-length script requires uniform image dimensions and labels encoded in filenames. If your images vary in size, or your labels live in a CSV, you are writing the adapter yourself. And if your target is a CAPTCHA family not in the five listed, the repository offers no general framework to extend; you would take the YOLOv5 recipes and the config conventions and build the rest.

Editorial conclusion

Adopt this repository if you already have a labelled image set and need a starting point for one specific CAPTCHA family, particularly the YOLOv5 slider or same-object variants, since those come with 100 and 300 annotated images respectively. Do not adopt it if you need an installable package with a stable API, or if the target CAPTCHA can be bypassed or brute-forced, which the README itself tells you to check first. Before committing, verify that your images match the naming rule (captcha_text_serial.jpg), that all images in a directory share one size, and that the JSON config keys in fixed_length_captcha.json line up with your directory layout.

Official sources

1. anexplore/cnn_for_captcha on GitHub
2. Issues
3. License: Apache-2.0
4. README