SketchCode: A 2018 Image-Captioning Model That Turns Wireframe PNGs Into HTML

The problem SketchCode was built to attack in 2018

The README frames the project as an experiment from the Insight AI Fellowship: take a hand-drawn website wireframe and emit working HTML. The bet was architectural rather than product-driven. Image captioning was the technique of the moment, a CNN encoder feeding an LSTM decoder, and the authors treated HTML as the caption. The README states plainly that this worked on wireframes close to the dataset the model was trained on. That sentence is the whole scope of the project. It is aimed at people who want to read or reproduce a specific research idea, not at teams looking for a design-to-code tool. The repository also credits its lineage: the synthetically generated dataset and model architecture come from pix2code by Tony Beltramelli, and the Design Mockups project by Emil Wallner is named as a second influence.

Encoder, decoder, and where the HTML actually comes from

The architecture is image captioning with the output vocabulary swapped for markup. A convolutional network encodes the input PNG into a feature representation, and a recurrent decoder produces the HTML sequence one token at a time. The README describes this directly: the model uses an image captioning architecture to generate HTML markup from hand-drawn wireframes. Two artefacts define a trained model, and both are passed explicitly on the command line: a model_json.json holding the architecture and a weights.h5 holding the parameters. There is no bundled inference server, no checkpoint registry, and no configuration file. Every conversion run names its own JSON and H5 files, which means you can keep several trained variants side by side and select one per invocation. Evaluation is separate from generation and uses BLEU, with dedicated scripts for a single GUI prediction and for a batch of them.

Getting the data, the weights, and one conversion out the door

Setup assumes Python 3 and pip, and the README states the project is not compatible with Python 2. Dependencies install with pip install -r requirements.txt. Data and pretrained weights are fetched by two shell scripts rather than by a package manager: sh get_data.sh and sh get_pretrained_model.sh, run from the scripts directory after cloning the repository. The README gives the dataset size as 1,700 images totalling 342mb. A single conversion looks like this, run from src: python convert_single_image.py --png_path ../examples/drawn_example1.png --output_folder ./generated_html --model_json_file ../bin/model_json.json --model_weights_file ../bin/weights.h5. Batch conversion substitutes convert_batch_of_images.py and takes --pngs_path instead of --png_path. Training uses train.py with --data_input_path, --validation_split, --epochs, --model_output_path and --augment_training_data; the README notes that the last flag adds Keras ImageDataGenerator augmentation for training images. Passing --model_json_file and --model_weights_file to train.py resumes from a pretrained checkpoint instead of starting from scratch.

Augmentation is the only defence against a small dataset

The training set is synthetic and small, and the README's own note says the model is not built to generalise to the variability of sketches seen in actual wireframes, with performance depending on wireframes resembling the core dataset. The response in the code is augmentation. Setting --augment_training_data 1 wraps the training images in Keras ImageDataGenerator, which perturbs them during training. That is a reasonable move for a dataset of this size, but it is bounded. Augmentation perturbs existing drawings; it does not invent new layout conventions, new component types, or the messier line quality of a sketch made by a person on paper. If your inputs come from a whiteboard photo, a tablet, or a designer's notebook, the gap between those and the synthetic training distribution is exactly the gap the README warns about. Treat the pretrained weights as a demonstration of the pipeline, not as a model you can point at arbitrary drawings.

TensorFlow 1.1.0 is the real adoption cost

The badge in the README pins TensorFlow 1.1.0, and the surrounding text says the code targets TensorFlow 1.x and Keras of that era and is no longer maintained. That single constraint dominates every practical decision. TensorFlow 1.x graph-mode code does not run unmodified on TensorFlow 2.x, and the Keras API surface has moved since. There are no recent releases to fall back on; the repository has no release artefacts at all. So the upgrade path is not a version bump, it is a port, and the README offers no migration notes. If you only want to run inference on the published weights, you still need an environment where that old stack installs cleanly, which on current Python versions is not something the documentation addresses. Budget for environment archaeology before you budget for anything else.

When a general vision-language model is the better answer

The README itself sets up the comparison: SketchCode stands as a record of what this problem took before general-purpose vision-language models started doing it zero-shot. The difference in approach is the point. SketchCode trains a task-specific encoder-decoder on a fixed synthetic corpus and can only emit markup in the vocabulary and layout patterns it saw during training. A general vision-language model is prompted rather than trained, needs no dataset of paired wireframes and HTML, and is not locked to a particular markup dialect. The trade is the other way too: SketchCode's output distribution is narrow and predictable, which is exactly what you want if your wireframes match the training set, and exactly what fails when they do not. If your goal is a working design-to-code feature today, the zero-shot route is the one the project's own retrospective points to. SketchCode remains useful when you want to study the captioning formulation itself.

Licence: MIT on the surface, Apache-2.0 underneath

The repository is MIT licensed, and the README points to the LICENSE file. It also states that portions derive from pix2code, which is licensed under the Apache License, Version 2.0, and directs readers to NOTICE for the required attributions and the terms that apply to those portions. That is a two-licence situation, not a single one, and the NOTICE file is the document that tells you which parts carry which terms. If you copy code out of this repository into another project, the MIT terms cover the repository's own code while the pix2code-derived portions carry Apache-2.0 obligations. Read both files before redistributing. This is a description of what the repository states, not legal advice.

What to verify before you clone it

Three things are checkable without training anything. First, whether the pinned TensorFlow 1.1.0 and the Keras version in requirements.txt install in your environment; the README does not describe a modern alternative. Second, whether your wireframes resemble the synthetic dataset; the README's generalisation note is the deciding factor, and the examples directory gives you one drawn sample to compare against. Third, whether the pretrained JSON and H5 pair downloaded by get_pretrained_model.sh matches the file paths the conversion command expects, since both are passed explicitly and a mismatch fails at load time rather than at setup. If all three check out and you accept a frozen 2018 codebase, the pipeline is short enough to read end to end. If any of them fails, the general-purpose route the README itself names is the more direct path to working HTML.

Editorial conclusion

Adopt SketchCode only as a reference implementation or a teaching artefact: it is an unmaintained TensorFlow 1.x codebase whose own README states the model is not built to generalise beyond wireframes resembling its 1,700-image synthetic dataset, so it is the wrong tool for production front-end generation. Before running anything, check that your environment can install the pinned TensorFlow 1.1.0 and the matching Keras from requirements.txt, and read NOTICE alongside LICENSE so the Apache-2.0 portions inherited from pix2code are handled correctly.

Official sources

1. ashnkumar/sketch-code on GitHub
2. Issues
3. License: MIT
4. README