# im2markup: a neural decompiler that reads a formula image back into LaTeX

> im2markup is a Torch and Lua system from the Harvard NLP group that infers the LaTeX or HTML source behind a rendered image, trained on IM2LATEX-100K. It suits research and dataset work, but the GPU-only, do-it-yourself toolchain limits it to people with CUDA hardware and patience for manual dependency setup.

**harvardnlp/im2markup** — Neural model for converting Image-to-Markup (by Yuntian Deng yuntiandeng.com)

- Repository: https://github.com/harvardnlp/im2markup
- Website: https://im2markup.yuntiandeng.com
- Stars: 1,254 · Forks: 214
- Language: Lua
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/harvardnlp-im2markup

## Going backwards from a rendered page to its source markup

im2markup is a general purpose system that decompiles an image into presentational markup. The canonical example in the project documentation is a LaTeX formula rendered on a full page, and the target is the source text that would render that image again, tokenized and spaced out in the way the tokenizer produces. The audience splits in two. One group wants the IM2LATEX-100K dataset of paired images and labels, which the project hosts with a distribution guide and version history, and the same group wants the trained model that goes with it. The other group wants to study how an image to markup model works, since the code is a full training pipeline rather than a single inference script. The model was introduced in the Image-to-Markup research by Yuntian Deng and collaborators, and the project asks that any use of the dataset, a processed copy, or the model include their citation. A hosted interactive demo exists as well, so the model can be tried in a browser before any dependency is installed.

## A convolutional layout encoder paired with an attention decoder

The architecture is described in one sentence in the project documentation: a convolutional network handles text and layout recognition in tandem with an attention based neural machine translation system. That pairing is what makes the task tractable. A CNN can tell the decoder that a glyph is present and roughly where it sits, while the attention decoder decides what token to emit and where to point in the source image. The attention weights are not just an internal step, because they also give an alignment from the generated markup back to the original image, and the project renders that alignment in its interactive demo over the test set. Training is done in Torch, with the model code written in Lua against the `tds`, `class`, `nn`, `nngraph`, `cunn`, `cudnn` and `cutorch` libraries, and Python is used for preprocessing instead. The training entry point is `src/train.lua`, and the repository separates code into `src/` for the model and training scripts, `scripts/` for preprocessing, `data/` for samples, `third_party/` for vendored evaluation code and `site/` for the static documentation.

## Running the toy Math-to-LaTeX example end to end

The repository ships a small toy example so the pipeline can be walked through before touching the full dataset. The working directory is assumed to be `im2markup` throughout, and the data layout is fixed: a `data_base_dir` holding the images, a label path holding every label such as a LaTeX formula, and a `data_path` listing the samples as pairs of image name and label index, where the index is the line number of the label starting from zero. The first step crops the formula area out of the page images and groups images of similar size to make batching easier.

```bash
python scripts/preprocessing/preprocess_images.py --input-dir data/sample/images --output-dir data/sample/images_processed
```

Next the formulas are normalized with the KaTeX based parser.

```bash
python scripts/preprocessing/preprocess_formulas.py --mode normalize --input-file data/sample/formulas.lst --output-file data/sample/formulas.norm.lst
```

The documentation warns that this step prints error messages, because some formulas in the sample cannot be parsed. Train, validation and test lists are then produced, with a filter pass that drops large images from training and validation and skips formulas with too many tokens or with grammar errors.

```bash
python scripts/preprocessing/preprocess_filter.py --filter --image-dir data/sample/images_processed --label-path data/sample/formulas.norm.lst --data-path data/sample/train.lst --output-path data/sample/train_filter.lst 
```

The last preprocessing step builds the vocabulary from the training list, excluding any token that occurs once or fewer times.

```bash
python scripts/preprocessing/generate_latex_vocab.py --data-path data/sample/train_filter.lst --label-path data/sample/formulas.norm.lst --output-file data/sample/latex_vocab.txt
```

Training is then launched through Torch and its parameters can be listed with the help flag.

```bash
th src/train.lua -h
```

## Preprocessing quietly decides what the model actually sees

The four preprocessing scripts are where most of the real decisions live, and each one narrows the data in a way that changes results. Cropping removes the blank page area around the formula, and grouping by size is a batching convenience, so the pixels your model sees are not the pixels in the original dataset release. Normalization runs the formulas through a parser, which means the labels are rewritten into a canonical token form and that a subset of formulas is dropped silently, since unparseable ones only produce error messages rather than aborting the run. The filter pass then applies different rules per split: large images are excluded from training and validation, and formulas with too many tokens or with grammar errors are ignored everywhere, with the test list built through a no-filter variant of the same script. The vocabulary step makes a further cut by discarding rare tokens. The result is that the effective training set is a series of decisions layered on the raw pair files. Anyone publishing numbers on this model needs to state which of these stages they ran, because the project documentation says to identify the exact dataset distribution and preprocessing used, and notes that the original files and later Hugging Face exports have different split sizes.

## GPU only, with a dependency stack you assemble by hand

The most binding constraint is stated without hedging: the project supports GPU only, because cudnn is used in the convolutional part. That rules out CPU training and turns every prerequisite into a manual installation task. On the model side you need the Lua Torch libraries listed above, including the CUDA stack `cunn`, `cudnn` and `cutorch`. On the preprocessing side Python brings Pillow and numpy, and the project also mentions Node.js and KaTeX as optional pieces used during preprocessing. Evaluation pulls in its own set, including python-Levenshtein, matplotlib, the Distance package and Perl for BLEU, and the Distance package is installed by downloading a specific archive.

```bash
wget https://im2markup.yuntiandeng.com/third_party/Distance-0.1.3.tar.gz
```

```bash
tar zxf Distance-0.1.3.tar.gz
```

```bash
cd distance; sudo python setup.py install
```

This is the shape of the project: a research code drop with an assembled toolchain, not a packaged tool. The repository is not archived and the last push was on 2026-10-19, but there are no GitHub releases, so there is no versioned artifact to pin and upgrades mean tracking the master branch. The licence is MIT, which permits commercial and internal reuse of the code and the model, while the dataset terms sit with the project site and its download guide. Any deployment has to budget for a machine with a supported GPU, a matching cuDNN and CUDA stack, and the willingness to install Perl, ImageMagick and a LaTeX distribution by hand.

## Evaluation compares rendered output, not strings alone

The evaluation stack explains how correctness is judged. Three renderers are listed with distinct jobs: pdflatex renders LaTeX, ImageMagick convert renders LaTeX during evaluation, and webkit2png renders HTML. ImageMagick convert and webkit2png come from separate installation pages, and pdflatex is expected from a TeX Live install, so evaluating the HTML path and the LaTeX path means preparing two rendering toolchains. String level scoring comes from the Perl based BLEU computation, with python-Levenshtein, matplotlib and Distance supporting the image based evaluation that the project calls Python image based evaluation. The intent is that a predicted formula is rendered back into an image and compared with the original page image, which is a sensible check for a system whose output is markup, since a string match says nothing about whether the formula compiles or looks right. The tradeoff is that BLEU rewards token overlap rather than mathematical correctness, and two different markups can render to nearly the same picture. A user who needs semantic checking of a large prediction set has to add that layer themselves; the repository ships no formula equivalence checker.

## A learned model versus a rule based recognizer

The alternative approach to the same problem is a rule based pipeline, where a recognizer produces per symbol candidates and a grammar or layout module assembles them into markup. The difference in approach is where the knowledge lives. im2markup learns both the visual recognition and the sequence generation from paired data, so its quality depends on a training set that resembles the target domain, and the attention alignment gives you a way to inspect failures on a specific image. A rule based recognizer needs no training corpus of rendered formulas, but it needs a curated symbol set and hand written handling of the layout grammar, and it breaks in new domains unless those rules are extended. The practical comparison is about what you already have. If you have rendered formula images and matching LaTeX, an end to end attention model is a reasonable fit, and the IM2LATEX-100K download guide exists for exactly that. If you only have a handful of pages and no corpus, training a model of this size is the wrong shape of work, and the project offers nothing in the way of few shot adaptation. A second alternative is to skip training and use the hosted demo, but that is a browser toy rather than a service with an API contract.

## The dataset and the site are the other half of the deliverable

A good deal of the project's value is documentation rather than code. The static project and dataset documentation live in `site/`, and the documentation states that dataset archives, pretrained weights and recorded demo data remain available at their existing project site URLs. The data page acts as a download and version guide, and the citation page and the `CITATION.cff` file in the repository root cover the paper reference, with the project asking users to identify the exact dataset distribution and preprocessing used when they publish results. The larger dataset referenced by the toy example is linked on Zenodo, and a processed copy is offered on the project site, with the documentation noting that users of the processed version can skip the preprocessing commands entirely. The original archive is credited to coauthor Anssi Kanervisto. What this means for an adopter is that the dataset and the model are versioned separately, at least in practice, and that the split size differences between the original files and the later exports are a real source of mismatched evaluation numbers. A second, smaller point worth checking: the pretrained weights and the demo data are pointed at by URL rather than packaged in the repository, so they are not pinned to a commit.

## Conclusion

Adopt im2markup if you need a LaTeX corpus with matching page images, or you want to study attention based image to markup generation and can supply a CUDA GPU plus the manual dependency setup it expects. Skip it if you need a maintained, pip installable package, a working HTML path, or CPU only training, because the repository publishes no releases and its dataset notes warn that split sizes differ between the original files and later exports. Verify three things before starting: that a processed copy of IM2LATEX-100K from the project data page matches the split you intend to report, that your cuDNN and Torch versions load `src/train.lua`, and that KaTeX normalization leaves formulas intact, since the project itself notes that some formulas fail to parse.

## FAQ

### What hardware does im2markup need to train the model?

A GPU. The project states that it supports GPU only, since cudnn is used in the convolutional part, and the Torch libraries it lists include the CUDA packages `cunn`, `cudnn` and `cutorch`.

### Where do I get the IM2LATEX-100K dataset and pretrained weights?

The project data page acts as a download and version guide and hosts a processed copy, and the larger dataset is linked on Zenodo. Pretrained weights and recorded demo data remain at the project site URLs, and the documentation notes that a processed copy lets you skip the preprocessing commands.

### How is im2markup evaluated?

With string level BLEU computed through Perl, and with a Python image based evaluation that uses python-Levenshtein, matplotlib and the Distance package. Rendering is needed too, with pdflatex and ImageMagick convert for LaTeX and webkit2png for HTML.

### How do I turn a page of LaTeX into the input images im2markup expects?

Run the image preprocessing script, which crops the formula area out of the rendered page and groups images of similar size for batching. The formulas are normalized separately with the KaTeX based script, which prints error messages for formulas it cannot parse.

## Sources

- [harvardnlp/im2markup on GitHub](https://github.com/harvardnlp/im2markup)
- [Issues](https://github.com/harvardnlp/im2markup/issues)
- [License: MIT](https://github.com/harvardnlp/im2markup/blob/master/LICENSE)
- [Project website](https://im2markup.yuntiandeng.com)
- [README](https://github.com/harvardnlp/im2markup/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/harvardnlp-im2markup