UniTable trains one model to read tables, their cells and their boxes

One objective for structure, content and bounding boxes

Tables are a hard case for machines because they carry conventions that people created and never wrote down. The paper's framing is that tables convey factual and quantitative data with implicit human conventions that are often challenging for machines to parse, and that prior work on table recognition centred on complex task specific combinations of available inputs and tools.

UniTable's answer is unification on three axes at once. The training paradigm combines the simplicity of purely pixel-level inputs with the effectiveness and scalability that self-supervised pretraining gives from diverse unannotated tabular images. The training objective is language modelling. And the model architecture is a single transformer rather than a set of task-specific heads.

The concrete payoff is that the three classic table recognition tasks, extracting table structure, extracting cell content and extracting cell bounding boxes, are trained together as one task-agnostic objective. The README reports state-of-the-art performance on four of the largest table recognition datasets, and the model was fine-tuned across multiple of them. The released artefact is a Jupyter notebook of the entire inference pipeline, so the claim can be checked on your own image rather than taken on trust.

The notebook is the deliverable

The point of the release is stated in the news section: you can fully digitise your own tabular image in the Jupyter notebook, and the model weights are available on HuggingFace, with UniTable v1.0.0 online since April 2024.

The quick start has three steps and no manual installation of a deep learning stack. The first sets up the virtual environment, the second downloads all the model weights, and the third is the demo notebook, where the instruction that matters is to select the unitable kernel.

The reason the environment step exists is visible in the Makefile. The environment target creates a conda environment named unitable with Python 3.9, installs the requirements file, installs the package itself in editable mode, and then touches a marker file so the target does not run twice. Every other target in the Makefile activates that same environment before doing anything, so the whole repository has one consistent interpreter.

The second step is the weights download, and it is bigger than it sounds. The Makefile names nine specific weight files, and the download target checks how many are already present before it fetches anything.

Nine weight files, from a VQ-VAE to three task heads

The weights path is experiments/unitable_weights, and the nine files are named in the Makefile, which tells you the model family without opening a paper.

Two are VQ-VAE checkpoints, vqvae_1m.pt and vqvae_2m.pt. Two are self-supervised pretraining checkpoints for the visual encoder, ssp_1m_base.pt, ssp_1m_large.pt, ssp_2m_base.pt and ssp_2m_large.pt, at two scales and two pretraining sizes. Three are the actual UniTable heads: unitable_large_structure.pt, unitable_large_bbox.pt and unitable_large_content.pt, one per task.

That last group is worth pausing on, because it is the practical difference between a research framework and a packaged model. The three task heads are separate files, so a user who only needs table structure downloads and runs one of them rather than a single merged checkpoint. The README says the notebook is fine-tuned across multiple datasets and supports all three tasks, and the weights download target uses wget against the HuggingFace repository, with a resume flag on the download and an explicit count check that tells you how many of the nine are present.

The naming also tells you the model is a linear projection transformer rather than a convolutional stem one. The README says as much and points at a separate repository, tsr-convstem, for the early convolution variant, which is the paper about high performance transformers for table structure recognition needing early convolutions.

Make targets and Hydra configuration for every experiment

The training code is driven by Makefile targets and configured by Hydra, and the two fit together in a specific way. Experiment names are defined as EXP_<name> in a section of CONFIG.mk, and the README says each experiment has the command to launch its make target written in the comment above it, which is a pleasant way to discover the interface.

The targets are named after what they produce. Finetuning is offered per task, with a structure target, a bbox target and a cell content target:

# table structure
make experiments/ssp_2m_mini_html_base/.done_finetune

# cell bbox
make experiments/ssp_2m_mini_bbox_base/.done_finetune

# cell content
make experiments/ssp_2m_mini_cell_base/.done_finetune

Pretraining is separate. The VQ-VAE is trained with its own target, and the self-supervised visual encoder is pretrained with masked tabular image modeling, abbreviated MTIM in the README:

make experiments/vqvae_mini/.done_pretrain

make experiments/mtim_mini_base/.done_pretrain

The dot-prefixed target names and the marker file convention are a consistent idea across the Makefile, where each long-running step has a done file that is touched on success, so an interrupted run resumes rather than restarting. The Makefile also requires GNU Make 4.0 or later and refuses to run without it, and it uses the recipe prefix feature to keep the indented command style the author prefers.

The annotation format is HTML with boxes

Training data has to match a specific format, and the repository makes that concrete with a small sample rather than a description. Twenty samples of PubTabNet are provided for a quick walk through the training process, as a directory of images and a jsonl file where each line is a JSON object.

The structure of each line is the interesting part, because it is the same three tasks fused into one record. The filename and split identify the image, and the html key holds the table structure, the cell content and the cell bounding boxes together:

"filename": "tabular image filename inside one of the 'train', 'val', or 'test'",
"split": "One of 'train', 'val', or 'test'",
"html": "table structure, cell content, and cell bbox",
    "cells": "Array with all cell content and bbox",
        "tokens": "Array with the content of the cell",
        "bbox": "The bounding bbox of the cell in [x1, y1, x2, y2] format. Only provided when cell is non-empty.",
    "structure": "Dict with table structure",
        "tokens": "Array with html tags that describe the table structure. '[]' represents non-empty cell",

The tokens are HTML tags, which is what lets a plain language modelling objective cover structure, since the structure of a table is a sequence of tags and the content is text in between. The bbox field carries the cell rectangle in x1, y1, x2, y2 order, and is only provided for non-empty cells.

If you have your own data, the instruction is to convert it to this format. The datasets used in the paper are PubTabNet, SynthTabNet, FinTabNet, ICDAR 2019 B2 Modern and PubTables-1M, and after downloading them you set the root_dir for each dataset under configs/dataset.

Where this is the wrong tool

The first limitation is the deployment story. The released artefact is a notebook plus a set of weight files, and the model runs on PyTorch with a full training stack in the requirements, including torchmetrics, wandb, tokenizers and a set of evaluation packages. That is fine for research and for a demo, and it is not a serving path. There is no API, no export step and no stated latency or memory profile.

The second is the hardware assumption. The Makefile uses torchrun with a stacked single-node multi-worker launch and an NGPU variable, and the environment step creates a conda environment, so the intended workflow assumes a machine with a GPU and a conda installation. The quick start instructions are short but they are not a pip install.

The third is the data domain. The five datasets are document table recognition corpora, meaning tables that appear in published documents, financial filings and scanned pages. The model was tuned for that distribution, and a table photographed off a screen or drawn in a slide is a different input even though the pipeline will happily accept it.

The fourth is that the repository is a moving snapshot of a workshop project rather than a released library. There are no GitHub releases, the version in setup.py is 1.0.0, the news section stops in April 2024, and the last push to the repository was on 2026-04-21, with the repository not archived. The convolutional stem variant sits in a different repository, so following the code across the two is necessary if that is the variant you need.

Licence, dependencies and the research context

The repository is MIT licensed, with the LICENSE file at the root, and the package is defined in setup.py as unitable at version 1.0.0, installed from the packages found in the source tree.

The dependency list tells you the ecosystem. PyTorch with torchvision, torchaudio and torchtext, Hydra with hydra-colorlog for configuration and logging, wandb for experiment tracking, tokenizers for the text side, and the evaluation stack of torchmetrics, pycocotools and faster-coco-eval for the box and detection metrics. For the structural similarity measures that table recognition uses, there are apted, Distance and lxml pinned at 4.9.3, and einops and ptflops for tensor manipulation and model complexity counting. The presence of lxml is a good sign, since the output format is HTML.

Training progress is tracked with Weights and Biases, and the README asks you to register an account if you want to see training curves and reconstructed tables, with the note that the reconstructed table view applies to the VQ-VAE pretraining stage. That reconstruction view is the diagnostic tool for a vector quantiser, and it is the fastest way to tell whether a pretraining run is learning a sensible codebook.

The research context is three papers by the same group, on early convolutions for table structure recognition, on self-supervised pretraining for a table structure recognition transformer, and the UniTable paper itself, with the arXiv identifier of the last one given as the project homepage.

Editorial conclusion

UniTable fits a researcher who needs table structure, cell text and cell boxes from a table image and wants one model and one objective rather than a pipeline of task-specific tools, and who is comfortable in the PyTorch and Hydra ecosystem it is built on. It is the wrong tool if you need production inference at scale, because the released artefact is a research notebook with a model download, and it is the wrong tool if you want the convolutional stem variant, which lives in a separate repository. Before you start, check three things: that a conda installation is available, since the environment target creates a conda environment, that you have disk and patience for nine model weight files, and that your own table images resemble the five datasets the paper used, since the model was tuned for rendered and scanned document tables rather than arbitrary images. The code is MIT licensed, the last push was on 2026-04-21, and there are no GitHub releases.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. poloclub/unitable on GitHub
4. Project website
5. README