Falcon-Perception: a readable inference engine for a dense multimodal model that detects, segments and reads

This is the inference repo, and the models live elsewhere

The repository describes itself as the inference repo for the Falcon-Perception and Falcon-OCR model, early-fusion, natively multimodal, dense Autoregressive Transformer models. That description is dense with technical choices, and each one narrows what the model is.

Early fusion means the image and the text query go into the model together rather than through separate encoders joined later. Natively multimodal means there is no separate vision tower to bolt on. Dense means the classic pre-Mixture of Experts architecture, and autoregressive means the decoder generates tokens to describe what it sees, which is why the same model can emit bounding boxes, masks and text.

What the repository contains is an implementation, and the README is specific about the intent: a minimal, readable yet performant PyTorch inference engine implementation of Falcon Perception. The weights are on Hugging Face, as Falcon-Perception and Falcon-OCR, with the OCRBench-v2 and PBench datasets also on Hugging Face, a tech report on arXiv, a blog post, and an online playground at vision.falcon.aidrc.tii.ae.

The licence is Apache 2.0, with the LICENSE and NOTICE files listed as licence files in the package metadata, and the package version is 1.0.0. The classifiers say Development Status 4, Beta, which is a more honest description of an inference engine than the version number alone.

The package itself is falcon-perception, requires Python 3.10 or newer, and is maintained by the Technology Innovation Institute, with Phuc H. Le Khac named as maintainer.

One prompt, and the answer can be boxes, masks or text

The examples at the top of the README are the clearest statement of the capability. The first is a natural language query, Segment these expressions in the image: the cat on the left, and the result is described as bounding boxes plus pixel-level masks. The second is Extract the text content from this image, and the output is text, LaTeX formulas or an HTML table.

That second one is the more surprising of the two. A single model that reads a page and returns prose, LaTeX and structured tables in one pass is a different thing from a layout detector followed by an OCR engine, and the repository makes that point in the OCR section by recommending end-to-end OCR for most documents and reserving the layout-aware path for very dense pages such as multi-column newspapers.

The tasks are selectable at run time, which is how the same checkpoint serves several jobs. For detection and segmentation the demo takes an image and a query:

# Auto-downloads model + streaming sample image from Huggingface
python demo/perception_single.py

# Custom image and query, can be path or url
python demo/perception_single.py --image photo.jpg --query "cat"

# Detection only (no masks)
python demo/perception_single.py --image photo.jpg --query "cat" --task detection

The image argument accepts a path or a URL, and the default run pulls both the model and a streaming sample image from Hugging Face, so the first run of the code is a demonstration rather than a blank script.

The demo directory shows how far the repository goes: notebooks for perception and OCR, an agent notebook and an agent/ directory, a GPU worker for the paged engine, a Streamlit app, an OCR benchmark, a perception benchmark, and separate single-sample scripts for the PyTorch and MLX backends.

MLX runs the same model without PyTorch or transformers

The package supports two backends, PyTorch for CUDA GPUs and MLX for Apple Silicon Macs, and a bare pip install auto-detects which one to use, Mac to MLX and Linux to Torch.

The MLX path is the more interesting claim. The README notes that the MLX backend does not require PyTorch or transformers at all, because the core data pipeline and tokenizer run on numpy and PIL with the lightweight tokenizers library.

That is a genuinely different shape of deployment. A team with a Mac, or a cluster of Macs, can run both the perception and the OCR demos without pulling a CUDA stack into the environment, and the scripts are the same shape as the PyTorch ones:

python demo/perception_single_mlx.py --image photo.jpg --query "cat"

# Detection only
python demo/perception_single_mlx.py --image photo.jpg --query "cat" --task detection

The dependency markers in the package metadata match that split, with mlx required on Darwin arm64 only and torch and torchvision required everywhere else, which is how a single wheel list can serve two very different stacks.

The contents list shows that the MLX backend is not a one-off script but a parallel implementation, with an MLX Batch Inference Engine listed alongside the PyTorch Paged Inference Engine, the Paged OCR Inference Engine and the Batch Inference Engine. The architecture is therefore organised as engines rather than as a single code path, and the engines are named after how they schedule work: paged for one document at a time across pages, batch for throughput.

The extras decide whether you need a layout model at all

The installation table is the part of the README with the most practical detail, and the extras are the decisions.

A bare pip install auto-detects the platform. The explicit extras are: torch for PyTorch with CUDA, for a GPU server or an explicit Torch on a Mac; mlx for Apple Silicon; ocr for Torch with transformers, for layout-aware OCR, which needs a layout detection model; dev, which adds tensorboard, matplotlib and ipykernel; and server, which adds FastAPI and Uvicorn for the paged inference server.

The ocr extra exists because the layout-aware mode is a different pipeline. The README is explicit that end-to-end OCR, the ocr_plain default, is recommended for most documents, sending the full page straight to the model, while layout plus OCR is for very dense multi-column pages. The layout mode detects regions first and then extracts text per region, and it lazily downloads and runs a third party layout detection model, PaddlePaddle/PP-DocLayoutV3_safetensors, from Hugging Face.

That is a real operational difference between two flags on the same script:

# Custom document image
python demo/ocr_single.py --image document.png

# Layout-aware OCR (detects regions first, then extracts text per region)
# Requires the [ocr] extra: pip install -e ".[ocr]"
python demo/ocr_single.py --image document.png --task ocr_layout

The server extra matters for anything beyond a notebook, because the repository ships an inference server, a Streamlit demo app, and a vLLM Docker server for FalconOCR only, with a serving throughput section in the contents.

CUDA 12.8 is the default, and changing it means editing pyproject.toml

The recommended install path is uv, and the example is two commands:

# Example: from source with uv (recommended)
uv sync                       # installs torch with CUDA 12.8 wheels by default
source .venv/bin/activate

The CUDA default is then a decision you may have to change, and the README explains how in a way that saves an hour of guessing. uv sync defaults to CUDA 12.8 wheels for PyTorch, which works with an NVIDIA driver of 570.x or newer. For a different CUDA version you edit the [[tool.uv.index]] URL in pyproject.toml, with cu126 for a driver of 560.x and cu130 for 575.x, and then re-run uv lock && uv sync.

If you prefer pip, the order matters, because the README tells you to install PyTorch first with the correct CUDA version:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install -e .

The most recent model announcement is also in this section of the document. Falcon OCR 1.5 is described as a 270M early-fusion OCR model, improving end-to-end accuracy on full-page OCR and extending coverage to historical scans, handwriting, receipts and real-world scene text, while staying around three times smaller than 0.9B-class OCR vision language models.

That size point is the one to hold on to. The claimed advantage is not raw capability against the largest models, it is a small model that handles a page end to end, which is a different bet on what a document pipeline needs.

Where Falcon-Perception is the wrong tool

The first wrong fit is a workload that needs the strongest model available. The OCR model in this repository is 270M parameters, and its advantage is stated in terms of size and end-to-end accuracy on full pages rather than general capability. If your accuracy bar is set by the largest vision language models, a small early-fusion model is not where you will get there.

The second is a machine that is neither CUDA nor Apple Silicon. The MLX path is Darwin on arm64 only, and the PyTorch path wants a CUDA GPU, so a Windows workstation or a Linux box without a supported NVIDIA driver is not covered by the two backends the package declares.

The third is a pipeline that needs deterministic, structured output rather than generated text. This is an autoregressive model producing descriptions, and the output it returns for a detection task is model output, not a schema-validated record. The repository's own choice of pycocotools and opencv-python in the dependencies suggests the results are consumed with the usual detection tooling, and that expectation should be tested against your own data rather than assumed.

The fourth is a network-restricted environment. The demos auto-download the model and a sample image from Hugging Face, the layout mode pulls a third party detection model from the same place, and hf-transfer and hf-xet are in the dependency list to make those downloads fast. A locked-down air-gapped machine has to pre-stage the weights, and the README does not document an offline workflow.

The fifth is anyone expecting a training repository. This is inference only, with an eval/ directory and tests/ in the tree, and the weights are published separately.

On the state of the project: the tagged release is v1.0.0 on 2026-04-07, and the last push to the repository was on 2026-09-11 with the repository not archived, so the tree is moving while the release line is still young.

Editorial conclusion

Falcon-Perception fits a team that needs one model to answer questions about an image with natural language and wants to read the inference code rather than call a hosted endpoint, and that has either a CUDA machine or an Apple Silicon Mac. It does not fit a workload that needs the largest vision language model, since the OCR variant is 270M parameters, and it does not fit Windows users, because the MLX path is Darwin only. Before adopting it, check the Python version requirement and the CUDA driver version your index points at, since the default sync pulls CUDA 12.8 wheels that need a 570.x driver.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. tiiuae/Falcon-Perception on GitHub