# Monkey multimodal model setup, inference, and evaluation

> Monkey is a Python research project for running and training large multimodal models that combine image understanding with text-based question answering. The repository provides model code, pretrained weights, a demo, fine-tuning scripts, datasets, and evaluation utilities, but its setup depends on older PyTorch and Transformers versions and documented CUDA-specific installation steps.

**Yuliang-Liu/Monkey** — Monkey (LMM): Image Resolution and Text Label Are Important Things for Large Multi-modal Models (CVPR 2024 Highlight)

- Repository: https://github.com/Yuliang-Liu/Monkey
- Stars: 1,953 · Forks: 140
- Language: Python
- License: MIT
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/yuliang-liu-monkey

## What Monkey solves for multimodal researchers

Monkey addresses the problem of training and running a large multimodal model that answers questions about images. Its target users are researchers working with visual question answering, image captioning, document and chart interpretation, and related image-plus-text tasks. The repository is not a general-purpose chat service with a managed API. It exposes Python model definitions, training entry points, an inference script, a Gradio demo, and evaluation code that can be run locally.

The repository also points to related projects and papers, including Mini-Monkey and TextMonkey. The model zoo lists Monkey-Chat with a Qwev-7B language model and Mini-Monkey with an internlm2-chat-1_8b language model. These entries are presented as downloadable model artifacts rather than as an installation service. A user therefore needs to decide which model variant matches the task, download the corresponding weights, and supply a local image and question.

The README positions Monkey around image resolution and text labels as important factors in large multimodal model behavior. That framing makes it more relevant to research and experimentation than to a deployment team seeking a turnkey API with documented quotas, authentication, and managed hardware.

## Monkey repository structure and execution path

The visible repository layout separates the model implementation from the user-facing tools. The `monkey_model/` directory contains the model-related code, while `inference.py` provides the command-line path for image question answering. `demo.py` and `demo_textmonkey.py` provide demo entry points for the original Monkey demo and TextMonkey. Fine-tuning logic appears under `finetune/` and in files named `finetune_multitask.py` and `finetune_multitask_dialouge_doc.py`. The `eval/` directory contains the evaluation workflow.

For a basic inference request, the documented flow is straightforward. `inference.py` receives a model path, an image path, and a question. The script then invokes the Monkey or Monkey-Chat model against that image and text. The README does not describe batching, server mode, authentication, request limits, or a production deployment configuration, so those capabilities should not be assumed from the repository entry points.

Training has a separate path. The README directs users to `finetune_ds_debug.sh` for Monkey and `finetune_textmonkey.sh` for TextMonkey. It also links to a JSON file containing Monkey training data. The repository includes a data generation directory and identifies the `Monkey_Data` dataset on Hugging Face, but the README does not give a complete end-to-end command for converting raw data into every supported fine-tuning format.

## Installing Monkey and running the first image question

The README specifies a Conda environment using Python 3.9, followed by a clone and installation of `requirements.txt`. This is the documented setup path. The repository does not publish GitHub releases, so the source tree and linked model weights are the available starting points described in the documentation.

Create the environment and install the repository dependencies with the following commands. The README expects the clone to end in a directory named `Monkey`, and the final command installs the packages listed in `requirements.txt`.

```bash
conda create -n monkey python=3.9
conda activate monkey
git clone https://github.com/Yuliang-Liu/Monkey.git
cd ./Monkey
pip install -r requirements.txt
```

The requirements file pins `transformers==4.32.0`, `torch==2.0.1`, `torchvision==0.15.2`, and `torchaudio==2.0.2`. It also lists `accelerate`, `tiktoken`, `einops`, `transformers_stream_generator==0.0.4`, `scipy`, `deepspeed`, `gradio`, `peft`, and other packages. The README also gives a CUDA-specific `flash_attn` wheel for Python 3.9 and a Linux x86_64 environment. That wheel is not a universal installation command, so it should only be used when the filename matches the reader's platform.

After the model weights are available locally, the documented first inference command is:

```bash
python ./inference.py --model_path MODEL_PATH  --image_path IMAGE_PATH  --question "YOUR_QUESTION"
```

Replace `MODEL_PATH` and `IMAGE_PATH` with local paths and put the question in place of `YOUR_QUESTION`. The README does not state the expected output format, so the first check should be whether the script loads the weights and produces a response without a dependency or path error.

## Running the Monkey and TextMonkey demos

The demo paths serve a different purpose from `inference.py`. The README describes an offline mode that downloads model weights separately and changes `DEFAULT_CKPT_PATH="pathto/Monkey"` in `demo.py`. It also describes an online mode in which the demo downloads weights using the model identifier passed with `-c`.

For an offline demo, change the checkpoint path in `demo.py`, then start it with the documented command:

```bash
python demo.py
```

For the online download mode, the README gives this command:

```bash
python demo.py -c echo840/Monkey
```

TextMonkey uses a separate entry point and a model path supplied after `-c`:

```bash
python demo_textmonkey.py -c model_path
```

These commands are useful for a first interaction because they avoid writing a separate image-processing script. However, the README does not document the Gradio server address, port, browser behavior, upload limits, or offline weight format in enough detail to make the demo suitable as a production deployment without further inspection. The linked `Demo_chat` is described as an upgraded interactive demo, while the repository also provides source code and model weights for the original demo. Those are separate options with different setup expectations.

## Evaluating Monkey on visual question answering datasets

Monkey includes `evaluate_vqa.py` and an `eval/` directory. The README states that the evaluation code covers 14 visual question answering datasets. Before running it, the documentation requires configuring the environment, changing `sys.path.append("pathto/Monkey")` to the project path, preparing the required datasets, and then invoking the evaluation shell script.

The documented example uses ESTVQA. The expected layout places images below `data/estvqa/test_image` and stores the annotations in `data/estvqa/estvqa.jsonl`. Each annotation is a JSON object containing an image path, question, answer, and question identifier. The README gives this example format:

```json
{"image": "data/estvqa/test_image/011364.jpg", "question": "What is this store?", "answer": "pizzeria", "question_id": 0}
```

The `ds_collections` dictionary maps an evaluation name to a test file, metric, and maximum number of generated tokens. For ESTVQA, the documented entry uses the `anls` metric and `max_new_tokens` set to 100. The example is configuration, not a complete JSON file, so it should be placed in the repository's existing evaluation configuration rather than treated as a complete input file.

The launch command is:

```bash
bash eval/eval.sh 'EVAL_PTH' 'SAVE_NAME'
```

The README does not specify the exact contents of `EVAL_PTH` or `SAVE_NAME` beyond their role in the command. Dataset preparation is therefore a real part of adoption, not an incidental step. The listed data sources include CC3M, COCO Caption, TextCaps, VQAV2, OKVQA, GQA, ScienceQA, VizWiz, TextVQA, OCRVQA, ESTVQA, STVQA, AI2D, and DUE_Benchmark, with a requirement to follow each original dataset's protocols.

## Training, maintenance, and the cost of using the research stack

The repository supports fine-tuning, but the README provides a shorter path than the inference documentation. It names `finetune_ds_debug.sh` for Monkey and `finetune_textmonkey.sh` for TextMonkey, and links to a training JSON file. The presence of `deepspeed` in `requirements.txt` indicates that distributed training components are part of the dependency set, but the README excerpt does not explain the hardware topology, launcher configuration, checkpointing policy, or memory requirements. Those details need to be checked in the shell scripts and `finetune/` directory before planning a training run.

The repository is not archived, and its last push was on 2026-09-25. However, there are no GitHub releases, so release-based upgrade paths and compatibility notes are not provided by GitHub release metadata. Maintenance activity should therefore be judged from repository changes rather than assumed from the absence of a release tag. The dependency pins are especially important because they place the project on `torch==2.0.1` and `transformers==4.32.0`, which may not match the versions used by newer systems.

The MIT license covers the repository's code, subject to its terms. The data has separate obligations. The README says that the source datasets have their own protocols and must be complied with, while model weights and external datasets are obtained from separate links. Anyone planning redistribution or commercial use should review the applicable license and dataset terms rather than assuming that the repository license covers every downloaded artifact.

## Conclusion

Researchers who already have compatible CUDA hardware and need the Monkey or TextMonkey research workflow should adopt it after checking the pinned dependencies, model-weight source, and dataset licensing terms. Teams that need a current production inference stack, broad model lifecycle documentation, or a fully documented deployment path should first confirm that these older components meet their operational requirements, because the repository lists no GitHub releases and its installation instructions include a specific Python, PyTorch, and CUDA-compatible wheel combination.

## FAQ

### How do I install Monkey?

The README specifies creating a Conda environment with Python 3.9, cloning the repository, and running `pip install -r requirements.txt`. It also documents a separate CUDA-specific `flash_attn` wheel for Python 3.9 and Linux x86_64.

### How do I run Monkey on an image and ask a question?

After obtaining the model weights, run `inference.py` with `--model_path`, `--image_path`, and `--question`. The README provides the command syntax but does not specify the expected response format.

### Does Monkey include evaluation code?

Yes. The repository includes `evaluate_vqa.py` and an `eval/` directory, with evaluation documented for 14 visual question answering datasets. The README gives ESTVQA as an example and requires dataset preparation before running `bash eval/eval.sh 'EVAL_PTH' 'SAVE_NAME'`.

## Sources

- [Issues](https://github.com/Yuliang-Liu/Monkey/issues)
- [License: MIT](https://github.com/Yuliang-Liu/Monkey/blob/main/LICENSE)
- [README](https://github.com/Yuliang-Liu/Monkey/blob/main/README.md)
- [Yuliang-Liu/Monkey on GitHub](https://github.com/Yuliang-Liu/Monkey)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/yuliang-liu-monkey