Emotion-LLaMA reads faces and voices together to name a feeling

Why a single modality is not enough

The overview section states the motivation in a compact form. Accurate emotion perception matters for various applications, including human-computer interaction, education and counselling, but traditional single-modality approaches often fail to capture the complexity of real-world emotional expressions, which are inherently multimodal. Existing multimodal large language models face challenges in integrating audio and recognising subtle facial micro-expressions. Both halves of that sentence point at the same gap. A face alone does not distinguish a sarcastic remark from a sincere one, and audio alone does not tell you whether the speaker is looking at you. The project responds with two things, a dataset and a model, and the dataset comes first in the README because without it the model would have nothing to learn the alignment from. The evaluation results the README reports are on EMER, on the MER2023 challenge and on DFEW in a zero-shot setting, and the DFEW numbers are the ones that matter most for the claim, since zero-shot means no fine-tuning on the target dataset.

The MERR dataset and its two annotation passes

The MERR dataset contains 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories, and the README explains that it extends the range of emotional categories and annotations beyond those found in existing datasets. Each sample is annotated with an emotion label and described in terms of its emotional expression. The construction process is two stages, and it is worth understanding because the numbers only make sense together. The dataset was initially auto-annotated with coarse-grained labels for 28,618 samples from a large pool of unannotated data, and later refined to include 4,487 samples with fine-grained annotations. So the coarse set is the broad one and the fine set is a smaller, more carefully labelled subset on top of it. The annotation content is distributed as a download, and the construction strategy was released separately as the MER-Factory project. That is a meaningful split of artefacts, and it means the labels and the pipeline for making them are separate artefacts you can look at and criticise separately.

How the model is put together

Emotion-LLaMA is described as a model that integrates audio, visual and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, it enhances both emotional recognition and reasoning capabilities. The implementation detail that explains most of the practical work is in the file list. The repository carries a minigpt4/ directory, and the configuration files are named for MiniGPT-v2, so the project is built on the MiniGPT-v2 architecture rather than being written from scratch. The base language model is Llama-2-7b-chat-hf, downloaded from Hugging Face into a checkpoints/ directory and pointed at from the model config file with the llama_model key. The audio side uses a HuBERT-large model, the TencentGameMate Chinese HuBERT-large, which is a speech encoder tuned for Chinese, and that is specified in the demo config as audio_model_path. So the three inputs are handled by an audio encoder, a visual encoder from MiniGPT-v2, and text through the LLaMA backbone, and the alignment step is what the instruction tuning operates on.

Setting up the environment and the checkpoints

The setup section is a three step sequence. You clone the repository, create the conda environment and activate it:

git clone https://github.com/ZebangCheng/Emotion-LLaMA.git
cd Emotion-LLaMA
conda env create -f environment.yaml
conda activate llama

Then you download Llama-2-7b-chat-hf from Hugging Face into the checkpoints/ directory and point the model config at it, and the config snippet is a single key:

# Set Llama-2-7b-chat-hf path
llama_model: "checkpoints/Llama-2-7b-chat-hf"

and the MiniGPT-v2 checkpoint is pointed at in the fine-tuning config with the ckpt key:

# Set MiniGPT-v2 path
ckpt: "/home/user/project/Emotion-LLaMA/checkpoints/minigptv2_checkpoint.pth"

That pattern repeats for the demo, with the HuBERT-large path under checkpoints/transformer/ and the Emotion-LLaMA checkpoint under checkpoints/save_checkpoint/ in the demo config, and the two demo config keys are audio_model_path and ckpt. The environment is pinned in requirements.txt with exact versions, and the list is long and tells you what the stack is: gradio 3.47.1 for the interface, torch with the CUDA 11 and CUDA 12 nvidia packages alongside bitsandbytes for quantised training, transformers and huggingface-hub for the model loading, accelerate for distributed training, decord and moviepy and opencv for video decoding, soundfile and librosa for audio, and spacy with a pinned English model wheel. The repository also carries a requirements.txt alongside the conda environment file, which is the usual duplication in a research codebase.

The demo, the API and the batch path

There are three ways to run the model. The online demo is a Hugging Face Space, and the README shows two example videos in the examples/ directory that cover general tasks and then multimodal emotion recognition and reasoning. The local demo is a Gradio application launched with python app.py, which then serves on http://127.0.0.1:7860, and it needs three more packages installed, moviepy at 1.0.3, soundfile at 0.12.1 and opencv-python at 4.7.0.72. The batch path is a separate script, batch_infer.py, alongside eval_emotion.py for evaluation and eval_emotion_EMER.py for the EMER benchmark, and train.py with the train_configs/ directory for training. The repository also has infer_api.py and app_EmotionLlamaClient.py, and the README points at Gradio API documentation in English and Chinese, with a separate API section covering local inference. The Colab badge at the top of the README gives a third route for people who do not want to build the environment at all.

Three licences for three different things

The repository root is unusual in carrying multiple licence files, and the names make the split clear. There is LICENSE.md, a LICENSE_CC BY-NC 4.0 file, a LICENSE_EULA.md and a LICENSE_MiniGPT4.md. The code is under BSD-3-Clause per the repository facts, but the MERR dataset annotation content is distributed separately under a Creative Commons Attribution-NonCommercial 4.0 licence, and MiniGPT-4 carries its own licence and end user licence terms. So the parts of this project have different permissions, and the one most likely to constrain a commercial use is the data. A company that wants to use the MERR annotations in a product has to read the non-commercial terms, and a company that wants to build on MiniGPT-4 has to read the EULA as well as the MiniGPT4 licence file. The project is also published under a paper with an arXiv link and was accepted at NIPS 2024, and the news list records the challenge results, third place in the MER-OV track of the MER2024 Challenge and the championship in the MER-Noise track.

Editorial conclusion

Adopt Emotion-LLaMA when you are doing research on emotion recognition or reasoning from video and need a model that combines audio and visual cues rather than either alone, and you have the GPU and the checkpoint downloads to run it. Do not adopt it as a production classifier without reading the licence terms, because the repository carries several licences and the data itself is released under a non-commercial licence. Verify first that the MERR annotation content licence permits your use, that the Llama-2 base model terms apply to what you build, and that the paths in the model and demo configs point at the checkpoints you have actually downloaded.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-3-Clause
3. README
4. ZebangCheng/Emotion-LLaMA on GitHub