hello-diana/MASCOT: a two-stage multi-agent companion framework from an EMNLP 2026 paper

The problem MASCOT targets: persona collapse and social sycophancy

Most multi-agent LLM demos are built for task throughput. You wire several agents together, give them roles, and measure whether the group finishes a job. MASCOT starts from the opposite end. Its README frames the goal as user-agent interaction quality in affective and collaborative settings, and names two failure modes that show up when you keep several personas in one conversation. The first is persona collapse: agents drift toward generic assistant behavior and stop sounding distinct. The second is social sycophancy: agents echo each other or agree without adding complementary content, which turns a group into an echo chamber. The framework is aimed at researchers and engineers who want to train against those failures rather than prompt around them. Its two evaluation domains are emotional support (Empathetic Dialogues and ESConv) and workplace meetings (QMSum), and the roster is fixed per domain: The Anchor, The Catalyst and The Beacon for emotional support; Minutes Scribe, Decision Logger, Action Item Captain and Critic for meetings. If your use case is a single assistant answering tickets, this framing buys you nothing.

How the two-stage pipeline moves data from episodes to policies

Stage 1 aligns each Speaker separately. The entry point src/dataset/generate_and_annotate.py --mode individual produces K = 8 candidate responses per context and persona. Those candidates go to src/eval/rubric_evaluator.py, an LLM judge that scores them against rubrics. The scored candidates become preference pairs, up to K*(K-1)/2 = 28 unordered pairs per query before margin filtering, and src/train/train_reward_model.py fits a scalar persona reward model on Qwen3-0.6B. The Speaker policy, Qwen3-8B, is then optimized with GRPO in src/train/train_rl.py against that reward model plus small format and conciseness rewards.

Stage 2 adds a Director. src/generation/mascot_episode.py runs a directed episode: the Director picks the next speaker and emits a natural-language directive naming speaker, action and tone; the chosen Speaker responds conditioned on its persona, the directive and the history. Full trajectories are scored by a group reward model (also Qwen3-0.6B) trained through src/train/train_group.py --mode train-reward-model, and the Director (Qwen3-8B) is optimized with GRPO over trajectory rewards via --mode train-director-grpo. The README is explicit that GRPO is the paper's method and the default. The DPO path, --mode train-director-dpo, is offline preference optimization over stored directive and trajectory pairs from branching episodes, and the README states it is not the paper-reproduction path. That distinction matters if you are comparing your numbers against the paper.

Installing MASCOT and running a first rubric evaluation

The README gives a single install line for the core package, which covers episode generation and rubric evaluation. Everything else lives in optional extras declared in pyproject.toml: train pulls torch, trl, peft and accelerate; serve pulls vllm; metrics pulls torch and nltk for the automatic non-LLM metrics; api-judges adds the Anthropic and Google clients. Python 3.10 or newer is required.

pip install -e .
pip install -e ".[train,serve]"

The first command installs the core dependencies listed in pyproject.toml (openai, datasets, transformers, pandas, numpy, pydantic, python-dotenv, pillow, tqdm). The second adds training and local serving. Note that the README's install section is truncated in the repository listing after the first line, so treat the extras names as coming from pyproject.toml rather than from a documented tutorial.

The pipeline talks to a separately served OpenAI-compatible vLLM endpoint. The repository includes scripts/serve_vllm.py for local serving and an in-process VLLMModel that needs the serve extra. Generation and judging scripts are then pointed at that endpoint. The repository README does not spell out the exact flag or environment variable used to set the base URL in the excerpt available here, so check scripts/serve_vllm.py and the entry points under src/ before assuming a default port. A first real use is the Stage 1 candidate generation step:

python src/dataset/generate_and_annotate.py --mode individual --num-candidates 8

That produces the per-persona candidate set the rest of Stage 1 consumes. The README documents --num-candidates 8 as the K value used in the paper. After it finishes you should have candidate responses ready for scoring by src/eval/rubric_evaluator.py, which is the next step before any reward model is trained.

What the repository does not give you

There are no retrieved releases, so you cannot pull a trained checkpoint from the project's release page. The README notes that Speaker and Director policies train with LoRA adapters only and that base weights are referenced by Hugging Face name, which means training downloads Qwen3-8B and Qwen3-0.6B weights yourself. It also warns that reward-model checkpoints and explicitly merged checkpoints may contain full model weights, so disk and transfer costs are not limited to adapters.

The training extras are not small: torch, trl, peft and accelerate together, plus vllm for serving, and both a 0.6B reward model and an 8B policy in play per stage. There is no documented CPU path. The README reports that trainable policy parameters under LoRA with r=16 are 0.187% of the model, which describes what is trained, not what has to be resident in memory.

Reproducibility has a gap worth naming. The README states that train-director-dpo is retained as an optional alternative and is not the paper-reproduction path, but it does not document how to verify that a GRPO run matches the reported figures, nor does it describe rollback if a run diverges. The evaluation section lists human, multi-judge (GPT-4o, Gemma-3-27B, Phi-4) and automatic metrics across Empathetic Dialogues, QMSum and out-of-domain ESConv. Human evaluation is not something a single user can rerun from the repository. The README does not document the annotation protocol or rater pool.

How MASCOT differs from a single-agent role-play stack

The obvious alternative is a single LLM with a long system prompt that describes several characters, plus a turn-taking rule in application code. That approach costs nothing to train and is trivial to deploy. It also has exactly the two problems MASCOT was built to address: one model producing several voices tends to blend them, and a fixed turn order has no mechanism for deciding which voice adds something new. MASCOT replaces the prompt with a trained reward signal per persona and a trained Director that selects the speaker and issues a directive. The trade is real. You gain a measurable objective for persona fidelity and group contribution, and you inherit a training pipeline, a vLLM dependency and a fixed roster per domain. Personas live in config/persona_config.py and src/prompts/persona_templates.py, so adding a fifth meeting participant means touching configuration and prompts rather than editing a paragraph of prose. Another alternative is a framework that orchestrates agents for task completion, such as a planner-executor graph. Those optimize for whether the task got done; MASCOT optimizes for how the exchange reads, which is why its metrics are persona consistency, social contribution and head-to-head human preference rather than task success rate.

Maintenance, licensing and the cost of upgrading

The repository is not archived and the last push was on 2026-08-24, roughly three weeks before this writing. No releases were retrieved, so there is no versioned artifact to pin and no changelog to read for upgrade guidance. The package version in pyproject.toml is 0.1.0. Upgrading therefore means tracking the main branch, and the surface you would be tracking includes training scripts whose flags matter for reproduction: train_group.py alone carries three modes, --mode train-reward-model, --mode train-director-grpo and --mode train-director-dpo. A change to any of them changes what your numbers mean.

Licensing is Apache-2.0 for the source code, and the pyproject.toml comment points to THIRD_PARTY_NOTICES.md for the license scope of figures, website assets, datasets and third-party components. That file is the one to read before reusing media or dataset artifacts. The datasets named in the README (Empathetic Dialogues, ESConv, QMSum) carry their own terms, and the framework downloads base models by Hugging Face name, so each of those has a separate license. Nothing here is legal advice; the practical point is that Apache-2.0 on the repository does not automatically cover everything the pipeline touches.

Editorial conclusion

Adopt MASCOT if you are reproducing the paper's two-stage pipeline or studying persona collapse and director-style turn-taking with the provided entry points; the repository expects a separately served OpenAI-compatible vLLM endpoint and GPU capacity for LoRA training. Do not adopt it as a production companion backend: there is no packaged inference server, no released checkpoints, and the README does not document rollback or versioning for trained adapters. Before committing, verify the exact extras you need (train, serve, metrics) against pyproject.toml, confirm the vLLM server address your scripts will call, and check THIRD_PARTY_NOTICES.md for the license scope of datasets and figures, since Apache-2.0 covers the source code only.

Frequently asked questions

Official sources

1. hello-diana/MASCOT on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README