Qwen-MM-Plugins: Giving Agent Harnesses Native Multimodal Skills

The gap Qwen-MM-Plugins fills between an agent harness and media files

Most agent harnesses are text-first. When you point one at an image, a video or a PDF, the model either cannot see it or the harness shells out to some ad-hoc script. Qwen-MM-Plugins takes the opposite approach: each capability ships as an Agent Skill plus an optional MCP server named qwen-mm-plugins-<capability>, and the Skill decides which MCP tool to call. The project describes itself as making any agent harness multimodal-native, and the README frames the core plugin as the one that lets the main model read images, video and files natively rather than routing them through a separate API or ad-hoc shell commands. That is the specific problem: not adding another vision endpoint, but removing the routing step entirely for models that can already see.

The intended audience is narrow and specific. You need an agent running in Claude Code, CodeBuddy, Codex, Qoder, OpenClaw, Qwen Code or Gemini CLI for the guided installer, or one of WorkBuddy, QoderWork, QwenWork, DeepSeek Harness, Hermes Agent, opencode, pi and QwenPaw for manual setup. If you are building a plain Python service with no harness, this is not the layer you want.

How a Skill and an MCP server cooperate at runtime

The architecture is capability-scoped rather than monolithic. Each capability installs independently, and the README tells you to pick by your agent's main model. Three general capabilities apply broadly. The core plugin reads local images and video frames and visualizes documents, code, data, 3D models and NIfTI volumes for the agent to inspect, with media metadata, cropping, bounding-box annotation and page or frame export; the README states there is no API key in the default native mode. The api plugin calls model services for VL vision chat, OCR, grounding, Omni transcription, diarization, captioning, event analysis, ASR and SAM3 segmentation, configured per model family against DashScope or compatible self-hosted services. The search plugin does web search and page extraction through Serper, Exa, Tavily or Serply, with reverse-image search on Serper.

On top of those sit model-specific families. For Qwen VL series models the repository lists video-memory (a hierarchical memory of a long video, so questions are answered from the memory instead of re-watching), video-edit, blender, freecad and edu-agent. For Qwen Omni series models it lists omni-chatcut, omni-video2note, omni-skill-creator and omni-memory. The README is explicit that most harnesses cannot yet feed audio to the main model natively, and that for now audio is handled through the API instead. That single sentence explains why the Omni capabilities need a DashScope key while core does not.

The data flow is therefore: you reference a file in the prompt, the Skill reads the reference, selects an MCP tool, and the tool either processes the media locally or forwards it to a configured service. The README notes that core reads media at dynamic resolution, so manual resizing is normally unnecessary. It also states that NIfTI files stay local and are opened read-only, and that the visualization is not for clinical diagnosis.

Installing Qwen-MM-Plugins and running a first real task

The guided installer is a shell script fetched from the repository. It supports Claude Code, CodeBuddy, Codex, Qoder, OpenClaw, Qwen Code and Gemini CLI, and shared configuration lives in ~/.qwen-mm-plugins/config.

curl -fsSL https://raw.githubusercontent.com/QwenLM/Qwen-MM-Plugins/main/install.sh | bash

After that, you should see the installer's guided flow, which includes Configure and Verify actions for setting credentials and checking dependencies. Run both. The README states that uv provides uvx, which installs Python dependencies on demand, so uvx needs to be on PATH before the capabilities will work.

To update the capabilities already installed in one harness, the README gives a second form of the same script:

curl -fsSL https://raw.githubusercontent.com/QwenLM/Qwen-MM-Plugins/main/install.sh | bash -s -- update

Once a capability is installed, you reference a file and ask naturally; the Skill selects the relevant MCP tool. The README's examples look like this:

@report.pdf          Summarize page 3 and extract its table.
@meeting.mp4         Transcribe this with speaker labels and timestamps.
@place.jpg           Identify where this photo was taken and verify it on the web.

The first two map onto core and, for transcription, the api capability. The third combines an image reference with the search plugin. If you only install core, expect the photo-location example to fail at the verification step, because web search is a separate capability with its own provider credentials.

Where the dependency chain gets heavy

The core plugin is the cheap one. Local core tools need no API key in the default native-image mode, and the README notes that text-only caption fallback, cloud and search capabilities need their provider credentials. Everything past that adds moving parts. video-memory needs a DashScope key and ffmpeg. video-edit needs a DashScope key, ffmpeg and Node. blender needs Blender installed and running. freecad needs FreeCAD installed and running. edu-agent is Skill-only but still needs Node and ffmpeg. omni-chatcut needs the relevant generation and Omni services, ffmpeg and ffprobe, plus an optional external dubbing service for translated voice output.

The failure mode here is environmental, not logical. If ffmpeg is missing, an Omni capability cannot decode the audio track it was built around. If Blender is not running, the blender capability has nothing to drive, because the README describes it as driving a running Blender rather than launching one. None of these are bugs; they are the cost of reaching outside the Python process. The README also points to a dependencies section in the installation guide for exact versions and optional extras, which is where the precise pinning lives.

There is a second limitation worth naming. The README says most harnesses cannot yet feed audio to the main model natively, so the Omni capabilities route audio through the API. If your reason for adopting this project is to keep audio local, the current design does not do that.

Qwen-MM-Plugins versus a plain MCP server

A plain MCP server exposes tools and stops there. The model sees a list of functions and has to choose correctly on its own. Qwen-MM-Plugins pairs each capability with a Skill, which is the part that carries the usage guidance: when to reach for cropping, when to export frames, when to fall back to a caption. The README's instruction to pick by your agent's main model only makes sense in that framing, because the Skill is written against what the main model can already perceive.

The practical difference shows up when a capability is unavailable. With a bare MCP server, a missing credential surfaces as a tool error in the middle of a task. With this layout, the capability is either installed into the harness or it is not, and the README's guidance to install core for multimodal models is an explicit recommendation to keep the always-on path key-free. That is a design position, not a feature list, and it is the main reason to prefer this over assembling individual MCP servers yourself. The trade-off is that you inherit the project's opinion about which capabilities belong together.

Licence, packaging and the cost of keeping up

The project is Apache-2.0, and pyproject.toml declares license = "Apache-2.0" with requires-python = ">=3.10". The packaging is more careful than most: license-files lists the project LICENSE alongside third-party attribution and vendored MIT license texts, including NOTICE.md files for the blender, freecad and omni-chatcut capabilities, an OFL.txt for fonts shipped inside the omni-chatcut music-to-mv workflow, and the vendored licenses under the blender and freecad packages. If you redistribute a wheel, those files travel with it in .dist-info/licenses/. That is a packaging fact, not legal advice; read the notices yourself if you plan to ship the addons.

Upgrade cost is governed by how the capabilities are released. The README states that released capabilities use independent, immutable tags, which means an update touches only the capabilities you ask for rather than the whole set. The installer's update action is the supported path. What the README does not document is rollback: it points to the installation guide for local checkout installs and rollback, so that is the document to read before you upgrade in a working environment. The repository's last push was on 2026-09-17.

Editorial conclusion

Adopt Qwen-MM-Plugins if your agent already runs inside one of the supported harnesses and you want local image, video-frame and document inspection without wiring a separate vision API. Skip it if you only need text, or if you cannot install uv, ffmpeg, Node, Blender or FreeCAD on the machine. Before anything else, run the installer's Configure and Verify actions and confirm that uvx is on PATH; the README does not document rollback for released capability tags, so pin the tag you install and keep a copy of it.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. QwenLM/Qwen-MM-Plugins on GitHub
4. README