XiaoyaoSearch: a local, multimodal desktop search tool built on Faiss, Whoosh and a stack of AI models

The gap XiaoyaoSearch is trying to fill

Desktop search on Windows and macOS is filename-oriented. If you remember that a file is called budget_q3.xlsx, the built-in tools work. If you remember that a file is the one where someone explained the pricing model on a call, they do not. XiaoyaoSearch targets that second case. It indexes the content of local files and exposes three input modes: typed text, a voice recording of up to 30 seconds, and an uploaded image. The project description frames the goal as making search behave like a chat rather than a query box. The stated audience is knowledge workers, content creators and developers, which in practice means people who accumulate a large local corpus of documents, recordings and video and who do not want to upload it to a cloud service to make it searchable. The privacy angle is explicit in the README: local execution is the default and data is not uploaded, with cloud APIs available as an opt-in trade-off between speed and privacy.

Two indexes and four models: the retrieval mechanism

The architecture is a hybrid retrieval design, and the repository layout makes it concrete. Under data/indexes there are two sibling directories: faiss for vector search and whoosh for full-text search. Faiss handles semantic similarity, which is what lets a query about pricing concepts match a document that never uses the word pricing. Whoosh handles lexical matching, which is what stops the system from missing an exact identifier or a rare term that embeddings tend to blur. The README does not describe the fusion step, so how the two result sets are merged and ranked is not something I can confirm from the supplied material. That is a real gap for anyone evaluating retrieval quality.

The embedding side uses BGE-M3, and the multimodal side uses CN-CLIP for images, FasterWhisper for speech, and Ollama for a local language model. The default Ollama model named in the setup instructions is qwen2.5:1.5b, which is a small model; the README does not say what it is used for specifically, whether query rewriting, answer synthesis or something else. SQLite stores metadata alongside the index files. The backend is Python 3.10 with FastAPI and Uvicorn; the frontend is Electron with Vue 3, TypeScript, Ant Design Vue, Pinia and Vite. Since v1.3.0 the project has supported swapping the local model for OpenAI, DeepSeek or Alibaba Cloud compatible APIs, and since v1.6.0 the same applies to embedding APIs.

Getting it running: two paths, very different amounts of work

There are two documented deployment routes, and the difference between them is large. The integrated package route is Windows only and is described as one-click. You download a zip from Baidu Netdisk, extract it to a path without Chinese characters, run scripts/setup.bat, install Ollama by running runtime\ollama\OllamaSetup.exe, then run ollama serve and ollama pull qwen2.5:1.5b. Models come from a second Netdisk link and must be unpacked into data\models\embedding\BAAI\bge-m3\, data\models\cn-clip\ and data\models\faster-whisper\. Finally scripts/startup.bat brings up the backend and frontend. Users on RTX 50 series cards are told to run scripts/setup_rtx50显卡.bat instead, which installs a CUDA 12.8 build of PyTorch.

The developer route is cross-platform and starts with git clone https://github.com/dtsola/xiaoyaosearch.git, then cd backend and pip install -r requirements.txt followed by pip install faster-whisper. GPU support requires uninstalling torch, torchaudio and torchvision and reinstalling version-matched CUDA wheels, with the README giving separate commands for CUDA 12.1 (RTX 40 series and earlier) and CUDA 12.8 (RTX 50 series). ffmpeg and Ollama are separate manual installs. Configuration lives in backend/.env with keys including FAISS_INDEX_PATH, WHOOSH_INDEX_PATH, DATABASE_PATH, API_HOST and API_PORT, pointing at the data directory tree. Stated requirements are Python 3.10.11 or newer, Node.js 21.x or newer, 16GB of RAM and an RTX 3060 with 6GB or more.

Where the setup story breaks down

The dependency chain is the main risk. Model weights are not fetched by a package manager; they arrive through Baidu Netdisk links with extraction codes. Those links are a single point of failure, they are slow or inaccessible outside mainland China, and there is no checksum or mirror documented in the README. If a link dies, the integrated package route stops working entirely.

The GPU instructions are version-sensitive in a way that will bite people. The README itself notes that the CUDA version must be determined from your environment, and the two wheel sets it lists are pinned to specific torch versions, 2.1.0 for CUDA 12.1 and 2.10.0 for CUDA 12.8. Nothing in the material describes a CPU-only fallback path that is known to be usable for the full feature set; the default pip install -r requirements.txt is described as installing the CPU inference engine, but the performance implications for indexing a large corpus are not stated.

Platform coverage is asymmetric. The integrated package is Windows only. macOS and Linux users are pointed at the developer route, which means they own the ffmpeg, Ollama and PyTorch installation themselves. And the README does not document an incremental re-index strategy, so what happens when you add files to an already-indexed directory is unclear.

Licence: not open source in the usual sense

The GitHub licence field reads NOASSERTION, and the README explains why. The project ships its own Chinese-language agreement in LICENSE, with an English version in LICENSE_EN. The stated terms are that non-commercial use is free, modification and redistribution are allowed provided the copyright notice and agreement are retained, and commercial use requires authorisation. This is a source-available arrangement, not an OSI-approved licence. For an individual or a hobby project that distinction rarely matters. For a company, it does: a tool that indexes internal documents and is embedded in a workflow is plausibly commercial use, and the README does not define the boundary. I am not giving legal advice here, but the practical step is to read LICENSE before deploying this inside a company, not after. The presence of an English translation suggests the author anticipated non-Chinese users, though the LICENSE_EN file is listed in the repository tree without any statement about which version governs in a conflict.

AI assistant integration and the version history

Two features distinguish XiaoyaoSearch from a plain desktop search utility. Since v1.4.0 it exposes an MCP server, which the README says allows Claude Desktop to connect and search local files. Since v1.5.0 it provides Agent Skills for Claude Code, VS Code and Cursor. In other words, the index can be queried by an external assistant rather than only through the app's own interface. For developers already working inside one of those tools, this is the most interesting part of the project, because it turns a local corpus into context an assistant can retrieve from.

The release cadence visible in the material is fast: tag-1.8.0 on 8 April 2026, v1.9.0 on 12 April, v2.0.0 on 24 April, with the last push to main on 22 May 2026. v1.9.0 added a terminology library for term expansion and synonym matching, and v2.0.0 shipped a UI redesign plus a terminology management screen. The README claims a 60 percent recall improvement from the terminology library and a 67 percent visual comfort improvement from the redesign. Neither has a stated methodology, and no benchmark data is included in the repository material I have, so treat both figures as marketing rather than measurement.

Alternatives and the honest comparison

The closest well-known comparison is Recoll, a long-standing open source desktop search tool. The difference in approach is fundamental. Recoll builds an inverted index over extracted text and matches query terms, with optional stemming and language handling. It does not need a GPU, it does not download multi-gigabyte model weights, and it installs from a distribution package manager. XiaoyaoSearch inverts that: it accepts a heavy model dependency in exchange for semantic and cross-modal retrieval. Recoll cannot find a video by describing its content or match a query against an image. XiaoyaoSearch's README claims support for content search inside mp4 and avi video, mp3 and wav audio, and txt, markdown, Office and PDF documents, which is a capability set Recoll does not offer. If your problem is exact-term retrieval over text files, Recoll is the lower-cost answer and XiaoyaoSearch is the wrong tool. If your problem is finding a moment in a recording you cannot name, the model-based approach is the only one of the two that can address it.

Maintenance cost and who this is actually for

Running XiaoyaoSearch means maintaining a Python environment, a Node environment, an Ollama installation, an ffmpeg installation and a set of model files that live outside any package manager. Upgrades are not described as automatic. The release history shows three versions in under three weeks, which means the surface you pin today will look different within a month, and the README does not document a migration path for the Faiss or Whoosh indexes between versions. Rebuilding an index over a large corpus is the cost you pay each time the embedding model or the index format changes, and nothing in the material says whether that is required on upgrade.

The developer route also asks you to pin PyTorch wheels by hand, which is the kind of step that silently produces a CPU-only install if you get the index URL wrong. The project is explicitly described as having been built through Vibe Coding, with all source and development documents published for study. That is an honest disclosure and it explains the shape of the repository: extensive Chinese-language planning documents under docs/, including a market research file, a PRD, a technical plan and a task list. It also means you should expect the kind of rough edges that come with fast, single-author development. The author is identified as dtsola, an IT architect, and the contact channels listed are WeChat and Bilibili, which tells you where support actually happens.

Editorial conclusion

Adopt XiaoyaoSearch if you are a Windows user with an RTX-class GPU who wants a self-hosted semantic search layer over local documents, audio and video, and who is comfortable running setup scripts and pulling models manually. Do not adopt it if you need a permissively licensed dependency, a headless server deployment, or a one-command install on macOS or Linux. Before committing, read the Chinese licence text in LICENSE and confirm what commercial use requires, check that your GPU generation matches the PyTorch wheel in the setup script, and verify the model download links are still live.

Official sources

1. dtsola/xiaoyaosearch on GitHub
2. Issues
3. Project website
4. README
5. Releases