sist2: a file indexer that reads the files, not just the names

What it indexes, and what it does with the content

sist2 stands for simple incremental search tool, and the README describes it as a file system indexer and search tool. The list of features is where the design becomes clear.

It is fast, with low memory usage, and multi-threaded. It extracts text and metadata from common file types and generates thumbnails. It scans incrementally, so a second pass over a tree only processes what changed. It provides a mobile-friendly web interface, and a stats page with disk utilisation visualisation.

Two features go beyond a file listing. The first is tagging, both manual tagging from the interface and automatic tagging based on file attributes through user scripts. The second is recursive scanning inside archive files, so a zip full of PDFs is searched rather than treated as an opaque blob.

And there is OCR support with tesseract, which the feature list marks with three asterisks to point at its own section.

The interface side has its own feature, managing and scheduling scan jobs through a simple web interface, which the README notes is Docker only. That is a sensible constraint, since the admin interface is a Node application and the scanner is a native binary, but it does mean the scheduled-scanning workflow assumes a container deployment.

The README puts a warning at the top of the project title, that sist2 is in early development, and it is worth taking at face value.

The format table, and what each row costs

The format support table is the most useful part of the README for anyone deciding whether this fits their collection, and it is unusually honest about what is missing in each column.

Documents, pdf, xps, fb2 and epub, go through MuPDF, which gives text and OCR content, a thumbnail, and the author and title metadata. Comic archives, cbz and cbr, go through the project's own libscan library, with a thumbnail and no text. Plain text, html and xml also go through libscan, with text content and no thumbnail.

Media goes through ffmpeg. Audio gets ID3 tags and a thumbnail. Video gets the title, comment and artist, and a thumbnail. Images get OCR content, a thumbnail, the common EXIF tags and GPS tags.

Raw camera formats, the long list from raw and rw2 through dng, cr2, nef and arw to the Fujifilm variants, go through LibRaw, with no text content, a thumbnail, and the common EXIF and GPS tags. Fonts, from ttf and ttc through cff, woff, fnt and otf, go through Freetype2, with a bitmap thumbnail and the name and style.

Archives, tar, zip, rar, 7z and the rest, go through Libarchive, with text content and no metadata.

The structure of the table is the design. Each row names a library, which means the support is the library's support, and the two asterisk columns, content and thumbnail, are honest about which combinations actually work. A team with a large photo library will get metadata but no text, a team with documents will get text, and the cost of each row is a dependency in the build.

One scanner, two backends, three commands

The search backend is a choice made at install time, and the README frames it as a comparison rather than a recommendation. Elasticsearch needs a running instance, version 6.8.X or newer and ideally 7.14.0 or newer, and SQLite requires no installation at all.

The three commands are the same either way, which is the point of the separation between scanning and indexing. You scan a directory into a file:

sist2 scan ~/Documents --output ./documents.sist2

then you build the search index, which is the step that differs:

# Elasticsearch:
sist2 index --es-url http://localhost:9200 ./documents.sist2

# SQLite:
sist2 sqlite-index --search-index ./search.sist2 ./documents.sist2

and then you serve it:

# Elasticsearch:
sist2 web ./documents.sist2

# SQLite:
sist2 web --search-index ./search.sist2 ./documents.sist2

That split has a practical consequence. The scan output, a .sist2 file, is portable and backend independent, so you can scan once on a fast machine and index it elsewhere, and you can re-index the same scan into a different backend without touching the filesystem. The search index is a separate artefact again, which means the search index can be rebuilt from the scan without rescanning.

For the executable route, the README says to download the latest release and pick the file for your platform, marking the binary executable with chmod +x on Linux or running sist2-x64-windows.exe from a terminal on Windows.

The compose file, and a port you should not publish

The Docker Compose route is the documented one for the admin interface, and the compose file in the repository is the reference version.

services:
  elasticsearch:
    image: elasticsearch:7.17.9
    restart: unless-stopped
    volumes:
      # This directory must have 1000:1000 permissions
      - /data/sist2-es-data/:/usr/share/elasticsearch/data
    environment:
      - "discovery.type=single-node"
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
  sist2-admin:
    image: sist2app/sist2:latest
    restart: unless-stopped
    volumes:
      - /data/sist2-admin-data/:/sist2-admin/
      - /<path to index>/:/host
    ports:
      - 4090:4090
      # NOTE: Don't expose this port publicly!
      - 8080:8080
    entrypoint: node
    command:
      - /root/sist2-admin/server/main.js

Three details in that file matter in practice. The Elasticsearch data directory must have 1000:1000 permissions, which is the classic Elasticsearch-in-a-container trap and is the first thing that fails on a fresh host. The heap is pinned to 2 GB in both directions, so the container will not size itself to the machine. And the comment on port 4090 says not to expose it publicly, which is a note to whoever copies the file.

The admin service is interesting in a different way. Its entrypoint is node and its command is the admin server's main.js, so the image is doing double duty, carrying the native scanner and the Node admin application in one container, with the scanned tree mounted at /host so the scanner inside the container can read the host's files.

Why the portable binary is musl and the image is glibc

The Dockerfile header explains the build targets and then gives away the most interesting engineering note in the repository.

There are two builds of the same source. The portable binary is static musl, and the runtime image is glibc. The reason is mime detection. The comment says that mime detection spends most of its time in libmagic's regexes, and musl's TRE runs them about 1.5 times slower than glibc does, which is worth about 1.5 times on a scan of files without a usable extension.

That is a very specific performance claim, and it is about a real problem: a tree full of files with no extension is a common case for a media or asset collection, and libmagic's pattern matching becomes the bottleneck rather than the file reading.

The build itself is a multi-stage one. The frontend stage runs on node:22-slim and builds the two JavaScript applications, sist2-vue and sist2-admin, separately from the C build. The musl stage is alpine:3.22 with a long list of build dependencies, and it builds vcpkg from a pinned git commit with VCPKG_FORCE_SYSTEM_BINARIES=1 and a specific cmake installed from PyPI, because the prebuilt vcpkg tool is glibc-only and its port scripts need a newer cmake than Alpine ships.

And there is a comment on layer ordering that is the sort of thing that only matters at scale: dependencies first, because that layer is the expensive one and only rebuilds when the manifest changes, with the release triplets skipping the debug half of every port.

The repository layout follows the same split, with src/ for the C code, libscan/ for the scanning library, sist2-vue/ and sist2-admin/ for the two web applications, third-party/ and overlay-ports/ for dependencies, schema/ for the index schema, and a demo/ directory with its own compose file and a script to update it.

Where sist2 is the wrong tool

The first wrong-tool case is a document management system. The feature list is about search, extraction and visualisation, and there is nothing about versioning, retention policies, review workflows, access control or audit. A team that needs a records system with a lifecycle should not read the absence of those features as an oversight.

The second is a very large collection with an expensive index. OCR on images, recursive scanning inside archives, and thumbnailing for every media file all cost time and disk, and the index of a large tree is a database you now have to keep. The SQLite backend is the simple path, and Elasticsearch exists precisely because some collections outgrow it.

The third is a platform that cannot run the pieces. The scanner is a native binary, the admin interface is a Node application, and the scheduled scanning workflow is Docker only. A team with a hard container policy that forbids Docker, or a platform without a native binary for that platform, is out of the parts of this tool that matter.

The fourth is a project that needs stability. The README's own warning is that the project is in early development, the release numbering shows a project still finding its shape with 4.2.0, 4.2.1 and 4.2.3 published close together, and there is a demo instance and a Discord community rather than a support commitment. That is fine for a tool you run on your own files, and a consideration for anything else.

Editorial conclusion

Adopt sist2 if you want a self-hosted search over a mixed collection of documents, media and archives, and you are comfortable with a young project that says so in its own README. Do not adopt it as a document management system with versioning, retention or access control, since the feature list is about search and metadata, and do not expect Windows OCR to work, since the support table marks image OCR through the ffmpeg path while the OCR feature is tied to tesseract. Before you deploy, read the early development warning, decide between the SQLite and Elasticsearch backends rather than trying both, and check the disk and memory cost of indexing before you point it at a large tree, because the archives and OCR paths do the expensive work.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. README
4. Releases
5. sist2app/sist2 on GitHub