deepfates/memery: natural language image search over local folders with CLIP

The problem memery targets: folders you can only search by scrolling

The README opens with a scenario rather than a feature list. You have a large folder of images (memes, screenshots, datasets, product photos, inspiration albums) and you know one specific image is in there, but you cannot recall its filename or the day you saved it. The only fallback is scrolling through thumbnails and hoping you recognise the right one when it passes. Filenames do not help because they were never descriptive, and directory structure does not help because the folder grew by dumping rather than by filing.

Memery's answer is to index the folder once and then search it by describing what you remember about the image, for example the README's own example query, "a line drawing of a woman facing to the left". The audience is narrow and practical: people who keep images locally and want search to run on their own machine rather than on a hosted service. The README frames the author's motivation plainly, saying the project was written to solve the author's own needs and to learn notebook-based development. That framing matters when you evaluate it, because the scope follows one person's workflow rather than a product roadmap.

How the CLIP and Annoy pipeline actually works

The mechanism is described in the README and is visible in the dependency list. Memery uses CLIP, the Contrastive Language-Image Pretraining transformer released by OpenAI in 2021. CLIP trains a vision transformer and a language transformer into a shared latent space, so an image and a caption that describe the same thing end up close together. That shared space is what makes a text query comparable to an image at all.

The pyproject.toml shows the rest of the stack: torch and torchvision for the model, Pillow for loading images, annoy for the nearest-neighbour index, tqdm for progress reporting, and streamlit for the browser interface. The flow implied by the commands is: encode every image in a folder into a vector, store those vectors in an Annoy index on disk, then encode your query with the same CLIP text encoder and ask Annoy for the closest vectors. The CLI names these stages directly, with `build` to encode and index, `recall` to search, and `purge` to clear the encodings and index out again.

Two details are worth flagging. First, the README says you can query with text, with a filepath to an image, or with both, and that entering both makes memery search for the combination. Second, the README claims search times scale well under O(n) but also states the project is not optimised for performance yet. Those two statements sit together: the index lookup is sublinear, but encoding the query and loading results still cost something, and the README gives no measured numbers for either.

Installing memery and running a first search

The README specifies Python 3.10, 3.11 or 3.12, and pyproject.toml pins the range as >=3.10,<3.13, so a newer interpreter will not satisfy the constraint. The recommended install is straight from GitHub:

pip install git+https://github.com/deepfates/memery.git

The PyPI route needs a second command, because the CLIP dependency is pulled from a git revision rather than from a package index:

pip install memery
pip install git+https://github.com/openai/CLIP.git

The README states that memery currently defaults to a GPU installation and that this will probably change in a future version. If you have no GPU, it says to run this afterwards:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

On Windows the README documents two batch files at the repository root, `windows-install.bat` and `windows-run.bat`, as the quickstart path.

Once installed, the fastest way to see whether the idea works for you is the browser interface. Running it with no argument opens the default directory, which the README says is `./images`, the project's own example meme folder:

memery serve

You can also point it at a real folder, and the path is resolved relative to your current working directory:

cd ~/Pictures
memery serve memes

The README notes a quirk here: the sidebar Directory box needs a full directory path, because Streamlit has no folder-picker component at the time of writing. For scripted use, the CLI returns ranked paths and takes a count and both query types:

memery recall PATH/TO/IMAGE/FOLDER -t 'text_query' -i 'PATH/TO/IMAGE.jpg' -n 20

If you would rather build the index ahead of time, `build` takes a worker count and defaults to 0:

memery build PATH/TO/IMAGE/FOLDER --workers 4

In Python, the README's example imports the class and calls `query_flow` with a folder and a query, which returns a ranked list:

from memery.core import Memery
memery = Memery()
ranked = memery.query_flow('./images', 'dad joke')
print(ranked[:5])

The README shows the console output for that call, including a line reporting how many images were searched and the elapsed time, followed by a list of ranked file paths. Expect the first run to download the CLIP model; the pytest configuration in pyproject.toml describes the integration tests as loading the CLIP model and notes a roughly 338MB download on first run.

Where memery breaks down or is the wrong tool

The README is unusually candid about rough edges, and they are the right things to weigh. The browser GUI is described as rough, and the README warns that major errors appear in the right-hand panel as giant stack traces, with the suggestion to screenshot them and file a GitHub issue. That is not a failure mode you can configure around; it is the current state of the interface. The project also says it is not optimised for performance yet, so a folder of tens of thousands of images is an untested scale rather than a supported one.

The packaging situation is the larger constraint. The README states that someday memery will be packaged in an easy to use format, but that since it is a Python project it is hard to predict when that day will be. There is no desktop application, no Docker image, and no service binary. If your requirement is a tool a non-technical colleague can install and run, memery does not meet it today. The GPU default compounds this: a CPU-only machine needs a follow-up pip command that replaces torch and torchvision, which is exactly the kind of step that surprises people who expected the first install to be sufficient.

There is also a scope limit worth naming. Memery searches images by their visual and semantic content. It does not read text inside images, so a screenshot whose distinguishing feature is a line of code or a paragraph of text is not something the README claims to handle. If your folder is mostly screenshots of documents, a tool built around OCR will serve you better than one built around a vision-language embedding.

Memery versus a hosted image search service

The obvious alternative is a hosted visual search product, where you upload images and query them through a web interface. The difference is not mainly accuracy; it is where the data and the operational burden sit. A hosted service handles indexing, scaling and model updates for you, and it usually works from a browser on any machine. Memery runs the CLIP model on your hardware, keeps the index on your disk, and requires you to manage Python versions, torch builds and disk space for model weights. In exchange, nothing leaves the machine, and there is no per-image cost or upload step.

A second alternative is writing the same pipeline yourself. The pieces are public: CLIP from OpenAI, Annoy for approximate nearest neighbours, Streamlit for a UI. Memery's value is that the wiring already exists as three CLI verbs (`build`, `recall`, `purge`), a `Memery()` class with a `query_flow` method, and a browser interface. If you only need to search a few hundred images once, the setup cost of memery may exceed the cost of just scrolling. The project earns its place when the folder is large enough that scrolling has stopped working and you expect to search it repeatedly.

Maintenance, licence and what upgrading costs you

The repository is not archived, and the last push was on 2026-07-23, which is recent enough that the code is being touched. The version string in pyproject.toml is 0.1.0, and no releases were retrieved, so treat this as pre-1.0 software where the CLI surface and the Python API can still move. The README's own outline and the `query_flow` naming suggest the author is still settling on interfaces rather than preserving them.

Upgrade cost is dominated by dependencies rather than by memery itself. The CLIP dependency is pinned to a git revision, `rev = "main"`, which means an install can pick up whatever that repository's main branch holds at the time. torch is constrained to ^2.2.0 and torchvision to ^0.17.0, and the Python range stops below 3.13, so a Python upgrade can force you to wait for the project to widen its constraint. If you pin memery in a project, pin the CLIP revision too.

The licence is MIT, which is permissive and places few conditions on reuse. That is the whole of what can be said from the repository; questions about how MIT interacts with the licences of CLIP, torch or Annoy in your specific distribution are for your own legal review, not something the README addresses.

Editorial conclusion

Adopt memery if you have a local folder of images whose names tell you nothing and you want text or image queries without uploading anything. Do not adopt it if you need a packaged desktop app, a stable API, or CPU-only installs that work without a second command; the README says memery defaults to a GPU install and that CPU-only requires reinstalling torch and torchvision from the CPU wheel index. Before committing, verify that your Python version is one of 3.10, 3.11 or 3.12, confirm the first CLIP model download fits your disk and bandwidth, and run `memery build` on a copy of a few hundred images to see how long indexing takes on your hardware.

Frequently asked questions

Official sources

1. deepfates/memery on GitHub
2. Issues
3. License: MIT
4. Project website
5. README