StoryMem generates minute-long videos by carrying memory between shots

Shot by shot, with a memory that carries identity forward

StoryMem is the official code for Multi-shot Long Video Storytelling with Memory, a system that given a story script with per-shot text descriptions generates minute-long, multi-shot narrative videos with coherent characters and cinematic visual quality.

The mechanism is described in one line: shot-by-shot generation using a memory-conditioned single-shot video diffusion model. That is the whole idea, and it is worth being precise about why it matters.

A single-shot video diffusion model produces a few seconds of video at a time. Stitching independently generated shots together gives you a video where the character changes face between cuts, the clothing shifts and the lighting jumps, and the result reads as a slideshow of unrelated clips. The failure is not technical quality but identity, and identity in a video is a temporal problem.

StoryMem's answer is to make the model single-shot but the system multi-shot. Each shot is generated conditioned on a memory built from the shots before it, so the model carries character and scene information forward rather than starting from scratch each time.

The example script makes the sequence explicit. It first uses the text-to-video model to generate the first shot as the initial memory, then uses the memory-to-video model to generate the remaining shots shot by shot, automatically extracting keyframes and updating the memory after each shot. The keyframe extraction is what turns a generated clip into reusable memory for the next one.

The output is a narrative video with a story arc and multiple shots, not a loop of a single scene.

The install is a conda environment on Python 3.11

Installation is a five step sequence, and the pinning is specific enough to matter.

git clone --single-branch --branch main [email protected]:Kevin-thu/StoryMem.git
cd StoryMem
conda create -n storymem python=3.11
conda activate storymem
pip install -r requirements.txt
pip install flash_attn

The requirements file pins the core stack tightly: torch at 2.4.0, torchvision at 0.19.0, diffusers at 0.32.2, transformers at 4.45.2 and peft at 0.14.0, with numpy held below 2.0. The presence of peft is the tell that this is a LoRA workflow, and the presence of diffusers and transformers is what tells you the generation runs through the Hugging Face stack rather than a bespoke sampler.

Beyond the core, the file is a grab bag of what a video pipeline needs: opencv-python and decord for video IO, imageio with ffmpeg for writing output, accelerate for device placement, easydict for the argument namespaces the script uses, json5 for lenient config parsing, dashscope for the API surface, hpsv3 for a hypernetwork-style prior, tensorboard for logging, and a direct git dependency on OpenAI's CLIP.

flash_attn is installed separately and last, and the commented-out note in the documentation explains why. If the installation of `flash_attn` fails, try installing the other packages first and install `flash_attn` at last. There is also a commented-out hint about commenting out `torch` and `torchvision` in the requirements file and installing the appropriate CUDA build instead, which is the standard workaround when the pinned torch does not match your CUDA version.

The practical constraint is that this is a CUDA-oriented setup with a fixed torch version, not a portable one.

Two checkpoints, and the difference is what the memory is made of

The model download is a Hugging Face affair, and there are three artifacts to fetch: the Wan2.2 text-to-video MoE model, the Wan2.2 image-to-video MoE model, and the StoryMem memory-to-video LoRA.

The download commands use huggingface-cli with a local directory each:

pip install "huggingface_hub[cli]"
huggingface-cli download Wan-AI/Wan2.2-T2V-A14B --local-dir ./models/Wan2.2-T2V-A14B
huggingface-cli download Wan-AI/Wan2.2-I2V-A14B --local-dir ./models/Wan2.2-I2V-A14B
huggingface-cli download Kevin-thu/StoryMem --local-dir ./models/StoryMem

The LoRA repository contains two models, and the distinction is the conditioning, not the quality. `StoryMem/Wan2.2-MI2V-A14B` supports M2V and MI2V, which is memory plus first-frame image conditioning. `StoryMem/Wan2.2-MM2V-A14B` supports M2V and MM2V, which is memory plus the first 5 motion frames as conditioning.

That is a meaningful difference for a video system. A first-frame image gives you the appearance of the character at the start of a shot, which is a static anchor. Five motion frames give you the character in motion, which is a dynamic anchor, and is presumably why a mode with more of them exists.

Both are built on a 14B mixture-of-experts video model, which sets the hardware expectation. This is not a checkpoint that runs on a laptop.

The full set of weights is on Hugging Face, and the project page is at kevin-thu.github.io/StoryMem with the paper on arXiv at 2512.19539.

The argument list is where the design decisions are visible

The example is run with a single script:

bash run_example.sh

and the key arguments are documented, which is more useful than it sounds because they expose the choices in the pipeline.

The paths are the obvious ones: `story_script_path` for the story script JSON, `output_dir` defaulting to ./results, `t2v_model_path` defaulting to ./models/Wan2.2-T2V-A14B, `i2v_model_path` defaulting to ./models/Wan2.2-I2V-A14B and `lora_weight_path` defaulting to ./models/StoryMem.

Then the parameters that change behaviour. `seed` defaults to 0. `size` is the output resolution and defaults to 832*480. `max_memory_size` is the maximum number of shots kept in memory, defaulting to 10, and it is the single most interesting parameter because it is the memory budget of the whole system.

The two first-shot flags decide how the memory is seeded. `t2v_first_shot` uses the text-to-video model to generate the first shot as the initial memory. `m2v_first_shot` uses the memory-to-video model to generate the first shot, which is the MR2V setting where reference images are provided as the initial memory, and those reference images are expected in `output_dir` named `00_00_keyframe0.jpg` up to `00_00_keyframeN.jpg`.

The final two are the continuity switches. `mi2v` enables memory plus first-frame image conditioning to connect adjacent shots when `scene_cut` is False, and `mm2v` enables memory plus the first 5 motion frames conditioning for the same purpose.

So the system has two axes of choice, whether to cut between shots or to connect them, and if you connect them, whether the continuity is anchored on a still image or on motion frames.

ST-Bench measures the thing the system claims to fix

Evaluating multi-shot long video storytelling needs its own benchmark, and the project ships one in the story/ subfolder.

ST-Bench was built by prompting GPT-5 to create 30 long story scripts spanning diverse styles, each containing a story overview, 8 to 12 shot-level text prompts, and scene-cut indicators. In total it provides 300 detailed video prompts describing characters, scenes, dynamic events, shot types, and possibly camera movements.

The numbers line up in a way that makes sense. Thirty scripts at 8 to 12 shots each gives roughly 300 prompts, so each script is scored across its own shots. The scene-cut indicators are what make the benchmark able to distinguish a system that handles continuity from one that cuts cleanly, since a system that keeps the character consistent across a cut that should be a cut is also wrong.

The documentation also publishes the system prompt used to generate structured, shot-level story scripts, so you can follow the same recipe to create your own script. That is a deliberate inclusion, since it makes the benchmark reproducible and extensible rather than a fixed set of files.

The 8 to 12 shot range per script is also the scale the memory system is designed for, and it lines up with the `max_memory_size` default of 10. A story longer than the memory window means the system stops remembering, which is a design limit rather than a bug.

What the repository contains and what it does not

The top-level layout is small and gives a clear picture of the scope.

There is generate.py, which is the entry point the run script drives, pipeline.py, which holds the generation logic, and extract_keyframes.py, which is the memory-building step. The wan/ directory holds the Wan2.2 model integration, story/ holds the ST-Bench benchmark and the story script generation, assets/ has the teaser material, and requirements.txt plus run_example.sh are the setup pieces.

What is not there is a service, a CLI with subcommands, a training script or a dataset pipeline. The repository is the code for a published result, with pretrained weights released separately on Hugging Face. There is no training code for the LoRA in the tree, so the LoRA is a fixed artifact you download rather than something you reproduce.

The licence field reads Other, a custom licence GitHub cannot classify, and points to the LICENSE.md file, so the terms need to be read there rather than inferred.

On maintenance, the last push was on 2026-07-22 and there are no GitHub releases. The work comes from a research group with a long author list, with Liming Jiang marked as project lead and Xingang Pan as corresponding author, and the paper and project page give the full picture of what the system is meant to demonstrate.

For an engineer, the realistic use is running the example, changing the story script, and seeing how the character holds up across shots. Extending it to a different base model means reading the wan/ integration and the LoRA's expectations, which is a research task rather than a configuration change.

Editorial conclusion

Adopt StoryMem if you are researching multi-shot video consistency and want a memory-conditioned generation pipeline you can run and modify, since the shot-by-shot loop with automatic keyframe extraction is exposed as a script with documented arguments. Skip it if you need a production video service, because the repository is a research release with a single example script, a 3.11 conda environment and a hard flash-attn dependency, and the weights are a LoRA on top of very large base models. Verify first that your GPU can hold the Wan2.2 A14B models, that flash-attn installs against your torch build, and which of the two provided checkpoints you want, since the M2V and MI2V or MM2V settings condition on different reference material and give different continuity.

Frequently asked questions

Official sources

1. Issues
2. Kevin-thu/StoryMem on GitHub
3. README