# EgoX: generating a first-person video from a single third-person one

> EgoX is the research code for a CVPR 2026 paper that takes one exocentric video and produces the corresponding first-person view, using an egocentric prior and a LoRA adaptation on a large video diffusion model. It suits a research group with a large GPU and footage shot with a fixed camera, and it is not a general retargeting tool.

**DAVIAN-Robotics/EgoX** — Code for "EgoX: Egocentric Video Generation from a Single Exocentric Video"

- Repository: https://github.com/DAVIAN-Robotics/EgoX
- Website: https://keh0t0.github.io/EgoX/
- Stars: 762 · Forks: 54
- Language: Python
- License: not declared
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/davian-robotics-egox

## A viewpoint transformation model, not a general video generator

EgoX is the released code for a CVPR 2026 paper, and the project description gives the task in one line: it produces first-person, ego-view videos from a single third-person, exo-view video input. The About section adds the two ingredients that make it work, namely that it uses both exocentric observations and egocentric priors, so that the resulting view is realistic while preserving temporal consistency and scene structure.

The audience is a research group, not a studio. The code is Python, the entry points are infer.py for inference and finetune.py for training, and the model weights are distributed on Hugging Face under the DAVIAN-Robotics organisation, with a project page and an arXiv paper linked from the README. The authors are listed as Taewoong Kang, Kinam Kim, Dohyeon Kim, Minho Park, Junha Hyung, and Jaegul Choo, affiliated with DAVIAN Robotics, KAIST AI, and SNU.

What makes this different from a generic video diffusion model is the conditioning strategy. The README describes a unified approach that integrates spatial and channel-wise information within clean latent representations, and that requires only lightweight LoRA-based adaptation. The base model is a large-scale video diffusion model, and the trained artefact is a LoRA adapter rather than a full checkpoint, which is what keeps the downloadable weight set small relative to the base model.

## The hardware requirement is the first decision to make

The system requirements section is short and unforgiving. A GPU with at least 80GB of VRAM is required for inference, and at least 140GB for training. CUDA must be 12.1 or higher, Python must be 3.10, and PyTorch has to be a version compatible with CUDA 12.1.

That VRAM figure is the practical gate. An 80GB card is not a consumer purchase, and 140GB for training means either a multi-GPU node or a workstation class machine. A team that has to buy hardware before it can run the code should know that the cost sits in the accelerator, not in the software, and that the requirements file itself is a conventional Python dependency list rather than anything unusual.

The dependency list in requirements.txt is worth a look for what it implies about the method. diffusers and transformers carry the diffusion backbone, peft provides the LoRA machinery, accelerate and deepspeed handle distributed training, decord and opencv-python-headless do video decoding, imageio and imageio-ffmpeg handle frame writing, wandb is there for experiment tracking, and tyro and ftfy round out the tooling. The peft and decord entries are the ones that tell you how adaptation and video input are implemented.

## Setting up the environment and pulling the two weight sets

The installation is a conda environment followed by a PyTorch install pinned to the CUDA 12.1 wheel index, then the requirements file.

```bash
# Create conda environment
conda create -n egox python=3.10 -y
conda activate egox

# Install PyTorch with CUDA 12.1
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu121

# Install other dependencies
pip install -r requirements.txt
```

Two weight sets are needed. The first is the Wan2.1-I2V-14B pretrained model, which is saved into a checkpoints/pretrained_model folder, and the README gives a snapshot_download call for it.

```bash
pip install huggingface_hub
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='Wan-AI/Wan2.1-I2V-14B-480P-Diffusers', local_dir='./checkpoints/pretrained_model/Wan2.1-I2V-14B-480P-Diffusers')"
```

The second is the trained EgoX LoRA weights, and here the README offers two routes. The Hugging Face route filters to safetensors files only.

```bash
pip install huggingface_hub
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='DAVIAN-Robotics/EgoX', local_dir='./checkpoints/EgoX', allow_patterns='*.safetensors')"
```

The alternative is a Google Drive download saved into the same checkpoints/EgoX folder. The two routes matter for reproducibility, because the unpinned latest snapshot of a repository can change after you have downloaded it once.

## Running inference on the bundled examples

The repository ships example data in an example directory, split into an in_the_wild case and an egoexo4D case, and the README says inference can be run immediately on them. Two shell scripts are provided, one per case.

```bash
# For in-the-wild example
bash scripts/infer_itw.sh

# For Ego4D example
bash scripts/infer_ego4d.sh
```

The GPU ID and the seed are set inside the script, and the README tells you to edit them if needed, so the two knobs for reproducibility are the device index and the random seed. Results are written to a results directory.

The seed point matters for a research code release. A generative model that turns one video into another is stochastic, and a single seed tells you very little about the method's behaviour. The scripts are set up so that a fixed seed can be held constant while the input changes, which is the comparison anyone reading the paper will want to reproduce.

## meta.json carries the camera geometry, not just a file path

Running on your own footage means preparing a folder with a meta.json, a videos directory, and a depth_maps directory. The tree the README gives is explicit about the three video files per take, with an ego_Prior.mp4, an exo.mp4, and per frame depth maps stored as .npy files.

```
your_dataset/              # Your custom dataset folder
├── meta.json              # Meta information for each video
├── videos/                # Videos directory
│   └── take_name/
│       ├── ego_Prior.mp4
│       ├── exo.mp4
│       └── ...
└── depth_maps/            # Depth maps directory
    └── take_name/
        ├── frame_000.npy
        └── ...
```

The meta.json is where the method's assumptions show up as data. It contains the exocentric video path, the egocentric prior video path, a prompt, and camera intrinsic and extrinsic parameters, with a test_datasets array holding one entry per video.

```json
{
    "test_datasets": [
        {
            "exo_path": "./example/in_the_wild/videos/joker/exo.mp4",
            "ego_prior_path": "./example/in_the_wild/videos/joker/ego_Prior.mp4",
            "camera_intrinsics": [
                [634.47327, 0.0, 392.0],
                [0.0, 634.4733, 224.0],
                [0.0, 0.0, 1.0]
            ],
```

The prompt field is notable because it is structured with an Exo view section and an Ego view section, each carrying a scene overview. So the text conditioning is not a single caption but a paired description of the two viewpoints, which is the text side of the unified conditioning strategy. The camera matrices, including separate ego intrinsics and extrinsics, are the geometric side of the same idea.

## Fixed camera poses and fixed resolutions are hard limits

The Constraints section states the two restrictions plainly, and they are the part of this project most likely to end an evaluation early.

The first is the camera. EgoX is trained on the Ego-Exo4D dataset, where exocentric view camera poses are fixed, so input videos during inference must also have exocentric videos with fixed camera poses. A handheld third-person shot does not satisfy this, and there is no described way to relax it.

The second is the shape of the data. The model is trained on 448x448 for the ego view, 448x784 for the exo view, and 49 frames, so input videos must be preprocessed to those resolutions. That preprocessing step is delegated: the README points to a separate repository, EgoX-EgoPriorRenderer, for the instructions on preparing a dataset, and that renderer is also present as a git submodule entry in this repository.

A related point is that the egocentric prior is an input, not an output of the method. The folder layout wants an ego_Prior.mp4 alongside the exo.mp4, and the depth maps directory wants per frame .npy files. So producing the prior is a prerequisite, and the reader who assumes the model invents the first-person view from the third-person one alone has misunderstood the pipeline.

## Scaffolding a custom dataset with meta_init.py and caption.py

The README offers a lighter path for a user who has exo.mp4 files and nothing else. The custom dataset init structure is just a videos directory with one folder per take, and a meta_init.py script generates the meta.json for it.

```bash
python meta_init.py --folder_path ./your_dataset --output_json ./your_dataset/meta.json --overwrite
```

The output lands in the same folder alongside the videos, producing a structure with a meta.json and a videos directory containing the exo.mp4 for each take.

A second script, caption.py, generates a caption for each video, and the README gives its invocation as the following. The trailing argument in the documented command is the path to the json file, so the script is pointed at the metadata rather than at a directory.

```bash
python caption.py --json_file .
```

So the intended workflow for a new user is: put exo.mp4 files in a take folder, run meta_init.py to create the metadata skeleton, run caption.py to populate the prompts, then fill in or supply the remaining fields, the camera parameters, the egocentric prior, and the depth maps. The README does not document the remaining steps for producing the prior or the depth maps in this repository, which is where the EgoX-EgoPriorRenderer pointer becomes the operative instruction.

## No release tags, no stated licence, and a mid 2026 last push

Two facts about the repository's packaging deserve attention from anyone planning to build on it. The repository has no GitHub releases, so the code is consumed from the main branch and the version of what you run is whatever commit you cloned. The listed licence is unknown, which is unusual for academic code and leaves the terms of reuse unstated in the repository itself, so anyone intending to ship this inside a product needs to resolve the licensing question with the authors rather than assume a permissive grant.

The last push was on 2026-07-10, making it the most recent recorded activity on the repository. The repository includes a configs_acc directory for training configurations, a core directory, a scripts directory holding the inference shell scripts, and a.gitmodules file for the renderer submodule, so the training path is present in the tree even though the README concentrates on inference.

The project page is at keh0t0.github.io/EgoX, the paper is on arXiv as 2512.08269, and the models are on Hugging Face. Those three links are the durable references, since the git history itself carries no tagged versions to pin against.

## Conclusion

EgoX suits a research group that wants to reproduce or extend egocentric view synthesis, has a GPU with at least 80GB of VRAM, and can supply footage with fixed exocentric camera poses, and it does not suit a studio that needs arbitrary camera motion, a different subject, or a production pipeline, since the model is trained on one dataset with fixed poses and a fixed 448x448 and 448x784 resolution. Before anything else, check the exocentric camera constraint, because the README states that input videos must have fixed camera poses because the training data has fixed exocentric view camera poses, and confirm the GPU memory before downloading the 14B pretrained model.

## FAQ

### What hardware does EgoX need to run?

The README lists a GPU with at least 80GB of VRAM for inference and at least 140GB for training, with CUDA 12.1 or higher, Python 3.10, and a PyTorch version compatible with CUDA 12.1.

### Can I run EgoX on my own video?

Yes, but the input must have exocentric videos with fixed camera poses, because the model was trained on the Ego-Exo4D dataset where exocentric view camera poses are fixed, and the video must be preprocessed to 448x448 for the ego view, 448x784 for the exo view, and 49 frames.

### What does the meta.json file need to contain?

It holds the exocentric video path, the egocentric prior video path, a prompt, and the camera intrinsic and extrinsic parameters for each video, inside a test_datasets array with one entry per video. The README points to meta_init.py for creating it and caption.py for generating the captions.

### Where do the EgoX model weights come from?

The trained LoRA weights are on Hugging Face under DAVIAN-Robotics/EgoX and can be fetched with huggingface_hub snapshot_download filtered to safetensors, or downloaded from a Google Drive link. Separately, the Wan2.1-I2V-14B pretrained model has to be downloaded into a checkpoints/pretrained_model folder.

### Does EgoX have a versioned release?

The repository has no GitHub releases, so the code is used from the main branch. The last push was on 2026-07-10, and the listed licence is unknown, which means reuse terms are not stated in the repository.

## Sources

- [DAVIAN-Robotics/EgoX on GitHub](https://github.com/DAVIAN-Robotics/EgoX)
- [Issues](https://github.com/DAVIAN-Robotics/EgoX/issues)
- [Project website](https://keh0t0.github.io/EgoX/)
- [README](https://github.com/DAVIAN-Robotics/EgoX/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/davian-robotics-egox