# PhysX-Anything: turning one photograph into a URDF asset a physics engine can load

> PhysX-Anything is a research pipeline that takes a single image, reconstructs a 3D object, splits it into parts and emits a simulation-ready URDF and MJCF description with physical parameters. It suits robotics and reinforcement learning researchers who need assets, and it is built on a stack of heavy GPU dependencies with a deformable-part caveat.

**ziangcao0312/PhysX-Anything** — PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image (CVPR 2026)

- Repository: https://github.com/ziangcao0312/PhysX-Anything
- Website: https://physx-anything.github.io/
- Stars: 962 · Forks: 65
- Language: Jupyter Notebook
- License: NOASSERTION
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/ziangcao0312-physx-anything

## A photograph turned into a physics-ready asset

Robot learning and reinforcement learning research both hit the same wall: you need three-dimensional objects with physical properties, and the ones you have are photographs. PhysX-Anything takes a single image and produces a simulation-ready physical asset.

The project comes from S-Lab at Nanyang Technological University together with Shanghai AI Laboratory, with authors Ziang Cao, Fangzhou Hong, Zhaoxi Chen, Liang Pan and Ziwei Liu, and the news section records that it has been accepted by CVPR 2026. The paper is an arXiv preprint, arXiv:2511.13648, and the predecessor PhysX-3D, Physical-Grounded 3D Asset Generation, is arXiv:2507.12465. The project page is at physx-anything.github.io, and the datasets are hosted on Hugging Face as Caoza/PhysX-3D and Caoza/PhysX-Mobility.

The lineage is stated in the acknowledgements, where the data and code are based on PartNet-mobility, Qwen and TRELLIS, with a trellis/ directory in the repository root and the setup script pointing at the TRELLIS repository for its detailed usage. That inheritance is visible throughout: a vision language model reads the image, a structured latent decoder reconstructs the shape, and a post-processing stage converts the result into the asset formats a simulator expects.

The repository's primary language is listed as Jupyter Notebook, and the licence is a custom S-Lab License that GitHub cannot classify.

## The four scripts that make up inference

Inference is a sequence of four numbered scripts, and the numbering is the documentation. First the vision language model reads the input image, then a decoder produces the geometry, then a splitting step separates the object into parts, and finally a conversion step emits URDF and XML.

The first stage takes the image path, the checkpoint path, and two flags that matter for how your input is treated:

```bash
python 1_vlm_demo.py            # vlm inference
    --demo_path ./demo          # inputted image path
    --save_part_ply True        # save the geometry of parts 
    --remove_bg False           # Set this to false for RGBA images and true otherwise.
    --ckpt ./pretrain/vlm       # ckpt path
```

The remove_bg flag is a trap for first-time users: it should be false for RGBA images and true otherwise, and getting it backwards gives you a broken segmentation. The demo/ directory in the repository holds nineteen sample images, 0.png through 18.png, which is enough to see the pipeline work on something other than your own photograph.

The decoder and the splitting step take no arguments in the documented usage, and the final step is where the simulation parameters are chosen:

```bash
python 4_simready_gen.py        # convert to URDF & XML
    --voxel_define 32           # voxel resolution
    --basepath ./test_demo      # results path
    --process 0                 # use postprocess
    --fixed_base 0              # fix the basement of object or not
    --deformable 0              # introduce deformable parts or not
```

Before any of that runs, the pretrained model is fetched with python download.py, pulling from the Hugging Face repository Caoza/PhysX-Anything.

## Twenty-five views condition the generation

The conditioning is not a single image at inference time in the way the title suggests, and the training section explains why. You render the conditioning images based on your requirements, and the README states that there are 25 images per object.

For the PhysX-Mobility dataset that rendering is done by dataset_toolkits/render_cond_mobility.py, and for PhysXNet the README points to the preprocessing script in the PhysX-3D repository, PhysX-3D/dataset_toolkits/precess.sh. The dataset_toolits directory in this repository holds the Mobility side, alongside the dataset/ directory with the preprocessing scripts.

This is the part of the system that makes the physical grounding possible. A single view gives a vision model a shape; a canonical set of rendered views around an object gives it enough information to infer which parts exist and how they relate, which is what allows the pipeline to split the object into separate rigid bodies with their own properties rather than a single blob.

It also means the pipeline is not a consumer tool for arbitrary photographs. The image you feed at inference has to match the distribution of the rendered views the model was trained on, and the objects the training data covers are the objects the datasets cover, which come from PartNet-mobility-derived sources.

## Installing the environment, which is the hard part

The clone step pulls submodules, because the TRELLIS code is vendored as a submodule:

```bash
git clone --recurse-submodules https://github.com/ziangcao0312/PhysX-Anything.git
cd PhysX-Anything 
```

The environment is created with the bundled setup script, which is where the real dependency weight shows. The command names the extensions that are compiled:

```bash
. ./setup.sh --new-env --basic --xformers --flash-attn --diffoctreerast --spconv --mipgaussian --kaolin --nvdiffrast
```

The README notes that the detailed usage of setup.sh can be found in the TRELLIS repository, which is a fair sign of how much of the setup is inherited rather than project-specific. The flags cover attention kernels, sparse convolution, Gaussian rasterisation, Kaolin and nvdiffrast, so this is a build-from-source environment rather than a pip install.

The Qwen2.5 dependencies are then added separately:

```bash
pip install transformers==4.50.0
pip install qwen-vl-utils
pip install 'accelerate>=0.26.0'
```

There is a simpler alternative for inference only, using the released requirements file:

```bash
conda create -n physx-anything python=3.10
conda activate physx-anything
pip install -r requirements.txt
```

The requirements file itself is long and pinned exactly, with versions such as torch-adjacent CUDA packages, diffusers at 0.20.2, flash-attn at 2.5.8 and triton-dependent entries, so a clean install is a sizeable download either way.

## Fine-tuning the vision model on your own data

Training is a four-step path, and it starts with the datasets rather than the code.

You download the PhysX datasets from PhysXNet at Caoza/PhysX-3D and from Caoza/PhysX-Mobility on Hugging Face. Then three preprocessing scripts run in the dataset directory:

```bash
cd dataset
python 1voxel.py
python 2encode_representation_32_finetune.py
python 3generate_data_new_32_finetune.py
```

The names describe the stages, from voxelising to encoding the representation to generating the training data, and the README points at a template, training_data_template.json, that you can use to check the format.

The dataset paths are set in the training configuration at qwen-vl-finetune/qwenvl/data/__init__.py, and the structure is a dictionary per dataset with the annotation file path and the conditioning image path:

```python
PHYSXNET = {
    "annotation_path": "xx", #json file path
    "data_path": "xx",  # conditioning image path
}

PHYSXMOBILITY = {
    "annotation_path": "xx", #json file path
    "data_path": "xx",  # conditioning image path
}
```

Finally the fine-tune is launched, and the command is sbatch, so the script is written for a Slurm cluster:

```bash
cd qwen-vl-finetune
sbatch scripts/sft_7b.sh
```

A seven-billion-parameter model fine-tune on a cluster scheduler is the honest description of what this step is. The 7b in the script name is the giveaway, and it is a different order of task from running inference.

## Evaluating the assets inside a physics engine

The point of a simulation-ready asset is that you can load it and measure something, and the evaluation scripts close that loop.

Rendering the generated URDF files is the first step:

```bash
python render_urdf.py
```

The kinematic evaluation is then run with the VLM-based evaluation script:

```bash
python evaluation_kine.py
```

and for all other physical attributes the README directs you to a separate script:

```bash
python evaluation_phy.py
```

That split is a useful signal about what the project claims to verify. Kinematics, meaning how parts move and relate, is checked with a VLM-based evaluator, while the other physical properties, mass, friction, contact behaviour, are assessed by a dedicated script rather than by a learned judge.

The caveat that the README puts in bold is about deformable parts. Although the method can generate parts with physical deformable parameters, the deformable components are not stable in MuJoCo, so the recommendation is to set the deformable flag to 0 to obtain more reliable simulation results. In other words, the deformable branch exists in the code and in the paper, and it is turned off for the runs you are told to trust.

## Where this pipeline stops short

Several constraints are structural, and a researcher planning around them will save weeks.

The first is the environment. flash-attn, spconv, kaolin, nvdiffrast and xformers have to compile against a specific CUDA and PyTorch combination, the setup script is inherited from another project, and the requirements file pins everything exactly. Reproducing the environment on a new machine is a project in itself, and a machine without a suitable GPU is simply out of scope.

The second is the deformable-part instability, which the README itself flags. Third, the input distribution: the conditioning is a canonical set of 25 rendered views, so the system expects an object photographed or rendered in a comparable way, and the range of objects is bounded by the datasets.

Fourth, the output is an approximation with parts and physical parameters inferred rather than measured, so it is a starting asset for simulation research rather than an accurate digital twin of a real object. The evaluation scripts exist precisely because the quality of those inferences has to be measured rather than assumed.

Fifth, the project is a research artefact rather than a maintained tool. There are no GitHub releases, so you install from the repository, the last push was on 2026-04-28, and the fine-tuning path assumes a Slurm cluster. The licence is a custom S-Lab License that GitHub cannot classify, so read the LICENSE file for the actual terms.

## Against authoring the asset by hand

The alternative is to build the URDF and MJCF yourself, which is what a robotics simulation team does today when it has a specific object and specific parameters. You model the parts in a CAD tool, write the joint and inertial tags, tune the contact parameters against the simulator, and accept the days of iteration that tuning takes.

The difference in approach is where the physical parameters come from. In the hand-authored case every mass, friction coefficient and joint limit is a decision you make and defend. In this pipeline they are inferred from visual evidence and geometry, which is the interesting idea, and it is also the source of the risk: an inferred parameter can be plausible and wrong, and a simulator will happily integrate a plausible and wrong value.

That is exactly why the evaluation scripts are part of the repository rather than an afterthought, and why the deformable flag is set to 0 in the recommended configuration. The pipeline is a proposal, and the evaluation is how you check the proposal before you trust it.

For a research project that needs thousands of varied objects and can tolerate approximate parameters, generating them from images is a different order of magnitude of throughput than modelling them. For a fixed, small set of objects whose properties must match the real world to a tolerance, a hand-authored asset with measured parameters is more trustworthy, and this pipeline will not replace that.

## Conclusion

Adopt PhysX-Anything if your research needs simulation-ready assets from photographs, and if you can afford the environment it asks for, with CUDA-era extensions such as flash-attn, xformers, spconv, kaolin and nvdiffrast installed through setup.sh. Do not adopt it expecting stable deformable parts, because the README states the deformable components are not stable in MuJoCo and recommends the deformable flag set to 0. Verify first that the pretrained checkpoint downloads and that your generated URDF loads in your simulator, and read the LICENSE file, since the project is distributed under a custom S-Lab License rather than a standard identifier. The last push was on 2026-04-28 and the repository publishes no GitHub releases.

## FAQ

### What does PhysX-Anything generate from an image?

It produces a simulation-ready asset: the pipeline reconstructs the object, splits it into parts and converts the result to URDF and XML with physical parameters. The final script also takes a voxel resolution, a results path and flags for postprocessing, fixing the base and enabling deformable parts.

### How do I install PhysX-Anything?

Clone the repository with submodules, then run the bundled setup script with the extension flags, for example . ./setup.sh --new-env --basic --xformers --flash-attn --diffoctreerast --spconv --mipgaussian --kaolin --nvdiffrast, followed by the Qwen2.5 packages. For inference only, the README also offers a conda environment with python=3.10 and pip install -r requirements.txt.

### How many images are needed to condition the generation?

The README states 25 images per object, rendered based on your requirements, using dataset_toolkits/render_cond_mobility.py for PhysX-Mobility. For PhysXNet it points to the preprocessing script in the PhysX-3D repository.

### Should I enable the deformable parts flag in PhysX-Anything?

The README states that the deformable components are not stable in MuJoCo and recommends setting the deformable flag to 0 for more reliable simulation results, even though the method can generate parts with physical deformable parameters.

### How do I fine-tune the model on my own data?

Download the PhysX datasets, run the preprocessing scripts 1voxel.py, 2encode_representation_32_finetune.py and 3generate_data_new_32_finetune.py in the dataset directory, set the annotation and image paths in qwen-vl-finetune/qwenvl/data/__init__.py, then launch sbatch scripts/sft_7b.sh from the qwen-vl-finetune directory.

### How are the generated assets evaluated?

Render the generated URDF files with python render_urdf.py, run the VLM-based kinematic evaluation with python evaluation_kine.py, and use python evaluation_phy.py for all other physical attributes.

## Sources

- [Issues](https://github.com/ziangcao0312/PhysX-Anything/issues)
- [Project website](https://physx-anything.github.io/)
- [README](https://github.com/ziangcao0312/PhysX-Anything/blob/main/README.md)
- [ziangcao0312/PhysX-Anything on GitHub](https://github.com/ziangcao0312/PhysX-Anything)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/ziangcao0312-physx-anything