DeepLens: a differentiable optical lens simulator for PyTorch workflows

The lens as a set of learnable parameters inside a training loop

DeepLens is a differentiable optical lens simulator for computational imaging and optical design, and the reason it exists is stated plainly in the README: commercial optical design software such as Zemax and Code V is built for a human designer clicking through a GUI, whereas DeepLens is built for optics that live inside a PyTorch training loop, where the lens is a set of learnable parameters optimised jointly with a neural network and gradients flow through the optics.

That is a different goal from what a desktop design tool does, and it is worth being clear about the difference. A GUI tool produces a prescription and stops. Here, a surface curvature, a thickness or an aperture becomes a tensor, the optical forward model is a differentiable graph, and the optimiser can move the lens geometry and the image reconstruction network at the same time.

The README lists three intended uses: end-to-end optics-algorithm co-design, automated optical design, and image simulation for synthetic datasets. The repository reinforces the research framing with an AGENTS.md at the top level and a CITATION.cff and CITATION.md, so the project is set up to be cited and to be driven by an automated agent as well as by a person.

The package is published on PyPI as deeplens-core, the licence is Apache-2.0, the documentation lives at ai4optics.github.io, and the last push was on 2026-10-16, with tagged releases v2.5.5 on 2026-09-30, v2.5.4 on 2026-09-12 and v2.5.3 on 2026-09-12.

Four optical models, and the trade-off each one makes

DeepLens supports multiple optical models, and the choice among them is the main design decision a user makes.

Ray tracing is the geometric model, and it is the basis for the analysis features the README lists: spot diagram, PSF, MTF, distortion, and photorealistic image simulation with spatially varying, depth-dependent aberrations. Wave propagation is the model that treats light as a wave and is what diffraction and interference behaviour depends on.

The hybrid ray-wave model sits between them, described in the README as a differentiable ray-wave optical model for accurate lens aberration and diffraction element simulation, supporting end-to-end refractive-diffractive lens design. The point of a hybrid is that a purely geometric model misses what happens at a diffractive element, and a purely wave model is expensive and awkward for a full multi-element system.

The fourth is the surrogate PSF network, described as a surrogate network for efficient lens PSF representation, supporting fast and accurate image simulation with spatially varying aberrations and defocus. This is a learned approximation of the optical response, and it is the one option in the list whose fidelity depends on training data rather than on physics.

A fifth capability sits alongside the models: non-sequential polarization tracing, used to simulate the polarization state of light passing through a geometric waveguide AR display, with end-to-end optimization for coating film inverse design targeting the out-coupling eyebox response.

Installing from a clone and running the first example

The installation path is a clone plus a conda environment. The README asks for Python 3.12, and notes that on Linux and Windows with NVIDIA GPUs the requirements install PyTorch 2.10.0+cu128 and torchvision 0.25.0+cu128, while on Apple silicon macOS they install the native PyTorch 2.10.0 and torchvision 0.25.0 wheels.

git clone https://github.com/vccimaging/DeepLens
cd DeepLens

Then the environment, either created by hand.

conda create -n deeplens_env python=3.12
conda activate deeplens_env

python -m pip install -r requirements.txt

Or created in one step from the environment file in the repository root.

conda env create -f environment.yml -n deeplens_env
conda activate deeplens_env

The first thing to run is the demo script.

python 0_hello_geolens.py

The top level of the repository carries four hello scripts, one per model family, named for geometric, diffractive, hybrid and defocus lenses. There are more examples in a scripts directory, and the README asks for them to be run as modules from the repository root, with the geometric design example given as the command.

python -m scripts.1_design_geolens

The module form is not a style preference. The script filenames begin with a digit and contain dots, so they cannot be imported as ordinary modules by name, and running them with the -m flag from the repository root is what makes the package importable. A user who tries python scripts/1_design_geolens.py from another directory will hit import errors instead of the example.

What the comparison table claims, and what it leaves out

The README includes a table contrasting DeepLens with Zemax and Code V, and it is worth reading the rows closely because each one states a specific difference in approach rather than a vague advantage.

On gradients, the difference is analytic, via autograd, against finite differences. That is the core technical claim, and it is the one that matters most: finite differences estimate a derivative by perturbing a parameter and re-evaluating, which is expensive and noisy, while autograd computes the derivative of the optical model directly, so a lens can be optimised with the same machinery that trains the network.

On Python, the difference is native, tensors in and tensors out, against an external API bridge. On hardware, the difference is GPU with CUDA or ROCm, batched, against mostly CPU-bound work. End-to-end co-design is listed as not supported in the commercial tools, since optics, ISP and network cannot be placed in one graph there. Agent-scriptability is a plain Python, no GUI in the loop design against a GUI-centric workflow. And the licence row is free under Apache-2.0 against commercial.

What the table does not say is just as informative. It does not claim better optical accuracy, and it does not compare the design tools' mature workflows such as sequential ray tracing with tolerancing and analysis reports. A commercial design package solves a different problem, and a team that needs a manufacturable prescription with tolerances and a drawing is not looking at this table.

The DeepLens side of the comparison is also narrower than the tool as a whole, because the automated design capability was moved out to a separate project.

Automated lens design now lives in the AutoLens repository

One of the six application sections in the README is Automated Geometric Lens Design, described as fully automated lens design from scratch with gradient-based optimization and advanced optimization algorithms. Directly under it sits a note that changes the picture.

The note says automated lens design is now actively maintained in the AutoLens project, and that if your focus is automated lens design, the recommendation is to use the AutoLens repository instead, because it receives dedicated updates and improvements for that use case.

So DeepLens is being repositioned around its actual strength, which is the differentiable simulation core, with the automated design workflow forked out into a dedicated repository. The README links an AutoLens quickstart badge and a Nature Communications 2024 paper badge alongside the animated examples for that path.

For an evaluator this is a useful signal rather than a problem. It tells you that the simulation engine is the durable part of the project and the design automation is a separate, more focused codebase. It also means a team whose real goal is automated lens design should start from AutoLens, and that a team whose goal is joint optimisation of optics and an algorithm is looking in the right place with DeepLens.

The related projects follow the same pattern of separation by function. End2endImaging is a distinct end-to-end differentiable computational imaging framework where DeepLens serves as the differentiable optics engine, integrating optics, sensor and ISP simulation, and neural reconstruction networks into a single PyTorch computation graph for joint optimization of the entire camera pipeline. A surrogate PSF approach is associated with a separate project on aberration aware depth from focus.

A Python 3.12 only package with a curated dependency list

The packaging tells you what kind of project this is and what it will cost to install.

The distribution is named deeplens-core, the build backend is setuptools with a requirement of setuptools 61.0 or later, and the version in the project metadata is 2.5.6, one patch ahead of the newest tag in the release list. The version is a plain literal rather than dynamic, and the project is classified as Beta with an Intended Audience of Science and Research.

The Python constraint is the strictest thing in the file: requires-python is set to a range of 3.12 up to but not including 3.13. That is a single minor version, and it matches the README instruction to use Python 3.12. A team on 3.11 or 3.13 cannot install the published package without changing interpreters, which is worth knowing before a plan depends on it.

The runtime dependencies are compact: torch at 2.10 or later, torchvision at 0.25 or later, opencv-python, matplotlib, scikit-image, lpips, einops and tqdm. The requirements file pins the platform specific wheels, using the cu128 build of PyTorch on Linux and Windows and the native build on macOS, and pulls an extra index URL from the PyTorch download server. It also includes pyyaml, jupyter, notebook and ipywidgets, which the comments in the project metadata identify as needed by the config driven scripts rather than as library runtime dependencies, and the requirements file notes that pyyaml is used by two of the scripts and to regenerate a refractive index data file.

Optional dependency groups add ruff, jupyter, notebook, ipywidgets and pyyaml for development, and a set of documentation tooling packages for building the site, including mkdocs, mkdocstrings and pymdown-extensions.

In-house modules exist but are not part of the public release

The README has a section called In-House Extensions, and it is short but consequential. The modules below it are described as implemented and working in house, but not part of the public release yet, and the project says that if your project needs one, you can get in touch and the maintainers are happy to grant access or collaborate.

The list is cut off in the README as published, so the specific modules are not enumerable from the documentation, but the first entry is visible as a GPU related item. The general point stands regardless of how many there are and what they are. This is a research codebase where some working code is deliberately held back, and access is arranged case by case rather than through the repository.

For an adopter that is a procurement question before it is a technical one. You cannot depend on an in-house module the way you can depend on the published package, because you cannot see it, you cannot test it in advance, and there is no version or release process behind it. If a capability you need sits in that list, the conversation with the maintainers is the first step, and the timeline of that conversation is outside your control.

Everything else in the repository is open. The top level directory listing shows the public hello scripts for the different lens types, the deeplens package itself, a scripts directory of examples, a configs directory, a datasets directory, a test directory, and a pyproject.toml. The presence of a test directory and a dedicated test folder in the tree says the project expects its code to be exercised, even though the README does not describe how to run the suite.

Editorial conclusion

DeepLens fits a research team that wants the lens inside the training loop, so that optical parameters and a neural network are optimised against the same objective, and it does not fit an optical designer who needs a mature sequential design workflow with tolerancing and a familiar interface. Before building on it, confirm your Python version is 3.12, since the package metadata constrains requires-python to the 3.12 series only, and read the model list before choosing a backend, because the four optical models differ in what they can represent and the automated lens design work has been moved out to the separate AutoLens project.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. vccimaging/DeepLens on GitHub