PyPose: Lie Group Tensors and Second-Order Optimizers Inside PyTorch

The gap PyPose fills between perception models and pose solvers

A convolutional network can regress a rotation matrix that is not orthogonal. A pose graph solver can produce a rotation that is perfectly orthogonal but cannot be trained with backpropagation. PyPose is aimed at the space in between. The README frames the motivation directly: deep learning generalizes poorly to changing environments, while physics-based optimization generalizes better but lacks semantic information and requires manual parameter tuning. The library's stated goal is to combine the two, and the mechanism for doing so is to make the manifold operations themselves differentiable PyTorch operations.

The intended user is not a general machine learning practitioner. It is someone who already writes PyTorch modules and needs SE3 or SO3 quantities to be first-class citizens in those modules, with gradients, batching and device placement handled the same way tensors are. The topic list confirms the scope: pose estimation, SLAM, Kalman filtering, planning, control. If your work never touches a rotation or a rigid transform, the library has little to offer you.

LieTensor: typed manifolds that behave like torch.Tensor

The core abstraction is LieTensor. The README lists Lie groups SO3, SE3, Sim3 and RxSO3, and Lie algebras so3, se3, sim3 and rxso3. A LieTensor is not a plain tensor with a convention attached. It carries a type, so the printed representation of pp.randn_so3(2) shows so3Type LieTensor with a 3-vector per batch element, and calling .Exp() on it returns an SO3Type LieTensor with a 4-vector per element, the quaternion. The type tag is what lets the library decide which exponential map, which inverse and which adjoint to apply.

Batching is the design centre. In the first README example, pp.randn_so3(2, requires_grad=True) produces two Lie algebra elements at once, .Exp() maps both to rotations, the @ operator rotates a random point, and p.sum().backward() fills r.grad with a 3-vector per batch element. That is the whole pitch in eight lines: manifold operations that participate in autograd without a custom backward pass written by the user. The README also claims parallel computation for the Jacobian of Lie group and Lie algebra operations, with an efficiency and memory comparison against Theseus normalized to 1x, pointing at the PyPose paper for detail. I have not reproduced that comparison, and the chart alone should not be treated as a settled result.

Optimizers, strategies and schedulers as separate objects

PyPose splits second-order optimization into three collaborating pieces, which is the part of the API most likely to shape how you write code. The optimizer is GaussNewton or LevenbergMarquardt. The strategy is the damping policy, for example Constant(damping=1e-4) or TrustRegion. The scheduler decides when to stop, for example StopOnPlateau(optimizer, steps=10, patience=3, decreasing=1e-3, verbose=True).

The README's second example builds an InvNet whose parameter is a pp.Parameter wrapping pp.randn_SE3(2, 2), and whose forward pass computes (self.pose @ input).Log(). The loss is the log of the residual transform, which is the standard way to turn a pose error into a vector you can minimize. Two usage modes are shown and the README explicitly says to remove one: either scheduler.optimize(input=input) runs the full loop, or a while scheduler.continual() loop calls optimizer.step(input) and scheduler.step(loss) manually. The second form matters when you want to interleave optimization with something else, such as a training step or a data loader, and it is the reason the scheduler exposes continual() rather than hiding the loop.

Separating the strategy from the optimizer is a deliberate choice. You can swap Constant damping for TrustRegion without touching the residual definition. The cost is more objects to wire up, and the README does not show a table of which strategy suits which problem, so the choice is left to the reader.

Sparse Jacobian tracing arrives in v0.9.5

The release notes for v0.9.5, dated April 2026, describe sparse Jacobian tracing as new, aimed at sparse second-order optimization and specifically at bundle adjustment. The README's third example shows the API. A parameter is declared with pp.Parameter(poses, sjac=True), and the function that assembles the residual is decorated with @psjac, described as parallelizing the assembly of the sparse Jacobian. The example then imports PCG from pypose.optim.solver and TrustRegion from pypose.optim.strategy, which suggests the intended combination is a conjugate-gradient solver with a trust-region damping policy for large sparse problems.

The example is truncated in the supplied material, so the full forward signature and the meaning of the cidx and pidx arguments cannot be confirmed. What is clear is the direction: dense Jacobians for bundle adjustment do not scale, and PyPose is adding an explicit opt-in path for sparsity rather than changing the default behaviour. Since this is a v0.9.5 feature, code written against v0.9.0 will not have it, and the sjac and psjac names should be treated as new API surface that may still move.

Installing PyPose and what the commands actually do

The simplest path is pip install pypose. From source, the README requires PyTorch to be installed first, then pip install -r requirements/runtime.txt, then git clone https://github.com/pypose/pypose.git, cd pypose && pip install -e ., and pytest to run the test suite. The editable install is the right choice if you intend to read the source, since the manifold implementations are where the interesting decisions live.

Note the ordering. PyTorch is a prerequisite you install yourself, and the README points at pytorch.org rather than pinning a version. That means the runtime requirements file is the only place a version constraint could appear, and the supplied material does not show its contents. If you are on a CUDA build that lags the current PyTorch release, check compatibility before assuming pip install pypose will resolve cleanly.

The examples use torch.device("cuda") directly, so the library expects a GPU to be available for the optimizer path shown. Nothing in the README states that CPU-only execution is unsupported, but the shown examples are written for CUDA, and that is what the maintainers appear to exercise.

Where PyPose is the wrong tool

PyPose is a research-oriented library, and several of its design choices follow from that. The optimizer loop is Python. Even with parallel Jacobian assembly, a Levenberg-Marquardt step involves a strategy object computing damping, a solver, and a scheduler checking for plateau. If your application needs a pose graph optimized at a frequency where Python overhead is visible, a compiled solver will beat this, and the README makes no claim otherwise.

The feature list also reveals what is not there. There is no loop closure module, no map representation, no place recognition, no sensor driver, no ROS integration, and no visualization. The topics mention SLAM, but the README's contribution to SLAM is the components (IMU preintegration, PnP, Lie group operations, second-order optimizers), not a complete SLAM system. If you want a working SLAM pipeline rather than the building blocks for one, this is the wrong layer.

The version history is a further consideration. Three releases appear in the supplied material, v0.7.5 in December 2025, v0.9.0 in April 2026, and v0.9.5 four days later. That cadence is fast, and the jump from 0.7 to 0.9 in four months suggests API movement. There is no statement of a stability guarantee for the pre-1.0 API, and the sparse Jacobian feature is brand new. Treat any code you write against sjac as likely to need revision.

How PyPose differs from Theseus and from Ceres-style solvers

The README's own efficiency chart compares PyPose against Theseus, normalizing Theseus to 1x for batched Lie group operations. Theseus is the closest analogue: a library that also puts differentiable nonlinear optimization on manifolds inside PyTorch. The difference visible in the material is one of emphasis. PyPose ships a named catalogue of robotics modules (LTI, LTV, NLS, EKF, UKF, PF, EPnP, LQR, IMUPreintegrator) alongside its optimizers, so the library expects you to build a filtering or control pipeline, not only a least-squares problem. Theseus is oriented toward the optimization layer itself.

Against Ceres Solver, the difference is categorical rather than incremental. Ceres is C++, has no autograd, and expects analytic or automatically differentiated Jacobians outside a Python runtime. PyPose's reason to exist is that the residual is a torch.nn.Module and the parameter is a pp.Parameter, so a learned component can sit inside the same objective as a geometric one. If you have no learned component and no Python in the loop, Ceres remains the more direct answer, and nothing in the PyPose material argues otherwise.

The comparison chart in the README is the maintainers' own, taken from their paper. It is a starting point for your own measurement, not a substitute for one.

Licence, maintenance and what to check before adopting

PyPose is Apache-2.0. That is a permissive licence with an explicit patent grant, which matters for robotics code that may ship in a product. It does not oblige you to publish modifications. This is a description of the licence identifier, not legal advice; if patent or redistribution terms affect your situation, read the full text with counsel.

Maintenance cost is dominated by the pre-1.0 version line. The supplied material shows v0.7.5, v0.9.0 and v0.9.5 within roughly nine months, with the last push in September 2026. A library moving that fast will occasionally rename or restructure optim modules, and the sparse Jacobian API is the newest and least settled part. Budget for reading release notes before each upgrade rather than assuming pip install --upgrade pypose is safe.

What to verify first, concretely: run the README's first example and confirm that r.grad is populated with the shape shown, which tests the autograd path through Exp() and the @ operator on your PyTorch build. Then run the second example with the scheduler.optimize path and confirm that StopOnPlateau terminates. If you intend to use the v0.9.5 sparse path, check whether PCG and the sjac parameter flag exist in the version you installed, because the README example is truncated and the full signature is not documented in the supplied material.

Editorial conclusion

Adopt PyPose if your pipeline is already PyTorch and you need pose parameters to sit inside a module that a second-order optimizer can refine. Do not adopt it if you need production-grade SLAM back ends with loop closure at scale, or if you cannot accept a Python-level optimizer loop. Before committing, verify that your PyTorch version matches the one the release was built against, and run the two README examples verbatim to confirm that gradients flow through Exp() and through the LM scheduler on your hardware.

Official sources

1. License: Apache-2.0
2. Project website
3. pypose/pypose on GitHub
4. README
5. Releases