Autograd: Reverse-Mode Gradients on Ordinary NumPy Code

The gap Autograd fills: gradients for code you already wrote

Most automatic differentiation libraries ask you to write your model against their array type and their execution model. Autograd asks for neither. The README states that it can differentiate native Python and NumPy code, handling loops, ifs, recursion and closures, and that it can take derivatives of derivatives of derivatives. The intended application named in the README is gradient-based optimization.

The audience is therefore narrower than a general deep learning framework. It suits researchers and engineers who have a numerical function expressed in NumPy and want its gradient for an optimizer, a Hamiltonian Monte Carlo sampler, or a sensitivity analysis. The README points at Sampyl, a pure Python MCMC package with HMC and NUTS, as an example of that pattern. It also lists a fluid simulation example, which is a useful signal: the library is aimed at scientific code with control flow, not only at neural networks.

If your function is already a chain of NumPy ufunc calls, the adoption cost is close to zero. If it is a mix of NumPy, hand-written C extensions and third-party libraries with opaque internals, the cost is higher, because Autograd can only differentiate through operations it knows how to record.

Tracing, not symbolic rewriting: how the gradient actually gets built

The README example shows the surface API: import autograd.numpy as np, described in the source comment as thinly-wrapped numpy, then from autograd import grad. You define a function using that wrapped NumPy and call grad(tanh) to obtain a new function. Calling grad_tanh(1.0) returns np.float64(0.419974341614026), which the README compares against a finite-difference estimate of 0.41997434264973155.

That wrapping is the mechanism. autograd.numpy exposes the same names as NumPy but each one records itself on a trace as the function executes. When you call the gradient function, Autograd replays the recorded trace backwards, applying the vector-Jacobian product for each recorded primitive. This is why Python control flow works: a loop or an if is executed normally during the forward pass, and the recorded trace captures only the operations that actually ran. The README describes this as reverse-mode differentiation, also called backpropagation, which is efficient for scalar-valued functions of array-valued arguments.

The README also states that forward-mode differentiation is supported and that the two modes can be composed arbitrarily. That composition is what makes higher derivatives practical. The tanh example in the README builds a fourth derivative by nesting elementwise_grad four times: egrad(egrad(egrad(egrad(tanh)))). There is no separate higher-order API to learn; nesting is the API.

The cost of tracing is that it happens at Python speed. Every recorded operation carries bookkeeping, and every gradient evaluation re-executes the forward function. The README does not publish throughput numbers, and the asv badge in the header only indicates that benchmarks exist, not what they show.

Installing Autograd and the scipy extra

Installation is a single pip command. The README gives:

pip install autograd

Some features require SciPy. The README offers it as an optional dependency installed alongside Autograd:

pip install "autograd[scipy]"

There is no configuration file, no environment variable and no plugin registration step documented in the README. The import line is the configuration: use autograd.numpy in place of numpy inside any function you intend to differentiate. The README's own examples follow that rule, and the neural net, convnet, rnn, lstm, Bayesian neural net, Gaussian process and fluidsim examples in the examples directory are the reference for how far the pattern extends.

The README does not document a supported Python version range, a minimum NumPy version, or any platform constraints. The presence of a publish workflow and a checks workflow in the repository suggests releases are automated, and the recent release list shows v1.9.1 in June 2026, v1.9.0 a week earlier, and v1.8.0 in May 2025. The gap between v1.8.0 and v1.9.0 is roughly thirteen months, which is worth knowing if you depend on prompt fixes for edge cases.

Where the tracing approach breaks down

The limitations follow directly from the mechanism, even where the README does not spell them out. Differentiation works on operations Autograd knows about. Any call that leaves the traced NumPy world, such as a compiled extension, a Cython kernel, or a library that allocates its own buffers, will either raise an error or silently produce a gradient that treats that call as a constant. The README's scipy extra exists precisely because SciPy functions need wrapped equivalents; if you import scipy directly inside a differentiated function, the trace has nothing to record.

The second limitation is performance. Because the forward pass is re-executed in Python on every gradient call, the overhead scales with the number of Python-level operations rather than with the size of the arrays. A vectorized function with a handful of large array operations is fine. A function with a Python loop over a million scalar steps is not, and no amount of NumPy vectorization inside the loop body will fix the interpreter cost.

The third is memory. Reverse mode has to keep the intermediate values from the forward pass alive until the backward pass consumes them. For a long computation on large arrays, that retention can exceed the memory needed by the forward pass alone. The README does not describe checkpointing or any mechanism to trade compute for memory, so this is a constraint to plan around rather than a feature to configure.

Finally, the README does not claim GPU support, distributed execution, or compilation. If your workload needs any of those, Autograd is the wrong tool regardless of how well it fits your code style.

Autograd versus JAX, and what the difference costs you

The topic list on the repository includes jax, and the comparison is unavoidable because JAX grew out of the same research lineage and the same NumPy-as-interface idea. The difference is in what happens after the trace is recorded.

Autograd interprets the recorded trace in Python. JAX traces a function and then hands the result to XLA, which compiles it into a fused kernel that can run on CPU or accelerator. That compilation step is what buys JAX its throughput on large models, and it is also what constrains it: control flow that depends on traced values has to be expressed through constructs like jit-compatible conditionals rather than ordinary Python if statements, because the compiler needs the whole graph before execution.

Autograd keeps ordinary Python semantics. The README's claim about loops, ifs, recursion and closures is exactly the property JAX gives up in exchange for compilation. A function that branches on a runtime value, or recurses to a data-dependent depth, runs under Autograd with no rewriting. The same function under JAX requires either a static branch or a lax control-flow primitive.

The trade is speed and hardware reach against code fidelity. If your function is a fixed sequence of array operations on large tensors, JAX will almost certainly be faster and will run on a GPU. If your function is scientific code with data-dependent control flow, small arrays, and a strong preference for staying in plain NumPy, Autograd will accept code that JAX would force you to restructure. There is no benchmark in the README to quantify the crossover, so the decision has to be made by profiling your own function, not by reading a comparison table.

Maintenance, releases and the MIT licence

The README names the original authors (Dougal Maclaurin, David Duvenaud, Matt Johnson, Jamie Townsend) and states that the package is currently maintained by Agriya Khetarpal, Fabian Joswig and Jamie Townsend. It also credits Jasper Snoek and the HIPS group, Barak Pearlmutter for foundational work on automatic differentiation, and Analog Devices (Lyric Labs) and Samsung Advanced Institute of Technology for support. That is a research-project lineage with a small named maintainer group, and the release cadence reflects it: v1.8.0 in May 2025, then v1.9.0 and v1.9.1 in June 2026.

The upgrade cost is low by design. The public surface described in the README is grad, elementwise_grad and the autograd.numpy wrapper. There is no serialization format, no model file, no checkpoint compatibility contract to manage across versions. Upgrading means changing the pinned version and re-running your tests. The risk of a breaking change is concentrated in the wrapper: if a NumPy function changes signature or a new ufunc appears that Autograd has not wrapped, code that previously worked may fail. The repository's test and check workflows are the place to look for coverage before you upgrade.

The licence is MIT, which permits commercial and closed-source use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive licence with no copyleft obligation and no patent grant clause, which is worth noting if patent exposure matters to your organization. None of this is legal advice; read the LICENSE file in the repository and consult counsel if the distinction matters to you.

What to check before you depend on Autograd

The decisive test is mechanical. Take your existing function, replace the numpy import with autograd.numpy, and call grad on it. If it returns a value that matches a finite-difference check on a few sample points, the library fits your code. The README demonstrates exactly this comparison for tanh, and it is the only correctness signal the documentation offers, so reproduce it on your own function rather than trusting the example.

If the call raises, the error will usually point at an operation the tracer does not recognize. That is your answer about scope: either wrap the operation in terms of autograd.numpy primitives, or accept that this part of the computation has no gradient. If the call succeeds but is slow, profile the forward function and count Python-level operations rather than array elements. A function that is fast in NumPy because it is vectorized will be fast under Autograd; one that is fast because it calls into compiled code will not be differentiable at all.

Two things the README does not answer and that you will have to determine yourself: the supported Python and NumPy version ranges, and whether the scipy extra covers the specific SciPy functions in your code. Both are checkable from the package metadata and the autograd.scipy module contents, and neither should be assumed from the README alone.

Editorial conclusion

Adopt Autograd when your model is already written as NumPy functions with loops, conditionals and recursion, and you want gradients without rewriting it in a framework with its own array type. Do not adopt it if you need GPU execution, JIT compilation, or gradients through very large graphs where per-operation Python overhead dominates. Before committing, verify that your hot loop uses autograd.numpy rather than plain numpy, that any SciPy calls are covered by the autograd.scipy wrappers, and that the v1.9.1 release on PyPI matches the master branch you are reading.

Official sources

1. HIPS/autograd on GitHub
2. Issues
3. License: MIT
4. README
5. Releases