lion-pytorch: A Wrapper Around Google's Evolved Sign Momentum Optimizer
🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch
At a glance
- What is it?
- lucidrains/lion-pytorch packages the Lion optimizer from Google Brain's symbolic discovery paper as a drop-in PyTorch optimizer. It is a thin, MIT-licensed implementation whose main value is the usage notes on learning rates and batch sizes, not the code volume.
- Who is it for?
- Adopt lion-pytorch if you are training a large-batch language or text-to-image model and are prepared to tune the learning rate specifically for Lion rather than reusing an AdamW schedule. Do not adopt it for reinforcement learning, feedforward networks, or hybrid LSTM-convolution architectures, where the README reports negative results, and do not adopt it if your batch size is below 64.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 69 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What Lion Changes and Who the Package Is For
Lion stands for Evolved Sign Momentum, an optimizer discovered by Google Brain through a program search process, described in the arXiv paper the README cites as Symbolic Discovery of Optimization Algorithms. The discovery method is the interesting part: rather than deriving an update rule from theory, the authors searched a space of symbolic programs and arrived at a rule that uses the sign of an interpolation between the gradient and a momentum term. The practical consequence is that the update is bounded in magnitude per coordinate, so the optimizer carries only momentum state and no second-moment estimate.
lucidrains/lion-pytorch exists to make that rule available as a normal PyTorch optimizer. The README is candid about the scope: the implementation is described as nearly a straight copy from Google's own lion_pytorch.py in the automl repository, with minor modifications. So the package is not a research contribution. It is a distribution channel. That matters when you decide whether to depend on it. If you want the reference behavior, this is close to it. If you want a maintained fork with its own algorithmic ideas, this is not that.
The target user is someone already training a model in PyTorch who wants to swap the optimizer line and see what happens. The README's own framing is that the optimizer is simple enough that it may as well be made accessible quickly. The author's update log, however, tells a more specific story: positive results for language modeling and text-to-image training, negative results for problems and architectures outside what the paper evaluated, and a recommendation to use it only at batch sizes of 64 or above.
Installation and the Two Code Paths
Installation is a single command, either pip install lion-pytorch or conda install lion-pytorch. The import is from lion_pytorch import Lion, and the constructor takes model.parameters() alongside the usual lr and weight_decay arguments. The README's toy example constructs a nn.Linear(10, 1), instantiates Lion with lr=1e-4 and weight_decay=1e-2, then runs the standard backward, step, zero_grad sequence.
There is a second path. Setting use_triton=True in the constructor routes the parameter update through a CUDA kernel written in Triton, the language from the Tillet et al. paper cited in the README. That path requires a separate install step first: pip install triton -U --pre. The --pre flag is worth noticing. You are pulling a pre-release Triton build, which means version drift between Triton and your CUDA toolkit is a real possibility and not something the package can insulate you from. If you are on CPU or on a non-CUDA accelerator, the Triton path is not available to you at all, and you fall back to the eager implementation.
Nothing in the README describes a CLI, a configuration file, or any environment variables. There are no config keys beyond the constructor arguments. This is a library you call from Python, and that is the whole interface surface.
The Hyperparameter Rules Are the Actual Documentation
The most useful content in this repository is not code. It is the transcription of Section 5 of the paper into practical guidance, plus the author's own contradicting experiments.
The paper's rule, quoted in the README, is that a suitable learning rate for Lion is typically 3 to 10 times smaller than for AdamW, and that because effective weight decay is lr * lambda, the decoupled weight decay coefficient should be 3 to 10 times larger to keep the same strength. The README adds a constraint the paper states less prominently: initial, peak, and end values in the learning rate schedule should all be rescaled simultaneously by the same ratio. If you scale the peak but leave the warmup start where it was, you have changed the shape of the schedule, not just its magnitude.
The defaults differ too. AdamW uses beta1 0.9 and beta2 0.999 with epsilon 1e-8. Lion's defaults, found by the search process, are beta1 0.9 and beta2 0.99. The README relays the authors' suggestion that beta1=0.95, beta2=0.98 can help mitigate instability, drawing the parallel to the common practice of lowering beta2 and raising epsilon in AdamW.
Then the author's own update log complicates the paper's rule. Update 3 reports that dividing the learning rate by 3 gave better early results than Adam. Update 4 reports that the paper's 10x-smaller rule of thumb produced the worst run. The summarized position is that Lion at 3x smaller beats Adam, but that 10x smaller is worse. Treat the paper's 3-to-10x range as a search interval, not a setting.
Where Lion Fails and Where It Is the Wrong Tool
The README's update log is unusually direct about failure. Update 5 states that negative results cluster in problems and architectures outside what the paper evaluated: reinforcement learning, feedforward networks, and hybrid architectures combining LSTMs with convolutions. The same update notes that negative reports point to sensitivity in batch size and in the amount of data or augmentation.
Update 7 converts that sensitivity into a hard recommendation: use this optimizer only in the setting of high batch sizes, 64 or above. That is a real constraint, not a stylistic preference. If your training loop runs at batch size 32 because of memory limits, the README's own guidance says this is not your optimizer.
Update 2 records an early experiment where Lion looked much worse than Adam when the learning rate was held constant. That is the failure mode in miniature: Lion's advantage depends on retuning, and if you treat it as a drop-in replacement with an unchanged schedule you may conclude it is worse when you have simply misconfigured it. The inverse error is also available. Update 4 shows that over-applying the paper's ratio produces the worst result in the author's runs.
There is one more scaling caveat. Update 5 mentions a positive result at open-clip that turned negative as model size increased, later resolved by the author of that project setting a higher initial temperature. So at least one reported failure was not intrinsic to Lion. That cuts both ways: it means some negative reports elsewhere may also be configuration artifacts, and it means you cannot assume a failure you observe is the optimizer's fault without checking the surrounding hyperparameters.
How lion-pytorch Compares to Using torch.optim.AdamW Directly
The most honest alternative is not another optimizer package. It is torch.optim.AdamW, which ships with PyTorch, needs no third-party dependency, and has a decade of published hyperparameter recipes attached to it. The difference in approach is state and update shape. AdamW maintains both a first and a second moment estimate per parameter and divides the update by the square root of the second moment. Lion maintains only momentum and applies the sign of an interpolated direction, so the per-coordinate step size is uniform rather than adaptive to gradient magnitude. That is why the learning rate must be smaller: the sign operation removes the automatic scaling that AdamW's denominator provides.
If what you actually want is a fused, low-overhead optimizer step rather than Lion specifically, the Triton kernel here is not the only route. PyTorch ships fused implementations of its own optimizers, and those carry no pre-release dependency. Choosing lion-pytorch for the fused kernel alone would mean accepting an unstable Triton install requirement in exchange for something the framework already offers.
If you want Lion but not this wrapper, Google's automl repository contains the file the README says this package copies from. Depending on that directly means tracking a monorepo rather than a small pip package, which is a different kind of maintenance burden, not obviously a smaller one. The package's own citation list also points at follow-up work on cautious optimizers and cautious weight decay, which suggests the Lion line has continued to develop outside this repository. If you need those variants, this package is not where you will find them.
Maintenance, Versioning and Licence
The package is MIT licensed, which permits commercial use and modification provided the copyright notice and permission notice are retained. That is a permissive arrangement, and it is the same licence family as PyTorch itself, so there is no licence interaction to think about when combining them. This is a description of the licence text, not legal advice; if your organization has specific obligations around attribution in distributed binaries, read the LICENSE file.
Version history shows releases at 0.2.1 and 0.2.2 on the same day in June 2024, followed by 0.2.3 in November 2024. The repository is not archived and shows a push in 2026, but the release cadence suggests the API is stable rather than actively expanding. For a library this small, that is reasonable. The optimizer has a fixed number of constructor arguments, and the algorithm is not changing.
The upgrade risk is concentrated in the optional Triton path rather than in the package's own code. Because the README instructs installing Triton with --pre, a future Triton release can change kernel compilation behavior without any corresponding change in lion-pytorch. If you use use_triton=True in production, pin your Triton version explicitly and treat Triton upgrades as a separate change to test, not something that rides along with a lion-pytorch version bump. The eager path has no such exposure.
Editorial conclusion
Adopt lion-pytorch if you are training a large-batch language or text-to-image model and are prepared to tune the learning rate specifically for Lion rather than reusing an AdamW schedule. Do not adopt it for reinforcement learning, feedforward networks, or hybrid LSTM-convolution architectures, where the README reports negative results, and do not adopt it if your batch size is below 64. Before committing, verify two things in your own setup: that a learning rate roughly 3x smaller than your AdamW value produces better early loss than Adam, and that the Triton fused kernel path installs cleanly on your CUDA version, since the README requires a pre-release Triton build.
Community notes