kyegomez/zeta: Modular PyTorch Building Blocks for Transformer Models
Build high-performance AI models with modular building blocks
At a glance
- What is it?
- Zeta is a PyTorch component library that ships attention variants, feedforward blocks, quantization layers and full encoder-decoder stacks under one import. It is useful if you assemble architectures from parts, and awkward if you want a maintained, documented framework.
- Who is it for?
- Adopt zeta if you are prototyping a transformer variant and want attention, feedforward, quantization and encoder-decoder blocks importable from one package, and you are willing to read the source when the documentation stops. Do not adopt it as the foundation of a production training stack: the README promises production readiness, but the release history is thin, the packaging metadata is inconsistent, and no migration or deprecation policy is documented.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 8 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What zeta is for, and who it is actually aimed at
Zeta is a component library, not a training framework. The README describes it as "a modular PyTorch framework designed to simplify the development of AI models by providing reusable, high-performance building blocks", and the package description in pyproject.toml calls them "LEGO blocks for AI". The intended user is someone who already knows what a decoder block looks like and does not want to write multi-query attention, SwiGLU, relative position bias or BitLinear from scratch for the fourth time.
The catalogue in the README covers attention mechanisms (multi-query, sigmoid, flash), mixture-of-experts routing and gating, feedforward networks, normalization and activation layers, quantization, and complete architectures including transformers, encoders, decoders, vision transformers and the PalmE multi-modal model. That is a broad surface for one package. The practical consequence is that zeta competes less with training frameworks and more with the habit of copying a reference implementation into your own repository. If your work is "take a known architecture, change one block, measure the difference", zeta is aimed squarely at you. If your work is "train a model to a deadline", you are borrowing someone else's building blocks and inheriting their bugs.
How the components fit together: imports, submodules and the PalmE example
The mechanism is plain Python composition. Top-level names like MultiQueryAttention are importable from zeta directly, while activation, feedforward and positional modules live under zeta.nn, quantization under zeta.quant, and the larger assembly pieces (AutoRegressiveWrapper, Decoder, Encoder, Transformer, ViTransformerWrapper) under zeta.structs. The repository layout matches: a zeta/ package directory holds the modules, with examples/, tests/ and experimental/ alongside it, plus a stray multi_query_attention.py at the repository root.
The clearest statement of the design is the PalmE example in the README. It builds a vision-language model by passing a ViTransformerWrapper around an Encoder as the image side, and a Transformer around a Decoder as the language side, with constructor flags such as cross_attend=True, alibi_pos_bias=True, rotary_xpos=True, attn_flash=True and qk_norm=True. Nothing in that class is zeta-specific plumbing; it is configuration passed down to layers the library provides. That is the whole architecture of the project: you describe a model as a tree of configured blocks, and zeta supplies the leaves.
The trade-off is visible in the same example. Because everything is constructor arguments, the behaviour of a model is spread across dozens of keyword defaults rather than a config file, and the README does not document what most of those flags do. The example is also truncated in the README, so the decoder's cross-attention wiring is not fully shown. You will be reading zeta/structs to fill that in.
Installing zetascale and running a first attention block
The README gives one installation command and it installs the distribution zetascale, not zeta. That naming split is the first thing to internalise: the PyPI package is zetascale, the import name is zeta.
pip3 install -U zetascaleAfter that, the README's first example constructs multi-query attention and runs a forward pass. The tensor shape is (batch, sequence, dim), the call returns a tuple, and the README prints only the first element.
import torch
from zeta import MultiQueryAttention
model = MultiQueryAttention(dim=512, heads=8)
text = torch.randn(2, 4, 512)
output, _, _ = model(text)
print(output.shape) # torch.Size([2, 4, 512])The comment claims the output shape is torch.Size([2, 4, 512]), so the block is shape-preserving: the same dim you pass in comes back out. If your print shows something else, you are on a different code path than the README describes. A second, smaller example exercises the feedforward module from zeta.nn, where the two positional arguments are input and hidden dimensions and the output takes the hidden size.
import torch
from zeta.nn import FeedForward
model = FeedForward(256, 512, glu=True, post_act_ln=True, dropout=0.2)
x = torch.randn(1, 256)
output = model(x)
print(output.shape) # torch.Size([1, 512])Note the input here is 2D, not 3D. The README does not say whether FeedForward accepts a batch and sequence dimension, so treat the 2D shape as the documented case and check the source before feeding it a (batch, seq, dim) tensor.
Where zeta breaks down: packaging drift and undocumented behaviour
The most concrete problem is metadata drift, and it is not cosmetic. The repository's pyproject.toml declares license = "MIT" and the README badge says MIT, while the GitHub repository metadata for this project reports Apache-2.0. The README's PyPI badge points at the zetascale project, but the badge image URL in the README is for a differently named package. Version numbers disagree too: pyproject.toml declares version 2.8.8, while the most recent release listed for the repository is 2.3.7 from 2024-04-06, preceded by 0.0.111 and 0.0.11 in July 2023. If you pin a version in a lockfile, confirm which artefact you are actually resolving.
The second issue is that the dependency list is long and pinned in places for reasons the project does not explain. requirements.txt carries a comment, "Pin compatible versions to prevent import errors", above joblib>=1.3.0,<1.4.0 and scikit-learn>=1.5.0,<1.6.0. A library that has to fence off scikit-learn minor versions to import cleanly is telling you something about how tightly coupled it is to its transitive dependencies. Installing zeta into an environment that already has a different scikit-learn or transformers version is a realistic source of conflict.
The third is documentation coverage. The README shows six short examples and a truncated PalmE class. It does not document rollback, deprecation, version compatibility, or what the keyword arguments in the larger constructors mean. The project lists a readthedocs homepage and a docs/ directory, but nothing published about zeta describes what those pages contain. For a library whose value proposition is "drop-in replacements", the absence of a stated compatibility policy is the limitation that will bite hardest.
The alternative: writing the blocks yourself, or using a maintained reference implementation
The honest alternative is not another framework. It is the pattern zeta was built to replace: copying a reference implementation of multi-query attention or SwiGLU into your own repository. That approach costs you an afternoon and a test, and it buys you total control. You know exactly which version of the block is in your model, you can change it without waiting for a release, and you never inherit a transitive dependency conflict from someone else's requirements.txt. The topics on this repository point at exactly that lineage.
The difference in approach is about who owns the code. With zeta you import MultiQueryAttention, FeedForward and BitLinear and accept the project's defaults, its dependency graph and its release cadence. With a local copy you own a hundred lines per block and the maintenance that comes with them. For a research prototype where you are trying several attention variants in a week, importing is faster. For a model you intend to train repeatedly for a year, a local copy of the three blocks you actually use is usually less risk than a dependency on a package whose version numbers and licence metadata do not agree with each other.
A middle path is available: use zeta as a reference for what the block should look like, then vendor the specific module you need from the zeta/ package directory into your own tree.
Maintenance cost, release cadence and licence questions to settle before adopting
The last push to the repository was on 2026-09-07, so the project is not dormant. The release history tells a different story: the most recent listed release is 2.3.7 from 2024-04-06, with nothing between then and the recent commits. That pattern, commits without releases, means the version you install from PyPI may not match the code on master. If you depend on a fix, check whether it exists in a release or only in the repository.
Upgrade cost is dominated by the dependency pins rather than by zeta's own API. Because requirements.txt pins scikit-learn and joblib to narrow ranges to avoid import errors, an upgrade of zeta can force an upgrade or downgrade of those packages in your environment. The pyproject.toml also constrains transformers to >=4.20.0,<5.0.0, which is a wide band, and leaves torch, torchvision, accelerate, bitsandbytes and several attention packages unpinned. Unpinned heavy dependencies plus pinned light ones is an unusual combination and worth resolving in a throwaway environment before you touch a working one.
On licensing, do not treat this as settled. The repository metadata says Apache-2.0, the README badge and pyproject.toml say MIT, and the classifiers list "License :: OSI Approved :: MIT License". Those are different licences with different patent and notice requirements. Read the LICENSE file at the repository root and confirm which one it actually contains before you redistribute anything. This is a fact-checking step, not legal advice.
Editorial conclusion
Adopt zeta if you are prototyping a transformer variant and want attention, feedforward, quantization and encoder-decoder blocks importable from one package, and you are willing to read the source when the documentation stops. Do not adopt it as the foundation of a production training stack: the README promises production readiness, but the release history is thin, the packaging metadata is inconsistent, and no migration or deprecation policy is documented. Before you commit, verify three things: which import path your installed version actually exposes, whether the pinned dependency ranges in pyproject.toml resolve against your existing torch and transformers versions, and what the LICENSE file in the repository says, since the README badge and the package metadata disagree.
Frequently asked questions
What is zeta used for?
Zeta is a modular PyTorch library of building blocks for AI models, including attention mechanisms, mixture-of-experts routing, feedforward networks, normalization, quantization and complete transformer, encoder, decoder and vision transformer implementations. The README positions it as components you assemble into a model rather than a training framework.
How do I install zeta?
The README gives a single command: pip3 install -U zetascale. The distribution name on PyPI is zetascale while the import name is zeta, so you install one name and import the other.
Is zeta the same as zetascale?
Yes. The repository is kyegomez/zeta, the import package is zeta, and the PyPI distribution declared in pyproject.toml is named zetascale.
What Python version does zeta require?
The pyproject.toml declares python = "^3.10" and lists Python 3.10 in its classifiers, so 3.10 or later is the stated requirement.
What happens if I import zeta and get an error?
The requirements.txt file pins joblib to >=1.3.0,<1.4.0 and scikit-learn to >=1.5.0,<1.6.0 with the comment "Pin compatible versions to prevent import errors", which suggests dependency conflicts are a known cause of import failures. Check those two packages against your environment first.
What licence is zeta released under?
The GitHub repository metadata reports Apache-2.0, while the README badge and the pyproject.toml both say MIT. The LICENSE file at the repository root is the source to check.
Community notes