kyegomez/Gemini: A PyTorch Skeleton for Natively Multimodal Transformers

What kyegomez/Gemini actually is, and who it is written for

The README opens by calling this "the open source implementation of Gemini", and that phrasing sets the expectation too high. There is no checkpoint, no training log, and no evaluation table in the material. What exists is a PyTorch module tree under the gemini_torch package, distributed on PyPI as gemini-torch under the MIT licence, that lets you construct a transformer which accepts text, image and audio tensors in one forward pass. The README states the intent directly: inputs are "transformed into tokens, which are then processed by a transformer", with conditional decoding afterwards. The author also describes a staged plan, starting with image embeddings, then audio, then video. Treat this as a reference architecture you can read and modify. The audience is narrow: engineers who want to study or extend a multimodal transformer in PyTorch, and who already have the compute and the dataset to train one. If you arrived hoping to run Gemini locally the way you run a quantised language model, this repository will not do that, and the README never claims it will.

The design choice that separates it from Flamingo-style stacks

Most open multimodal transformers bolt a separate vision encoder onto a language model and project its outputs into the text embedding space. The README says this implementation does the opposite. It notes that the architecture "bears resemblance to Fuyu's architecture" and that "instead of utilizing a visual transformer (vit) encoder, Gemini simply feeds image embeddings directly into the transformer". Modality is signalled by special tokens, with the README suggesting markers such as [IMG], <img>, [AUDIO] or <audio>. That decision removes a whole component and the alignment training that goes with it, which makes the codebase smaller and the data path shorter. It also means the transformer has to learn visual structure from raw patch embeddings without a pretrained encoder's prior. Whether that trade is worth it is an empirical question the repository does not answer, and the README does not present a comparison against an encoder-based baseline.

Instantiating the model: the arguments the README prints

Installation is a single line: pip3 install gemini-torch. The README then gives a text-only example that imports Gemini from gemini_torch.model and constructs it with num_tokens=50432, max_seq_len=4096, dim=1280, depth=16, dim_head=64, heads=12, use_abs_pos_emb=False, attn_flash=True, attn_kv_heads=2, qk_norm=True, attn_qk_norm=True and attn_qk_norm_dim_scale=True. Input is a LongTensor of shape [batch, seq_len] and the README shows the output as [batch, seq_len, dim]. The multimodal example uses the same class with a smaller configuration (num_tokens=10000, max_seq_len=1024, dim=320, depth=8) plus two extra flags, post_fusion_norm=True and post_modal_transform_norm=True, and calls the model with keyword arguments text, img and audio. Note the different return signature: the multimodal call unpacks two values as y, _ = model(...), while the text-only call returns y directly. The README comments in that block are worth reading as a warning about scale, since every dimension is annotated as "Reduced from" a larger value. The listed capabilities are multi-grouped query attention, flash attention, RoPE, ALiBi, xPos, QK norm, no positional embeddings and a KV cache.

LongGemini, ring attention, and the text-only constraint

The repository ships a second model class, LongGemini, imported from the gemini_torch package root rather than from gemini_torch.model. The README describes it as "An implementation of Gemini with Ring Attention, no multi-modality processing yet", which is a plain statement that the long-context path handles text only. Its constructor takes dim, depth, dim_head, long_gemini_depth, heads, qk_norm and ring_seq_size, and the example passes ring_seq_size=512 against an input of 1024 tokens. The presence of a separate long_gemini_depth parameter suggests an interleaving scheme where some blocks use ring attention and others do not, but the README does not explain the split. Anyone planning long-context multimodal work should read this as two separate code paths that have not been merged.

The tokenizer, and how much of the multimodal pipeline is unfinished

Tokenization is handled by MultimodalSentencePieceTokenizer, constructed with tokenizer_name="hf-internal-testing/llama-tokenizer". The README says the project uses "the same tokenizer as LLAMA with special tokens denoting the beginning and end of the multi modality tokens", and the example calls tokenizer.encode("Audio description", modality="audio") then tokenizer.decode on the result. That modality argument is the whole multimodal contribution at this layer: it selects which special tokens wrap the text. The README is candid that the tokenizer "does not fully process img, audio, or videos now" and asks for help. The todo list is equally candid. The image feature embedder is marked done, with a quote from the Gemini report about visual encoding inspired by Flamingo, CoCa and PaLI. Audio is marked done via USM features sampled at 16kHz, again quoted from the report. Video is unchecked, with a truncated quote about encoding video as a sequence of frames in a large context window. So the repository claims two of three modalities at the embedding stage, and the tokenizer section contradicts the optimistic framing by admitting the tokenizer does not process them. That tension is the most useful signal in the README.

Where this is the wrong tool

The clearest limitation is that nothing here is trained. The README's own code comment says "After much training" before showing model.eval() and a detokenize step, and no training script, dataset loader or loss function appears in the material. If your goal is inference on real prompts, you are looking at the wrong repository. A second limitation is the absence of any reported result: no loss curve, no benchmark, no comparison to a baseline. That makes the architecture unvalidated, and it means you cannot tell from the repository whether the no-vision-encoder design holds up. Third, the multimodal forward pass expects an audio tensor shaped [batch, audio_seq_len, dim] in the example, which is not a raw waveform and not the 16kHz USM feature extraction the README quotes from the report; the conversion step from audio to that tensor is not shown. Fourth, the README's own reference list reads like a research wishlist rather than a roadmap, mixing PPO or MPO, RLHF, speculative decoding and the Algorithm of Thoughts. Those are aspirations, not features.

What to compare it against

The nearest architectural relative named in the README is Fuyu, which the author cites as the closest analogue because it also skips a separate vision encoder. The practical difference is packaging: Fuyu is published as a trained model with weights and an inference API, whereas this repository is the architecture alone. If you want a working multimodal model today, a hosted API or a released open-weights model gets you there faster. If you want to study early-fusion design in readable PyTorch, this is more legible than a production training stack, because the whole model fits in a handful of files and the constructor arguments map directly onto the paper's components. The honest comparison is not "this versus Gemini the product" but "this versus writing the transformer yourself from the Gemini report": the repository saves you the scaffolding and the attention variants, and leaves you the training problem.

Licence, maintenance and upgrade cost

The licence is MIT, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is permissive, and it means you can vendor the code into a proprietary training pipeline. It also means the project offers no warranty and no support obligation, so the maintenance burden lands entirely on you. The README gives no version history and no changelog, and the material lists no releases, so pinning gemini-torch to a specific version is the only way to make upgrades deliberate. The API surface is wide: the Gemini constructor alone exposes more than a dozen keyword arguments in the examples, and the multimodal call returns a tuple while the text call does not. Any upstream change to those signatures will break call sites. Budget for reading gemini_torch/model.py before each upgrade rather than trusting the README, since the README already contains configurations marked as reduced from larger values and may lag the code.

Editorial conclusion

Adopt kyegomez/Gemini if you want a readable PyTorch skeleton for a natively multimodal transformer and you are prepared to write your own training loop, data pipeline and evaluation harness. Do not adopt it if you need a downloadable checkpoint, a reproducible benchmark, or a drop-in replacement for a hosted multimodal API, because the README describes no released weights and no reported results. Before committing, verify three things against the repository itself: that pip3 install gemini-torch resolves on your Python and PyTorch versions, that the MultimodalSentencePieceTokenizer still loads hf-internal-testing/llama-tokenizer, and that the Gemini constructor signature in gemini_torch/model.py matches the keyword arguments printed in the README.

Official sources

1. Issues
2. kyegomez/Gemini on GitHub
3. License: MIT
4. Project website
5. README