Transformers.jl: BERT and GPT-style models inside a Flux.jl pipeline
Julia Implementation of Transformer models
At a glance
- What is it?
- Transformers.jl is a Julia package that loads pretrained transformer checkpoints from the Hugging Face Hub and runs them as ordinary Flux layers. It is useful if your model already lives in Julia; it is a poor fit if you need the breadth of architectures the Python ecosystem ships.
- Who is it for?
- Adopt Transformers.jl if your training or inference code is already Julia and Flux, and the checkpoint you need is reachable through hgf. Do not adopt it if you depend on architectures or tokenizer behaviour that only the Python reference implementation provides, or if you cannot absorb the 0.1.x to 0.3.x rewrite.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 47 days ago.
- What is it written in?
- Mainly Julia, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap Transformers.jl fills between Flux and pretrained checkpoints
Flux.jl gives you layers, gradients and an optimizer. It does not give you a BERT checkpoint. A Julia user who wants to fine-tune an existing pretrained encoder has, without this package, two options: reimplement the architecture and the weight-loading code by hand, or shell out to Python. Transformers.jl exists to remove that step. It is a Julia implementation of transformer-based models built on Flux, and its practical purpose is to make a pretrained model a value you can call like any other layer. The audience is narrow and identifiable: people writing Julia for scientific computing, numerical work or a Julia-native service, who want transformer inference or fine-tuning inside that same process rather than across a language boundary. If you are starting a new deep learning project with no Julia commitment, this package is not the shortest path, and the README does not pretend otherwise. It assumes you already chose Julia.
How the hgf macro turns a model name into a callable Flux layer
The mechanism visible in the README is a two-object split: a text encoder and a model. The line textencoder, bert_model = hgf"bert-base-uncased" does the loading. The hgf string macro is the Hugging Face entry point, exported from Transformers.HuggingFace, and it resolves a model identifier into a tokenizer/encoder object plus a model object. The encoder is not a passive wrapper. In the example, encode(textencoder, text) performs tokenization, adds the special tokens, applies truncation or padding, and one-hot encodes the result. The README asserts the round trip explicitly: decode(textencoder, sample.token) is checked against a list containing [CLS], wordpieces such as "peck" and "peppers", the subword splits "##led" and "##zzy", and [SEP] separators. That assertion is the useful part of the example, because it documents the pipeline in one place. The input is a batch, written as a nested vector [[text1, text2]], described in the README as one batch of contiguous sentences. The model is then called directly on the encoded sample, and the output is a named field: bert_model(sample).hidden_state. So the data flow is text, then encoder, then a struct with named outputs, then whatever head you attach. The named-field output is a design choice worth noting: it keeps the model's internal activations addressable instead of collapsing everything into a single tensor, which matters when you want the hidden state rather than a classification logit.
Installation and the minimum working example
Installation is one line in the Julia REPL, using the package manager mode: ]add Transformers. That is the entire documented setup. There is no separate build step, no model download command, and no configuration file described in the README. The download appears to be implicit in the hgf macro: the model name is the configuration. The README gives the working example in full, and it is short enough to reproduce: bring in Transformers, Transformers.TextEncoders and Transformers.HuggingFace, call hgf on a checkpoint name, build a batch, encode it, and index into the result. The example uses bert-base-uncased and reads .hidden_state off the model call. Two details in that example are the ones most likely to trip you up in your own code. First, the batch shape: the input is a vector of vectors, not a flat vector of strings, so a single sentence still needs the extra nesting. Second, the field access: the model returns a struct, and you must know which field you want. The README points to the example folder for the complete version and to the documentation site for anything beyond this. Treat the README snippet as a smoke test you run before writing your own code, not as an API reference.
The 0.1.x to 0.3.x break is the main adoption risk
The README carries a notice in plain language: the current version is almost completely different from the 0.1.x version, and users on the old version should either update their code or stay on the old release. That is the single most important sentence in the repository for anyone evaluating it. It tells you the API has already moved once in a way that invalidates existing code, and it tells you the maintainer is willing to make that kind of change. The released versions support this reading: v0.2.8 in October 2023, then v0.3.0 in June 2024, then v0.3.1 later the same month. A jump from 0.2 to 0.3 within a package that has already rewritten itself once means you should pin an exact version in your Project.toml rather than tracking the latest. The other limitation is scope. The package describes itself as an implementation of transformer-based models with Flux, and the README's only worked example is BERT. Nothing in the supplied material lists which architectures are supported, how complete the coverage is, or how the package handles models whose tokenizers differ from the BERT wordpiece scheme shown. If your work depends on a specific checkpoint family, that is a question you answer by trying it, not by reading the README.
Transformers.jl versus calling the Python reference from Julia
The obvious alternative is not another Julia package. It is PyCall or PythonCall, driving the Python transformers library from inside Julia, or running inference as a separate service and talking to it over a socket. The difference in approach is real and it is not about speed. The Python library is the reference implementation for most published checkpoints: when a model card says a tokenizer behaves a certain way, that behaviour is defined by that code. Calling it means you inherit that fidelity, including every quirk of a tokenizer you did not write. Transformers.jl reimplements the architecture and the preprocessing in Julia, which buys you a single-language stack, no Python environment to manage, and the ability to treat a pretrained model as a Flux layer that participates in Julia's own autodiff and type system. What you give up is the guarantee that your tokenization and your forward pass match the reference exactly. The README's decode assertion is evidence that the maintainer cares about this, and it is a good sign, but one asserted example for one checkpoint is not a compatibility guarantee across the model zoo. If exact numerical agreement with a Python baseline is a requirement in your project, the Julia reimplementation is the riskier choice, and you should decide that before you write training code around it.
Maintenance, versioning and the MIT licence
The package is MIT licensed, which is permissive: it allows commercial and closed-source use, modification and redistribution, provided the copyright notice and licence text are retained. That is the whole of the licence implication here, and it is a favourable one for adoption inside a company. It is not legal advice, and if you redistribute a model you loaded through hgf, the licence of that checkpoint is a separate question from the licence of this package. On maintenance, the material shows a repository that is not archived and a last push date in 2026, with the most recent tagged release being v0.3.1 in June 2024. Those two facts together suggest ongoing activity without a steady release cadence. The upgrade cost you should budget for is the one the README warns about directly: a version bump may require rewriting the code that constructs and calls your models, because that has already happened once at the 0.1.x boundary. Pin the version, read the release notes before moving, and keep the encoding round-trip assertion from the README in your own test suite so a silent change in tokenization shows up as a failing test rather than as degraded model output.
Editorial conclusion
Adopt Transformers.jl if your training or inference code is already Julia and Flux, and the checkpoint you need is reachable through hgf. Do not adopt it if you depend on architectures or tokenizer behaviour that only the Python reference implementation provides, or if you cannot absorb the 0.1.x to 0.3.x rewrite. Before committing, verify three things yourself: that hgf"<model-name>" resolves and downloads for your specific checkpoint, that encode(textencoder, text) produces the token sequence you expect for your input, and that the package version you pin is the one your code was written against.
Community notes