keras-attention: A Single Attention Layer for Keras, With Luong and Bahdanau Scores

What keras-attention replaces in a Keras model

Recurrent layers compress a sequence into a final hidden state or a stack of per-timestep states. If you want the model to weight timesteps rather than treat them uniformly, you normally write the scoring and softmax by hand. This package exposes that as one layer, Attention(units=128, score='luong'), so the weighting becomes a line in the model definition. The README describes the layer as supporting the score functions of Luong and Bahdanau, and the 5.0.0 release note says the same thing, which tells you the additive variant was not there in earlier versions. The audience is narrow and specific: someone already inside Keras with TensorFlow 2.x, using an LSTM or comparable recurrent layer that returns sequences, who wants a single attention vector rather than a full attention block. The examples in the repository point at the same scale of problem: adding two numbers separated by delimiters, and finding the maximum of a 1D sequence.

The mechanism: a 3D tensor in, a weighted vector out

The layer takes a 3D tensor of shape (batch_size, timesteps, input_dim). Internally it computes a score between the current hidden state and each encoder state, using either the multiplicative form attributed to Luong or the additive form attributed to Bahdanau, then normalizes those scores into weights and returns the weighted sum. The output is a 2D tensor of shape (batch_size, num_units), where num_units is the units argument. That output shape is the part worth pausing on. You get one vector per sample, not a sequence, so the layer collapses the time axis. If you need per-timestep outputs to feed another recurrent layer, this layer as documented will not give you that, and you would be stacking it differently or not at all. The README points to examples/add_two_numbers.py for visualizing the attention weights, which implies the weights are reachable for inspection rather than buried, though the README does not spell out the accessor.

Installing it and the argument that matters most

Installation is a single command: pip install attention. The layer is then imported as from attention import Attention. Two arguments are documented. units is an integer for the number of output units in the attention vector. score is a string, either 'luong' or 'bahdanau'. The README's example builds an LSTM with return_sequences=True, passes its output to Attention(units=32), then to Dense(1). The return_sequences=True detail is not decoration: without it the LSTM emits a 2D tensor and the layer's documented 3D input shape is not satisfied. The example also trains with loss='mae' and optimizer='adam' and runs ten epochs on random data, which the comment explicitly says has nothing to learn. Treat that snippet as a wiring check, not a tutorial.

The custom_objects requirement when you save a model

The example saves to HDF5 and reloads with load_model('test_model.h5', custom_objects={'Attention': Attention}). That is the practical friction of a custom layer. Keras cannot reconstruct Attention from the file alone unless the class is resolvable, so any serving path, batch job or notebook that loads a saved model has to import this package and register the class. The example asserts the reloaded model produces predictions almost equal to the original with np.testing.assert_almost_equal, which is the right check to keep in your own test suite if you ship a model file. If your deployment pipeline cannot carry a Python dependency alongside the model artifact, this is a real obstacle, not a footnote.

Where the documentation is thin and where the layer is the wrong tool

The README does not document masking behavior. For variable-length sequences, which is the common case in text, you would normally want padded timesteps excluded from the softmax. Nothing in the supplied material states whether the layer respects a mask, so that is something to verify in the source before committing to it on padded data. The layer is also single-head and self-contained: it attends over one sequence, and the material gives no cross-attention mode where a decoder queries a separate encoder. If you need multi-head attention or a transformer block, this is the wrong tool and you should look at Keras's own attention utilities instead. The comparison worth making is with tf.keras.layers.Attention and AdditiveAttention, which are part of TensorFlow itself. The difference in approach is packaging and scope: TensorFlow's layers ship with the framework, so there is no extra dependency and no custom_objects registration, but they are built around query, value and key tensors for cross-attention rather than the single-sequence reduction this package performs. This package trades that generality for a smaller call signature.

The accuracy numbers in the README and what they do not cover

The IMDB experiment reports two LSTM networks of 250K parameters each, one with the attention layer and one with a fully connected layer, over ten runs and ten epochs. The table gives max accuracy of 88.22 without attention and 88.76 with it, average accuracy of 87.02 against 87.62, and standard deviation of 0.18 against 0.14. The README's own framing is that the boost is expected and that the reduced run-to-run variability is a side benefit. Read those gaps honestly: under a point of average accuracy, on one dataset, with one architecture pairing. It is evidence that the layer does not break training, not evidence that attention is a general accuracy upgrade. The two other examples, delimiter addition and sequence maximum, are qualitative: the README says the attention map converges to the ground truth and that the layer converges perfectly on the maximum task, shown as images rather than tables.

Version history, licence and what to check before adopting

The project is Apache-2.0 licensed, which permits commercial use and modification with the usual notice and patent terms; that is a statement about the licence text, not legal advice, and you should read the LICENSE file for the actual conditions. Two releases are listed. Version 3.0, from September 2020, is described as attention in Sequential. Version 5.0.0, from March 2023, is the one that supports both Luong and Bahdanau scores. The README states testing against TensorFlow 2.8 through 2.14 as of September 2023. The repository's last push is dated March 2026, so there is activity after that testing note, but the README does not extend the compatibility list. Verify three things before you build on it: that the installed version exposes score='bahdanau' if you need the additive form, that the layer behaves correctly under masking if your sequences are padded, and that your model loading path passes Attention through custom_objects. The repository has no homepage listed, so the README and the examples directory are the documentation.

Editorial conclusion

Adopt it if you are building a Keras model with an LSTM or similar recurrent stack and want attention weights as a plain tensor you can feed into a Dense layer, without pulling in a full transformer library. Do not adopt it if you need multi-head attention, cross-attention between two separate sequences, or a maintained compatibility guarantee beyond the TensorFlow versions the README lists. Before wiring it in, verify the version you install exposes the score argument, since Bahdanau only arrived in 5.0.0, and confirm your save/load path passes Attention through custom_objects.

Official sources

1. Issues
2. License: Apache-2.0
3. philipperemy/keras-attention on GitHub
4. README
5. Releases