DiffAbXL: A Benchmark for Ranking Antibody Designs by Log-Likelihood

What DiffAbXL Actually Does

DiffAbXL is a re-implementation of the DiffAb generative model, but it is not presented as a generator. The paper title, 'Exploring Log-Likelihood Scores for Ranking Antibody Sequence Designs', makes the purpose clear: use the model's log-likelihood as a score to rank candidate antibody sequences by predicted binding affinity. The intended user is a researcher or engineer who has a set of designed antibody sequences, possibly with structural information, and needs to pick which ones to test experimentally. The repository provides the model, training code, and an evaluation pipeline that computes Spearman correlation between log-likelihood scores and measured binding affinities across several datasets. That is a narrow but useful job, and the README frames it as a benchmarking tool rather than a general antibody design package.

The Mechanism: Log-Likelihood as a Ranking Score

The core mechanism is straightforward: given a list of antibody sequences, the model assigns a log-likelihood to each, and those scores are used to rank the sequences. The README notes that scores are computed using a 'naive approach, as outlined in Equation-11 in the paper'. That equation is not reproduced in the README, so the exact formula is not available from this material. The important detail is that the same naive scoring method is applied to all models in the benchmark for consistency. That is a deliberate trade-off: it makes comparisons fair across different model types, but it may not be the most accurate way to compute likelihoods for autoregressive or masked language models. The authors acknowledge this and say they plan to investigate more principled methods. For a user, this means the scores are comparable within the benchmark, but their absolute quality as likelihood estimates is unproven.

The Leaderboard: What It Shows and What It Does Not

The README includes a leaderboard with 16 models ranked by average Spearman correlation across five datasets. DiffAbXL-A-DN tops the list with an average rho of 0.41, followed by DiffAbXL-A-SG at 0.274. Other models like IgBlend, Antifold, and ESM sit lower. The leaderboard is informative, but it has clear limits. Scores of zero are assigned to models that did not show statistically significant correlation or were not suitable for score computation, such as models requiring an antigen. That zero-assignment policy can flatten differences: a model with a small but real negative correlation gets a zero, which may misrepresent its behaviour. Also, the leaderboard is based on five specific target datasets: Absci HER2, Nature HEL, Nature HER2, and AZ Target-2. If your target is outside those, the ranking may not transfer. The README does not provide confidence intervals or error bars, so the differences between ranks 2 and 5, for instance, may not be meaningful. Still, as a single table, it gives a quick sense of which approaches have been tested.

Getting It Running: Training and Interface

The README lists a 'Training' section and a 'Structure of the repo' section, but the cleaned text truncates before showing commands or file layout. What is available is the interface specification for benchmarking external models. The README asks that you implement a Python method in a class that accepts a list of antibody sequences, optional structure information (a PDB file path), and additional model-specific parameters. The method must return a dictionary with log-likelihood scores and other metrics like RMSD or pAE. A code template is provided, starting with 'def benchmark(self, sequences, structure=None, mask=None, **kwargs)'. That is the only concrete code shown. For training, you would need to clone the repository and inspect the source, since no commands are given in the README. The project is in Python, and the license is Apache-2.0, so you can reuse the code with attribution. The absence of a quick-start command in the README is a real friction point for new users.

A Genuine Limitation: The Naive Log-Likelihood

The most significant limitation is the naive log-likelihood computation. The README is candid: 'Log-likelihood scores in this work are computed using a naive approach, as outlined in Equation-11 in the paper, to maintain consistency across models.' That means the scores may not reflect true likelihoods for all model types. For example, an autoregressive model generates sequences token by token, so its likelihood is a product of conditional probabilities. A masked language model scores differently. Applying one naive formula to both can produce biased rankings. The authors say more principled methods exist but are not implemented. For a user, this means DiffAbXL's scores are a heuristic, not a rigorous statistical quantity. If your application depends on accurate likelihood values, for instance in a Bayesian optimization loop, you should treat these scores with caution. The leaderboard's zero-assignment policy compounds this by hiding negative correlations, which can be informative for model selection.

Alternatives: DiffAb, IgBlend, and Structure-Aware Models

The most direct alternative is the original DiffAb, which DiffAbXL re-implements. The README links to the DiffAb paper and code. DiffAb is a generative model for antibody design, so it produces sequences, whereas DiffAbXL focuses on scoring them. If you need a generator, use DiffAb. If you need a scorer, DiffAbXL is built for that. Another alternative is IgBlend, which appears in the leaderboard in two modes: structure-only and sequence-only. IgBlend's structure-only mode achieves an average rho of 0.232, lower than DiffAbXL-A-DN but higher than DiffAbXL-H3-SG. IgBlend's approach likely differs in how it combines sequence and structure information. Antifold is another structure-aware model on the list. The key difference is that DiffAbXL is a diffusion-based model, while IgBlend and Antifold may use different architectures. The README does not explain these differences in detail, so you would need to consult each model's documentation. The practical takeaway is that DiffAbXL is not the only scorer, and its naive likelihood may be a reason to look at alternatives.

Maintenance and Upgrade Cost

The repository has no recent releases listed, and the last push is unknown. The README does not mention a versioning scheme or a changelog. That makes maintenance cost hard to assess. The paper is from 2024, so the code is relatively new. The Apache-2.0 license is permissive, so you can fork and modify without legal friction, but you must retain attribution. There is no mention of dependencies or installation instructions in the README, which is a red flag for reproducibility. You will likely need to read the source to figure out the environment. The 'Experiment tracking' section is mentioned but not detailed, so you may need to set up your own logging. In terms of upgrade cost, the code is a research artifact, not a maintained library. If you adopt it, plan to maintain it yourself or pin the version you use. The lack of releases means you cannot easily track changes.

Who Should Adopt It and What to Verify First

This tool is for researchers who want a reproducible baseline for ranking antibody designs, particularly those working with structure-aware generative models. The leaderboard provides a useful reference point, and the interface specification makes it possible to benchmark your own model against DiffAbXL. However, if you need a production-grade scoring function, the naive likelihood is a dealbreaker. Before adopting, verify that your target dataset is similar to the ones used in the paper, and check whether you have structure information, since the structure-guidance modes may require it. Also, confirm that your model can produce log-likelihood scores in the expected format. The README does not give a command to run the benchmark, so expect to dig into the code. If you are comfortable with that, DiffAbXL is a solid starting point for comparing antibody design models, but it is not a finished product.

Editorial conclusion

Adopt DiffAbXL if you need a reproducible baseline for ranking antibody designs by log-likelihood, especially if you work with structure-aware models and want to compare against a published leaderboard. Do not adopt it if you expect a production-ready scoring tool with principled likelihood estimates; the README explicitly states the scores use a naive approach that may not suit autoregressive or masked language models. Before using it, verify that your target dataset matches the five used in the paper, and check whether the zero-shot or structure-guidance mode fits your data availability. The repository is a research benchmark, not a polished library, so expect to adapt the interface to your own models.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. README
4. Releases