logpai/loghub: nineteen system log datasets, deliberately uncleaned

A repository whose contents are not source code

The first thing to notice is that this repository has no detected language, and that is not an oversight in the metadata. There is no build system, no package manifest, and no source tree. What is on the master branch is a set of directories named after the systems the logs came from, plus three files at the root: `README.md`, `LICENSE`, and `CITATION`.

The description on the project page reads "A large collection of system log datasets for AI-driven log analytics [ISSRE'23]", and the topics listed follow that framing: anomaly-detection, datasets, log-analysis, log-intelligence, log-parsing, logs, and unstructured-logs. The README opens with the same statement in plainer words, saying the collection is freely accessible for AI-driven log analytics research, and that some logs are production data released from previous studies while others were collected from real systems in a lab environment.

So the git repository is the catalogue and the description, not the payload. The actual logs are downloaded from Zenodo, and each row in the README table links out to a zip or tarball on zenodo.org under record 8196385. That split matters if you are deciding how to work with this: cloning the repo gives you sixteen system directories and some documentation, and the multi-gigabyte archives arrive separately. At roughly 2847 stars and 794 forks with only 2 open issues, and a last push on 2026-09-20, this is a stable, low-churn reference rather than something expecting contributions.

Six families, nineteen datasets, and a wide spread of scale

The README groups the datasets into labelled bands, and the grouping is more informative than a flat list would be. Distributed systems covers HDFS_v1, HDFS_v2, HDFS_v3, Hadoop, Spark, Zookeeper, and OpenStack. Super computers covers BGL, HPC, and Thunderbird. Operating systems covers Windows, Linux, and Mac. Mobile systems covers Android_v1, Android_v2, and HealthApp. Server applications covers Apache and OpenSSH. Standalone software covers Proxifier alone.

Scale varies by four orders of magnitude, which is the practical constraint when you pick something to start on. Thunderbird is the largest at 211,212,192 lines and 29.60GiB, and Windows is close behind at 114,608,388 lines and 26.09GiB. HDFS_v2 carries 71,118,073 lines in 16.06GiB, Spark 33,236,604 lines in 2.75GiB, and Android_v2 30,348,042 lines in 3.38GiB. At the other end, Proxifier is 21,329 lines in 2.42MiB and Linux is 25,567 lines in 2.25MiB, with Apache at 56,481 lines in 4.90MiB.

The time spans, where the table records them, are worth reading alongside the line counts, because a large log can still be a short window. Linux spans 263.9 days across only 25,567 lines, and Apache spans the same 263.9 days at 56,481 lines. Mac covers just 7.0 days and HealthApp 10.5 days. HDFS_v1 is measured in hours rather than days at 38.7 hours, while Thunderbird stretches across 244 days and BGL across 214.7. Several rows carry N.A. for time span, including HDFS_v2, Hadoop, Spark, OpenStack, HPC, and both Android datasets.

One formatting detail suggests the table is maintained by hand rather than generated. The Spark row is missing its Labeled cell, so its columns do not line up with the rest of the table. It is cosmetic, but it tells you the numbers are curated by hand and worth spot-checking against the per-system directory before you cite a figure.

Labeled ground truth exists, and one row points at an open issue

The Labeled column is the column that decides whether a dataset is usable for log parsing research, because supervised log parsing needs ground truth to train or evaluate against. Six of the nineteen datasets carry a check mark: HDFS_v1, HDFS_v3, Hadoop, OpenStack, BGL, and Thunderbird. Everything else, including all three operating system sets, both Android sets, Spark, Zookeeper, HPC, HealthApp, Apache, OpenSSH, and Proxifier, is unlabeled.

The Hadoop entry carries a footnote pointing at issue #56 rather than a plain confirmation, with the note "Check #56". That is an unusual thing to leave in a published table and it tells you the maintainers considered the labeling status of that dataset unsettled enough to redirect readers to a discussion. If Hadoop matters to your work, that issue is the place to look, and the same honesty applies to the N.A. time spans.

There is also a version distinction worth catching. HDFS appears three times: HDFS_v1 at 11,175,629 lines with labels, HDFS_v2 at 71,118,073 lines without, and HDFS_v3 as an instrumented trace log called TraceBench at 14,778,079 lines with labels. The presence of an instrumented variant tells you the collection includes not just passively collected logs but logs captured with added telemetry, which changes what you can ask of them. Android follows the same pattern with v1 at 1,555,005 lines and v2 at 30,348,042.

The two papers in the citation section matter here too. The ISSRE 2023 paper is the original Loghub release, while the ISSTA 2024 paper is a large-scale evaluation of log parsing techniques, which is where the newer versions tend to come from. If you are reproducing numbers from the literature, the version of the dataset those papers used is not always the one in the current table.

The logs are raw on purpose, and that is the main caveat

The sentence that should shape your expectations is in the README's second paragraph: wherever possible, the logs are NOT sanitized, anonymized or modified in any way. Those three words are the entire licensing posture in miniature, and they cut both ways.

For research, this is the point. Log parsing work depends on the artifacts that appear in real production output, and a log file with identifiers, hostnames, paths, and user-facing strings scrubbed out is measurably easier to parse than a real one. Preserving that mess is what makes the collection a fair benchmark. The README is also clear that some of these logs are genuine production data released from previous studies rather than anything generated or lab-simulated, which raises the stakes on what you do with them.

The practical consequence is that the datasets are described as freely available for research or academic work, and the license section says any usage or distribution should refer back to the repository URL and cite the loghub paper where applicable. There is no grant of commercial use in what the README states. Combined with the unsanitized note, that means anything derived from these logs, including model outputs, sits in territory where your own legal review matters more than the terms do.

The API reports the license field as NOASSERTION, and there is a `LICENSE` file at the repository root that the README does not quote. Those two facts sit in mild tension: a LICENSE file implies specific terms, while NOASSERTION tells you no machine-readable grant was detected. Read the file before relying on either, and treat the README sentence as the summary rather than the grant.

Download volume, Zenodo hosting, and the download-list wiki

The README claims the datasets have been downloaded by more than 450 organizations from both industry and academia, tracked through a Zenodo DOI and a download counter badge. There is a separate wiki page listing the organizations that have downloaded the data, linked from that sentence. The number is presented as a reach statistic for the collection rather than as a measure of dataset quality, and it is the sort of figure worth treating as an indicator of adoption and nothing more.

Hosting everything on Zenodo is a deliberate choice for a dataset of this size. Git LFS would work but carries repository bloat, and the archives here run to tens of gigabytes, which is well past the point where version control is the right tool. Zenodo gives each archive a DOI, which is what makes citation of a specific dataset version possible at all. The catch is that the links in the README table all point at a single Zenodo record, 8196385, so the individual files share one archival record rather than having per-dataset DOIs.

The repository itself is mostly a pointer layer. Sixteen system directories, one of them per system family, hold the parsing-related documentation for each dataset, and the README links into them with relative paths such as `./HDFS#hdfs_v1` and `./Android#android_v2`. Those per-system pages are where the template structures and labeled samples live. The top-level table is deliberately a summary: name, description, label status, span, lines, size, download link. Anyone doing real work on a dataset needs to open the corresponding directory.

The last section of the README routes discussion to GitHub Discussions rather than issues, with a direct link to open one. With only 2 open issues against 2847 stars, that split reads as an acknowledgement that questions about data are conversational and not bug reports. It is also the practical channel for asking whether a specific dataset behaves the way you expect before you build on it.

Editorial conclusion

Loghub is a reference collection rather than a tool, and its value is that it settles the question of what real system logs look like before anyone has cleaned them. The trade is stated plainly on the front page: the logs are NOT sanitized, anonymized or modified in any way, which is exactly what makes them useful for research on messy input and exactly what you need a legal review for before anything else. Start with the small labeled sets if the goal is log parsing, since Linux at 25,567 lines and Apache at 56,481 are far easier to iterate against than Thunderbird at 29.60GiB. Read the LICENSE file directly rather than trusting the license field, which the API reports as NOASSERTION, and cite both the ISSRE 2023 and ISSTA 2024 papers as the README asks. The per-system directories carry the parsing details that the top-level table deliberately leaves out.

Frequently asked questions

Official sources

1. Issues
2. logpai/loghub on GitHub
3. README