xet-core: the chunk-deduplicating storage client behind huggingface_hub

Chunked deduplication, and why it is not the same as Git LFS

xet-core enables huggingface_hub to use Xet storage for uploading and downloading to the Hub. The README lists what Xet storage provides: chunk-based deduplication, efficient storage and retrieval with local disk caching, and backwards compatibility with Git LFS.

The deduplication is the point, and it operates below the file level. The feature list describes it as avoiding the transfer and storage of chunks that are shared across binary files, with models and datasets given as the examples. Two different model repositories that share a tokenizer, a weight tensor or a data shard will share the chunks, and the second upload only sends what is genuinely new.

The difference from Git LFS is the unit of deduplication. Git LFS stores whole objects and only avoids duplicating a file that is byte-identical to one it already has, so two repositories that overlap partially still pay for the overlap twice. Chunking moves the unit below the file, which is what makes partial overlap free.

The compatibility statement is about the transition rather than the storage. Existing Git LFS behaviour keeps working, and git-xet is a Git LFS compatible command line tool, so a repository can move to Xet storage without changing how Git sees it.

The README also says the important thing about scope: this library is not meant to be used directly, and is instead intended to be used from huggingface_hub. That is a deliberate boundary. A storage client that is only ever driven by the hub library has a much smaller API surface to keep stable, and the CLI exists for the cases where you need one anyway.

Five crates, each with one job

The repository publishes five Rust crates on crates.io, and the split is a readable map of the system.

hf-xet is the high-level client library for uploading and downloading files with chunk-based deduplication, and it is the one an application would use. xet-client is the HTTP client for communicating with Hugging Face Xet storage servers, which the key feature list calls CAS, the Hub backend services, with the network layer described as concurrent communication to them.

xet-data is the data processing pipeline for chunking, deduplication and file reconstruction. That is the part that has to be correct in the hardest way, since a chunking scheme that cannot reconstruct the original file byte for byte is worthless, and the same code has to produce identical chunk boundaries on the client and the server or the deduplication index will not match.

xet-core-structures holds the core data structures, named as MerkleHash, metadata shards and Xorb objects. The Merkle hash is how a content-addressed chunk tree is summarised, the metadata shards carry the per-object metadata, and Xorb is the container for a group of chunks.

xet-runtime is the async runtime, configuration, logging and utility infrastructure that the other crates share.

So the dependency direction runs from hf-xet down through xet-data and xet-client to xet-core-structures and xet-runtime, and each layer can be tested on its own. The workspace Cargo.toml confirms the arrangement: the members are xet_runtime, xet_core_structures, xet_client, xet_data and xet_pkg, with git_xet listed as a top-level crate that is not published as a package.

The Python wheel and the git-xet binary

Two distributions come out of the same repository, and they serve different users.

The Python package is hf-xet on PyPI, described as the Python bindings for the Xet storage system used by huggingface_hub. It is built from the hf_xet/ directory using maturin, which is the PyO3 build tool, and the dependency list in the workspace Cargo.toml pins pyo3 at 0.29 with the abi3-py38 and multiple-pymethods features.

The abi3 feature matters more than it looks. abi3-py38 means the extension is built against the stable ABI rather than a specific CPython version, so one wheel covers Python 3.8 and later instead of needing a wheel per interpreter version. For a package that has to be installed on every machine that runs huggingface_hub, including machines the user does not administer, that single-wheel property removes an entire category of installation problem.

The second distribution is the git-xet binary, a Git LFS compatible command line tool for Xet storage, built from the git_xet/ directory and distributed through GitHub releases rather than through a package registry. The distinction is sensible: the binary is for people working with Git repositories and LFS, who look for tools in the same place they find other Git tooling, while the Python bindings are for applications.

Both are named hf-xet, the crate and the wheel, and the CLI keeps the git prefix, so the naming is consistent even though the distribution channels are not.

Telemetry that reports counts and nothing identifying

The telemetry section is specific enough to be checked, which is a good sign.

At the end of each upload or download, hf-xet reports a small performance summary to the CAS server. The summary contains byte counts, dedup effectiveness, throughput and wall time. The payload contains no file names, no paths, no hashes, no repository ids and no user ids.

The delivery guarantees are stated too: it is best-effort, never retried, and it cannot delay or fail a transfer. That is the property that matters. A telemetry call that can block a download is a liability, and this one is explicitly prevented from doing so.

Turning it off takes one environment variable:

HF_XET_TELEMETRY_ENABLED=0

But the precedence is the part to read carefully. Telemetry is also off whenever the huggingface_hub telemetry opt-outs are set, which are HF_HUB_DISABLE_TELEMETRY, DISABLE_TELEMETRY, DO_NOT_TRACK, HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, with any of the values 1, ON, YES or TRUE. And these win over HF_XET_TELEMETRY_ENABLED=1. So an environment that sets DO_NOT_TRACK is fully off regardless of the Xet variable, which is the behaviour most people would expect and is easy to get wrong if you only read the first paragraph.

There is also a tuning block for the unusual cases:

HF_XET_TELEMETRY_FINAL_FLUSH_TIMEOUT=2s  # how long a transfer waits for its final report; 0 = don't wait
HF_XET_TELEMETRY_HEARTBEAT_AFTER=5m      # long transfers report progress after this; 0 disables
HF_XET_TELEMETRY_HEARTBEAT_INTERVAL=5m   # and then at this interval
HF_XET_TELEMETRY_REQUEST_TIMEOUT=5s      # per-request budget
HF_XET_TELEMETRY_MAX_IN_FLIGHT=4         # concurrent reports before dropping

The heartbeat settings exist for long transfers, where a single end-of-transfer report would be a poor way to see progress.

Diagnostics scripts for a client that runs inside your process

Because hf-xet is a library linked into huggingface_hub, a failure in a transfer is a failure in the Python process, and diagnosing it means collecting information from inside that process.

The repository ships a scripts/diag/ directory with platform-specific scripts that download debug symbols, configure logging and capture periodic stack traces and core dumps. There is one for Linux, one for macOS, and one for Windows in Git Bash.

The usage pattern is to prefix the failing command with the script for your operating system:

./scripts/diag/hf-xet-diag-macos.sh -- python my-script.py

The double dash separates the script from the command it wraps, so anything you can run normally, you can run under the diagnostic wrapper.

The quick debugging environment variables are the next thing to reach for:

RUST_BACKTRACE=full          # full Rust backtraces on panic
RUST_LOG=info                # enable hf-xet logging
HF_XET_LOG_FILE=/tmp/xet.log # write logs to a file (defaults to stdout)

The third one is the one that changes behaviour in a useful way. By default the client logs to stdout, which in a hub client means it can end up interleaved with the output of whatever the user was doing, and a machine log is easier to attach to an issue than a terminal scrollback.

The README points at scripts/diag/README.md for the full usage, the output layout, dump analysis instructions and how to install the debug symbols manually, which is the file to read before a first diagnostic run.

The workspace, the release cadence, and the boundaries

The Cargo workspace gives the release picture. The version is 1.7.0 in the workspace package metadata, the edition is 2024, the licence is Apache-2.0, and the release profile is tuned with opt-level 3, link time optimisation enabled and debug information kept.

The excludes list is also informative. The simulation directory, the hf_xet Python package directory, the two wasm builds and the examples are all outside the workspace, so the published crate set is exactly the five libraries and the git_xet binary, with the bindings, the WebAssembly ports, the simulation harness and the Node example built separately.

That is a sensible separation. The wasm/hf_xet_wasm and wasm/hf_xet_thin_wasm directories tell you there are two WebAssembly flavours, a full one and a thin one, which is how a Rust storage client can run in a browser for a client side transfer, and the examples/xet_pkg_napi directory is a Node native binding example for the same library used outside Python.

The release history shows a project shipping about every two months, with a development build in between. v1.6.0 is titled Stream shard finalization progress, v1.7.0 is titled Telemetry, Dynamic Download Buffers, and fixes, and v1.8.0-dev0 was published on 2026-10-20, two days before the last push to the repository on 2026-10-22.

Those titles are worth reading for a different reason. They are the shape of the remaining work: finalising a shard as a stream ends, buffering downloads dynamically, and the telemetry machinery described above. For a user, the practical boundary is that this is infrastructure inside the hub client, with a stable public surface in huggingface_hub and a Rust layer underneath that moves quickly.

Editorial conclusion

xet-core suits engineers who work with large model and dataset repositories and want the transfer and storage savings that chunk-level deduplication brings, and who are already using huggingface_hub and will therefore get it without changing anything. It does not suit you if you need to drive the client directly, because the README states it is not meant for that, and it does not suit a workflow that requires Git LFS semantics unchanged, since the point of Xet is that objects are chunked. Check three things before you rely on it: which environment variables in your deployment already disable telemetry, since the hub level opt-outs take precedence over the Xet one, whether your client is new enough to pull in the hf-xet wheel, and whether the chunk cache location suits your disk budget. The last push to the repository was on 2026-10-22, the workspace version is 1.7.0, and a 1.8.0 development build was published on 2026-10-20.

Frequently asked questions

Official sources

1. huggingface/xet-core on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases