# obstore: one Python client for S3, GCS and Azure object stores

> obstore is a Python interface to Amazon S3, Google Cloud Storage, Azure Storage and other S3 compliant APIs, implemented in Rust through PyO3 with streaming everywhere and no required Python dependencies. It suits data pipeline and geospatial workloads, and it trades the fsspec ecosystem for a narrower, faster core.

**developmentseed/obstore** — The simplest, highest-throughput Python interface to S3, GCS & Azure Storage, powered by Rust.

- Repository: https://github.com/developmentseed/obstore
- Website: https://developmentseed.org/obstore
- Stars: 829 · Forks: 45
- Language: Python
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/developmentseed-obstore

## One interface over three object stores

obstore is presented as the simplest, highest throughput Python interface to Amazon S3, Google Cloud Storage, Azure Storage and other S3 compliant APIs, powered by Rust. The pitch is a single API surface over several backends, and the emphasis on no required Python dependencies is the part that surprises people who arrive from the cloud SDK world. The alternative to installing boto3, google-cloud-storage and the Azure SDK side by side is installing one package that carries its own Rust implementation of each protocol.

The library targets a specific kind of workload. The list of features points the same way: streaming downloads with configurable chunking, streaming uploads from files or from sync and async iterators, streaming list with no need to paginate, automatic multipart uploads for large file objects, and automatic credential refresh before expiration. These are the mechanics of moving a lot of data in and out of object storage from an application rather than from an interactive script.

That also explains the audience. Pipeline authors who today juggle several vendor SDKs, and geospatial workloads where the example directories include a zarr integration and a FastAPI service, are the ones who feel the duplication most.

## Rust core, PyO3 boundary, zero-copy buffers

The implementation is a Rust workspace with a single member, obstore, and the Python package is built with maturin, which appears in the development dependency list along with the maturin import hook. The binding layer is PyO3 at version 0.30 with the macros and indexmap features, and the async runtime bridge is pyo3-async-runtimes with the tokio-runtime feature.

The data path is what makes the Rust choice more than a packaging detail. The README lists zero-copy data exchange between Rust and Python via the buffer protocol, so bytes read from the store can be handed to Python without a copy in between. Combined with a file-like object API, that is what lets a download be treated as a file rather than as a fully materialised bytes object.

The core is the object_store crate at version 0.14.0, pulled in with the aws, azure, gcp and http features. That single crate is the reason the Python side can speak to S3, GCS, Azure and generic HTTP endpoints without four separate implementations in the repository.

The release profile in Cargo.toml is tuned accordingly: lto is enabled and codegen-units is set to 1, both of which trade build time for runtime performance in a compiled wheel.

## Streaming as the default rather than an option

The streaming features are the core of the design and they are worth understanding separately from the interface.

Downloads stream with configurable chunking, so the caller decides the granularity of the buffer rather than receiving one enormous object. Uploads stream from files or from sync and async iterators, which means a generator can feed an upload without buffering the payload, and a multipart upload is started automatically when the object is large enough to need one. Listing streams too, with no need to paginate manually, which removes the loop that every hand written S3 listing code tends to get subtly wrong.

Automatic credential refresh before expiration sits in the same list, and it matters for long running jobs: a pipeline that runs for hours does not fail because a token expired midway.

Both a sync and an async API are provided with full type hinting, so the same object store can be used from a script and from an event loop without an adapter layer in between.

## Arrow output and an fsspec escape hatch

Two integration choices deserve a mention because they tell you what obstore thinks it is for.

The first is optional Apache Arrow output for list results, which the README describes as faster and more memory efficient than building Python dictionaries. Listing a bucket with a million objects and getting back a million dicts is a common way to turn a cheap operation into an expensive one, and the Arrow path avoids it. The development dependencies include pyarrow, and there is a separate obspec package, which is the specification interface for object storage interfaces.

The second is fsspec integration, alongside the file-like object API. This is the pragmatic part of the design. obstore is not trying to replace the fsspec filesystem abstraction, it is offering a faster engine for the operations that matter while still handing back something the wider Python data ecosystem already understands. The README also gives preliminary benchmark figures, roughly nine times higher throughput than fsspec and 2.8 times higher than aioboto3, for many concurrent small get requests from an async context, and it is careful to label the benchmarking as ongoing and the results preliminary. Those numbers describe one access pattern, and small concurrent reads are exactly where a Rust core with connection reuse wins.

## Installing from pip or conda-forge

The installation is a single command either way. From PyPI:

```bash
pip install obstore
```

Or from conda-forge, which the README points out can also be installed with conda, mamba or pixi:

```bash
conda install -c conda-forge obstore
```

There is no compiler step for the user, since the release ships as a wheel. The Rust source in this repository is what the maintainers build, and the workspace is configured with rust-version 1.75 as the floor, so anyone rebuilding from source needs a reasonably current Rust toolchain and a Python environment matching the packaged metadata, where the tooling targets py311 and the documentation group expects mkdocs with the material theme, mkdocstrings, mkdocs-jupyter and mike for versioning.

The full documentation lives at developmentseed.org/obstore, and the README directs new users to the Getting Started page rather than restating the API here.

## The workspace split and a patch waiting on the next upstream release

The Cargo workspace has one member, obstore, and deliberately excludes two directories, pyo3-object_store and pyo3-bytes. The comment explains why, and it is a nice illustration of a real constraint of the ecosystem.

The object_store crate does not yet export a list stream to Python in a form that works, so the project patches the object_store version to get one. That patch lives in the arrow-rs pull request 6619, and the comment says it will be included in the next major release of object_store. The reason the directories are excluded from the workspace rather than included is separate: pyo3-object_store is published to crates.io, and crates.io cannot have git dependencies, so it has to live outside the workspace to keep its publishable form.

The two excluded directories are also useful on their own. The README points Rust developers who want to add object_store support to their own Python packages at pyo3-object_store, which is a reusable binding rather than part of this application. In other words, obstore is not only a client library but also a source of a building block for other projects.

The examples directory shows the intended usage shapes: a FastAPI service, a MinIO setup, a progress bar, a stream zip and a zarr integration. The FastAPI example is even its own uv workspace member, and the pyproject.toml dev group pulls in FastAPI, tqdm, boto3, aioboto3-core, azure-identity, google-auth, minio, docker, fsspec, pystac and pyarrow so that the examples and the pyright checks in CI can run.

## Where obstore is the wrong tool

There are clear cases for a different choice.

If your code already lives inside the fsspec ecosystem, obstore is the wrong shape even if it is the faster engine. fsspec is a Python level abstraction with a pluggable backend protocol and a wide range of implementations, and code that passes a filesystem object between tools depends on that protocol. obstore covers the major cloud backends and S3 compatible endpoints, and it does not present itself as a replacement for the whole fsspec backend catalogue.

The second limit is dependencies. The selling point of no required Python dependencies cuts both ways. If your environment already standardises on boto3 or google-cloud-storage for other reasons, the vendor SDKs are already installed, their credential chains and their configuration files are already configured, and obstore gives you a second path to the same service rather than replacing the first.

Third, the performance claim is narrow and should be read narrowly. The figures in the README cover many concurrent small get requests from an async context, which is a pattern that favours a Rust client with its own connection handling. A workload dominated by large sequential reads, by many small writes with high per request latency, or by a provider specific feature that has no equivalent in the object_store crate will not necessarily see the same gap. The README also says the benchmarking is ongoing, so treat the numbers as direction rather than a guarantee for your workload.

Finally, note the packaging metadata oddity in this repository: the root pyproject.toml declares the project name as test-env at version 0.1.0 with no dependencies, while the actual published package is obstore. The file is really a tool configuration for the docs and the examples, and it is a reminder to read the README rather than to trust the manifest.

## Conclusion

Adopt obstore if your Python workload reads or writes objects in bulk, especially many small objects over concurrent connections, and you want one code path across S3, GCS and Azure without pulling in the boto3 and google-cloud-storage stacks. Do not adopt it if you rely on the fsspec ecosystem of pluggable backends, or if you need the extra Python packages obstore leaves out. Verify three things first: how your credentials are supplied, since refresh before expiration is automatic but the provider specific setup still has to be correct; that Arrow list output and the file-like API behave as you expect against your backend, rather than only against the S3 compatible endpoints in the examples; and the Python version your environment runs, since the packaging targets py311 and the Rust toolchain expects 1.75 or later. The licence is MIT and the last push was on 2026-10-19, with py-v0.12.0 released on 2026-10-15.

## FAQ

### What is obstore and which backends does it support?

It is a Python interface to Amazon S3, Google Cloud Storage, Azure Storage and other S3 compliant APIs, implemented in Rust. One interface covers all of them, and the library requires no Python dependencies of its own.

### How do I install obstore?

Use pip install obstore from PyPI, or install from conda-forge with conda install -c conda-forge obstore, which also works with mamba and pixi.

### Does obstore support async use and streaming?

Yes. It offers a sync and an async API with full type hinting, streaming downloads with configurable chunking, streaming uploads from files or from sync and async iterators, and streaming list with no need to paginate.

### How does obstore compare with fsspec?

The README describes preliminary results of roughly nine times higher throughput than fsspec and 2.8 times higher than aioboto3 for many concurrent small get requests from an async context, and notes that benchmarking is ongoing. It also offers fsspec integration alongside its file-like object API.

### Can Rust developers reuse obstore's object_store binding?

Yes. The README points Rust developers who want object_store support in their own Python packages at the pyo3-object_store crate, which lives outside the Cargo workspace so it can be published to crates.io without git dependencies.

## Sources

- [developmentseed/obstore on GitHub](https://github.com/developmentseed/obstore)
- [License: MIT](https://github.com/developmentseed/obstore/blob/main/LICENSE)
- [Project website](https://developmentseed.org/obstore)
- [README](https://github.com/developmentseed/obstore/blob/main/README.md)
- [Releases](https://github.com/developmentseed/obstore/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/developmentseed-obstore