S3Proxy: put an S3 API in front of filesystems, Azure, GCS, Swift and SFTP

The gap S3Proxy fills: S3 clients, non-S3 storage

A large amount of tooling speaks S3 and nothing else. Backup agents, data pipelines, SDKs and test suites are written against the S3 API, and switching the storage underneath them usually means rewriting the client. S3Proxy takes the opposite approach: it implements the S3 API and proxies requests, translating them to Google Cloud, Microsoft Azure, OpenStack Swift, SFTP, or a plain local filesystem. The README frames the use cases as translation between S3 and those backends, testing without Amazon by using the local filesystem, extension via middlewares, and embedding into Java applications.

The audience is therefore narrow but specific. Platform engineers who have storage in Azure or GCS but clients that only understand S3 are the primary case. So are developers who want S3-shaped tests that run on disk instead of in a cloud account. The project is written in Java and requires Java 17 or newer to run, which matters if your build images are still on older runtimes.

How requests flow: a properties file, a blobstore, and middlewares

Configuration lives in a properties file. Keys prefixed s3proxy describe the proxy itself, such as s3proxy.authorization and s3proxy.endpoint, while keys prefixed jclouds describe the storage backend, such as jclouds.provider and jclouds.filesystem.basedir. The proxy accepts an S3 request, maps it onto the configured backend through the Apache jclouds abstraction, and returns an S3-shaped response.

Backends are selected by name. The README lists aws-s3, azureblob, filesystem, google-cloud-storage, openstack-swift (Keystone v3 only), sftp via Apache MINA SSHD, and transient for in-memory storage. Bucket assignment is explicit: s3proxy.bucket-locator.1 and s3proxy.bucket-locator.2 name buckets, and glob syntax can cover many buckets at once. A bucket or glob cannot be assigned to multiple backends, which is a deliberate constraint rather than an oversight.

Between the proxy and the backend sits a middleware layer. The README names bucket aliasing, bucket prefix scoping, bucket locator, eventual consistency modeling, large object mocking, latency, read-only, regex rename blobs, sharded backend containers, storage class override, user metadata replacer, and no cache override. Several of these exist to simulate behaviour a real S3 endpoint has and a filesystem does not, which is the honest reason a local-filesystem backend is useful for testing: you can inject latency or eventual consistency on purpose. The credentials story has a similar split. jclouds.identity and jclouds.credential name what the README calls the enduring half of a backend credential, and jclouds.session-token names the expiring third part, such as an AWS session token, an Azure shared access signature, a Google OAuth 2.0 access token, or a Keystone token. Three credentials are read only when the store is built because nothing about them expires: an Azure account key, a Google service account key, and the sftp password. Everything else can be refreshed.

Installing S3Proxy and running a first bucket against the filesystem

The README points at three routes. Docker Hub hosts an image with its own instructions. Reference manifests under examples/kubernetes/ show how to wire that image into Kubernetes, including health probes, graceful shutdown and Secret-based credentials, and the README notes a third-party s3proxy-chart for Helm users. Without Docker, releases are downloadable from GitHub, and developers build from source with mvn package, which produces a binary at target/s3proxy.

For a first run, start with the filesystem backend and anonymous access, which is the example the README gives. Write a properties file:

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy

The basedir has to exist before the proxy starts:

mkdir /tmp/s3proxy

On Linux and macOS the executable jar runs directly. On Windows the README requires invoking java explicitly:

chmod +x s3proxy
s3proxy --properties s3proxy.conf

Then create a bucket and list it. The README uses curl for both, and the second response is a ListAllMyBucketsResult XML document containing the bucket you just created:

curl --request PUT http://localhost:8080/testbucket
curl http://localhost:8080/

At that point you have a working S3 endpoint backed by a directory, with no credentials and no cloud account. Point an S3 client at http://localhost:8080 and it should behave as it would against S3 for the basic bucket and object operations. The README does not document rollback or a migration path off a backend, so treat the basedir contents as the source of truth.

What S3Proxy deliberately does not implement

The README's limitations list is long and worth reading before you commit. S3Proxy does not support ACLs other than private and public-read, including the x-amz-grant-* headers and any grant naming a specific grantee. It does not support BitTorrent hosting. Bucket inventory, analytics and metrics configuration are out. Bucket lifecycle configuration is out. Bucket logging is out. Bucket notification configuration is out. Bucket policies and bucket policy status are out. Bucket replication is out. CORS bucket operations are out.

That list rules out a whole class of deployments. If your application relies on lifecycle rules to expire objects, or on bucket policies for cross-account access, or on notifications to trigger downstream processing, S3Proxy is the wrong tool, and no amount of middleware will change that because the README states these are unsupported. The ACL restriction is the subtler one: private and public-read cover a lot of ground, but anything that names a specific grantee does not work, so an application that shares objects with individual accounts will break.

The credential model has a related sharp edge. A token written into the properties file serves no longer than the token itself does. Deployments that rotate credentials are expected to embed S3Proxy and hand BlobStores.create a Supplier<Credentials>, which is asked again for every request it signs rather than keeping the first answer. If you run the standalone binary with a static session token, you own the refresh problem yourself.

S3Proxy compared with MinIO and with nginx as an S3 front end

The closest alternative people reach for is MinIO, and the difference is architectural rather than cosmetic. MinIO is itself an object store: you deploy it and it holds the data. S3Proxy holds nothing. It is a translation layer in front of storage you already have, whether that is a filesystem, Azure Blob, Google Cloud Storage, Swift or SFTP. If you want a new S3-compatible store to own your bytes, MinIO is the shape you want. If you already have bytes in Azure or on an SFTP server and need an S3 API over them, S3Proxy is the shape you want, and MinIO would mean migrating the data.

The other comparison is nginx with an S3 gateway module. That approach also proxies S3 requests, but it is configured in nginx's own configuration language and lives inside the nginx request pipeline. S3Proxy is a Java application with a properties file, a jclouds backend abstraction, and a documented middleware list including eventual consistency modeling and latency injection. The middleware list is the practical difference: simulating eventual consistency or a slow backend is something S3Proxy exposes as configuration, which is aimed at testing rather than at production serving. Conversely, if your team already runs nginx and wants one fewer JVM, the nginx route has a smaller operational surface. The README also mentions embedding S3Proxy into Java applications and using the artifacts from Maven Central, which neither MinIO nor nginx offers.

Maintenance, licence and upgrade cost

The repository is not archived, and the last push was on 2026-08-26. Recent releases run s3proxy-4.1.0 on 2026-08-26, s3proxy-4.0.0 on 2026-08-04 and s3proxy-3.3.0 on 2026-07-04, so the 4.x line arrived recently and a 3.3.0 deployment is two major versions behind. The jump from 3.x to 4.x is the kind of change that deserves a read of the release notes before upgrading, and the README does not describe a compatibility policy between major versions.

The runtime requirement is Java 17 or newer. The Dockerfile shows how tightly the image is trimmed: it builds a custom JRE with jlink from a jdeps module list produced by mvn package -Pjdeps, adds jdk.crypto.cryptoki for PKCS#11 keystores such as FIPS deployments and java.instrument for the -javaagent option that S3PROXY_JAVA_OPTS exists to pass, and includes only the English locale subset because the proxy formats HTTP dates with Locale.US. The practical consequence is that the container image is small, but any deployment that needs an extra JVM agent or a non-English locale has to account for the trimming. The Dockerfile also notes that the base image choice exists because Ubuntu's OpenJDK build keeps jmods, which the eclipse-temurin 25 images dropped.

The licence is Apache-2.0, which is permissive and generally compatible with commercial deployment. This is not legal advice; check how your organisation handles Apache-2.0 attribution and any bundled dependency licences before shipping.

Editorial conclusion

Adopt S3Proxy when you need S3 semantics on top of a backend that does not speak S3, or when you want a local filesystem behind the S3 API for testing without Amazon. Do not adopt it if your clients depend on bucket policies, lifecycle rules, replication or ACL grants beyond private and public-read, since the README lists those as unsupported. Before rolling it out, verify the exact backend you intend to use against the storage backend examples on the wiki, and confirm how your deployment supplies rotating credentials, because a token written into the properties file is only valid for as long as the token itself.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. Release notes