PicoMQ: durable streams parked on object storage

The stream data lives on object storage

The design decision that shapes everything else is that the stream payloads do not live in a broker's memory or on a dedicated log volume. They live in S3-compatible object storage. On top of that substrate, PicoMQ serves durable, real-time streams to clients over two frontends: an HTTP interface speaking both the Pico protocol and Durable Streams, and a Kafka interface. The stream engine itself lives in a separate repository, s3stream, while this repository holds the host, meaning the metadata plane, the server, the protocol frontends, the client library, and the command line tool. Splitting the engine out is what lets the storage layer be swapped without the serving layer noticing. The metadata plane is deliberately separate from the payload store, and it is where the SQL dependency comes in: a single node uses a SQLite metadata log, while the workspace also carries a Postgres write-ahead log component for the clustered path.

Every flag has a PICO_ environment equivalent

Configuration is deliberately boring, which is a real virtue for something meant to run in containers. Each command line flag has an environment variable equivalent under a PICO_ prefix, so a deployment can choose between a flags file and an environment block without changing behaviour. A minimal single node needs two settings: where the metadata log lives and where objects are stored. That looks like this:

# single node: SQLite metadata log, local object storage
pico serve \
    --meta-url sqlite:./data/meta.db \
    --storage=-2@file://./objects

Note that the storage flag is not a plain URL. It carries a numeric prefix before the scheme, which is how the same flag expresses placement rather than just location. Operational endpoints are separated from the data plane: health and readiness checks are served on a dedicated admin listen address that defaults to loopback on port 9090, so a liveness probe does not have to traverse the public listener.

Authentication is off until you ask for it

The default posture is permissive, and the documentation is direct about it. Authentication is off by default. Binding to anything other than loopback therefore requires an explicit decision: either demand authentication, or pass the flag that explicitly allows remote access without it. That two-option design is a reasonable middle ground, because the flag name itself records that you have made a choice rather than overlooked one, which is not true of many services that simply have no auth. The same attention appears in the admin surface, which is loopback by default and separate from the protocol port. For a deployment where the node is reachable from a network you do not fully control, the choice between requiring auth and allowing remote access is the first decision to make, and it should be made per environment rather than inherited from the single-node quick path.

Streams have an explicit lifecycle

The command line client treats a stream as a resource with a lifecycle rather than as a topic you publish into and forget. You create a stream and declare its content type, append to it, read it back, follow it, and then close and delete it. The whole cycle is visible in the tool:

pico create /streams/orders --content-type text/plain
seq 1 1000 | pico append /streams/orders --batch 100
pico read /streams/orders
pico tail /streams/orders -f
pico close /streams/orders && pico delete /streams/orders

Two details carry most of the weight. The append path takes a batch size, so bulk ingestion is a pipe into a single command rather than a thousand invocations, and tail with the follow flag gives the live consumer behaviour from the same binary. Declaring the content type at creation means the stream's contract is fixed up front instead of inferred later. There is also a benchmarking mode on the same binary, invoked with an HTTP 2 flag and parameters for batch size, payload size, connection count, stream count, and duration, which suggests the throughput claims are meant to be reproducible rather than asserted.

The binary name collides with a common Linux tool

An amusing operational hazard comes from the name. The client is called pico, and there is a long-standing Linux display compositor with the same name, so a machine that already has the compositor installed can end up with the wrong binary first on the path. The install puts the new binary in the standard cargo bin directory, and the fix when the collision happens is to prepend that directory:

export PATH="$HOME/.cargo/bin:$PATH"

Installing from source is a single cargo install against the workspace member rather than a registry package:

cargo install --path picomq/pico-cli

For iterating on the code itself there is also the option to run it in place through cargo run with the package name and arguments, which avoids polluting the global bin directory during development. Nothing prevents installing from crates.io if a published version exists, but the documented path here points at the workspace path, which is consistent with a project at version 0.1.1.

Compose files stage four different deployments

The harness directory holds a set of ready-made deployments rather than one. You copy an example environment file, then pick a compose file for the shape you want. Running the base stack brings up Postgres and an object store, which is RustFS, with a single node. A second compose file runs the same stack as a two node cluster. A lite file removes the dependencies entirely and runs SQLite with file-based storage, which is the fastest way to see the thing work. Adding a further compose file on top of the lite one brings in the connectors runtime. There is also a separate harness that runs the same setup against an existing Postgres and object store of your own, configured through the environment file. The port map is fixed across these: the Pico endpoint on 4437, with the cluster variant adding 4438, the dashboard on 9090, and the connectors API on 8081.

The workspace carries fifteen sinks and four sources

The workspace definition shows how much of this repository is the connector surface rather than the stream core. Alongside the thirteen pico crates, which split common, protocol, auth, metadata, SQL, the Postgres write-ahead log, server, Kafka, HTTP, runtime, client, CLI, and schema concerns, there are twenty-eight connector members. Those are an SDK, a runtime, fifteen sinks, and four sources. The sinks reach Postgres, ClickHouse, Elasticsearch, HTTP, InfluxDB, Meilisearch, MongoDB, Quickwit, S3, SurrealDB, Doris, Iceberg, Delta, and Redshift, plus stdout for debugging, while the sources are random, Postgres, Elasticsearch, and InfluxDB. The container build mirrors that split: a Node stage compiles the dashboard and copies it into the HTTP crate, a Rust stage produces a locked release binary using persistent build caches so rebuilds stay incremental, and the runtime image is a slim Debian carrying only the certificate bundle and the SQLite library.

Editorial conclusion

PicoMQ suits a team that already stores its data in an S3-compatible bucket and wants durable append-only streams without standing up a Kafka cluster or a separate broker tier, especially if HTTP is the interface their clients already speak. It is a poorer fit if you need the full Kafka protocol surface, since Kafka is one of two frontends rather than the centre of the design, or if you want a hosted service rather than a binary you operate. Before deploying, decide how much metadata durability you actually need, since the default single-node path uses a SQLite log while the Postgres-backed path is only exercised by environment-gated tests, and turn on authentication deliberately, because it ships disabled and any non-loopback bind has to opt in explicitly.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. PicoMQ/picomq on GitHub
3. Project website
4. README
5. Releases