Conduit: a Go streaming tool that replaces Kafka Connect without a JVM

The pipeline is goroutines and channels, not a worker topology

Conduit's architecture is described plainly in the README. It is a data streaming tool written in Go, and its pipelines are built out of simple building blocks which run in their own goroutines and are connected using Go channels, which the README says makes them performant on multi-core machines. Two ordering guarantees follow from that design and are the parts worth knowing. Conduit guarantees the order of received records will not change, and it takes care of consistency by propagating acknowledgements to the start of the pipeline only when a record has been successfully processed on all destinations. That second property is the interesting one, because it is the difference between at-least-once and at-most-once behaviour at the source. A record is only acknowledged upstream once every destination has handled it, so a failing sink holds back the checkpoint rather than letting data be dropped silently. The same repository structure backs it up, with pkg/, cmd/, proto/, tests/ and a two_engines_test.go at the top level, and the module file pins Go 1.26.9 with a replace-aware dependency set that includes the schema registry, the connector SDKs and the processor SDK.

Connectors are gRPC plugins, so they can be written in any language

The plugin boundary is the most consequential design decision in the project. Conduit connectors are plugins that communicate with Conduit via a gRPC interface, and the README states plainly that this means plugins can be written in any language as long as they conform to the required interface. That is the difference between Conduit and the tool it positions itself against. A JVM-based connector framework ties every connector to one runtime and one dependency tree, while a gRPC contract lets a connector be a Go program, a Python process or a service written in anything else that speaks the protocol. The shipped connectors are separate Go modules pulled in as dependencies, covering file, generator, Kafka, log, Postgres, S3 and the protocol connectors, alongside separate SDK modules for connectors and processors, a schema registry, and an ecdysis module that handles schema adaptation. There is also a kcdrift directory at the top level, which points at the project's attention to drift between Kafka Connect setups and its own. The practical effect is that extending Conduit does not mean writing in Go, even though the core is Go.

Install paths, and the macOS quarantine trap worth knowing about

The recommended install is a script that detects your platform and installs through the native package manager, Homebrew on macOS or dpkg or rpm on Linux.

curl https://conduitdata.io/install.sh | bash

Homebrew is the route to prefer on macOS.

brew update
brew install conduit

Or download a release binary and run it.

./conduit run

That last path has a documented macOS wrinkle. The release binaries are ad-hoc signed, not Developer ID signed and not notarized, so a binary downloaded through a browser carries the com.apple.quarantine attribute and Gatekeeper refuses to run it with no error message at all, killing the process and leaving an empty terminal. The fix is to clear the attribute before the first run.

xattr -d com.apple.quarantine ./conduit

The README notes that curl and wget do not set the attribute, so a binary fetched that way runs as-is, and that Homebrew and the install script are unaffected. Linux packages are also published, with dpkg for Debian and rpm for RPM based systems.

dpkg -i conduit_<version>_Linux_<arch>.deb

Building from source needs Go 1.26 or newer and Make.

git clone [email protected]:ConduitIO/conduit.git
cd conduit
make
./conduit run

The quickstart proves the pipeline in one command

The fastest way to see Conduit working is the quickstart command, which scaffolds and runs a demo pipeline with a single command.

conduit quickstart

Sample records flow to your console within seconds, and the state is in-memory with nothing written to your working directory, so Ctrl-C stops it cleanly. That last property is what makes it a good first contact with the tool, because there is no state directory to clean up and no configuration to write before you see anything move. The longer route to the same result is the Docker Compose stack in deploy/docker-compose, which runs a pinned, signed release image with a pipeline already running, a generator writing to a file, Conduit state on a named volume, a readiness health check and a graceful stop period.

git clone https://github.com/ConduitIO/conduit.git
cd conduit/deploy/docker-compose
docker compose up -d --wait
docker compose exec conduit tail -n 3 /data/records.ndjson
docker compose down            # add -v to delete the state volume

The compose file publishes the API and UI on http://localhost:8080 on loopback only, and the README gives the reason plainly, that the API is unauthenticated. Swapping pipelines/quickstart.yaml for your own pipeline is the documented way to go further, and CI runs this same stack end to end under the deploy-quickstart job. The deploy directory also carries systemd, ECS and Nomad examples, which the README is careful to call examples rather than supported products.

The HTTP API is the control surface, and it has no authentication

Once the service is running the configured pipeline starts processing records automatically, and the README directs you to the Conduit API at http://localhost:8080/openapi to explore the HTTP API through Swagger UI. That is a deliberate choice to make the control surface an HTTP API rather than a configuration file alone, and it is what lets tooling drive pipeline state programmatically. The Dockerfile exposes two ports, 8080 for the HTTP API and 8084 for the gRPC API, which lines up with the plugin architecture, since the gRPC side is how connectors and processors talk to the core. The security consequence is straightforward and the project acknowledges it: the API is unauthenticated, which is why the compose example binds it to loopback only. If you intend to expose it beyond a single host, that is a change you have to make deliberately, by putting an authenticating proxy in front of it or by restricting network access, and the README does not offer a built-in mechanism for it. The API surface also has generated reference documentation under pkg/http/openapi, and the module dependencies include a gRPC gateway, so the HTTP layer is a translation over the gRPC API rather than a separate implementation.

Release cadence, the nightly tags and the Kafka Connect comparison

Conduit was created and open-sourced by Meroxa, and the project describes itself as a Kafka Connect replacement that requires no JVM. The release list shows both a stable line and nightly tags side by side, with v0.20.5 dated 2026-10-20 and nightly v0.21.0-nightly.20261020 from the same day, alongside v0.21.0-nightly.20261017. So there is an in-progress next minor version being published continuously while the current one receives patches. The last push to the repository was on 2026-10-22, so the work is current rather than archived, and the project is licensed under Apache-2.0. On quality, the Makefile runs the test suite with the race detector over the whole module, and coverage is written to a profile that CI feeds to a ratchet script against a stored baseline, which is a stricter arrangement than a plain coverage report. The comparison with Kafka Connect is worth stating honestly rather than repeating. Conduit offers a single binary, no JVM, a language-neutral plugin contract and first-class observability output, and it is written in Go with Go 1.26 in the module. What it does not yet claim is the breadth of an established connector ecosystem, and its own README frames the goal as the best user experience for building and running real-time pipelines rather than as feature parity.

Where Conduit is the wrong tool

There are clear cases where the Kafka Connect model still wins. If your organisation already runs Kafka Connect with a large set of connectors and a working operational practice, Conduit asks you to replace that, and the connector set here is smaller, covering file, generator, Kafka, log, Postgres, S3 and protocol. If you need the mature exactly-once semantics and transaction handling of the JVM ecosystem, the acknowledgement behaviour described in this project, where the source is only acknowledged once every destination has processed the record, is a different trade-off and needs evaluating against your failure modes. If you need schema registry features beyond what this project ships, the separate schema registry and ecdysis modules are where the work would land. And there is an operational wrinkle on macOS that bites exactly the users who try the tool casually, since a browser-downloaded binary is killed by Gatekeeper with an empty terminal and no message, which looks like a broken build rather than a quarantine attribute. Each of these is a reason to check before committing, not a reason to dismiss the project.

Editorial conclusion

Adopt Conduit when you want a lightweight streaming layer with connectors written in any language, a single binary and no JVM in the picture, and skip it when you need the breadth of an existing Kafka Connect deployment with its mature connector catalogue and exactly-once semantics you can rely on. Verify first that the connectors you actually need exist, since the shipped set covers file, generator, Kafka, log, Postgres, S3 and protocol, and check the HTTP API exposure on your network, because the API is unauthenticated and the compose example publishes it on loopback for that reason.

Frequently asked questions

Official sources

1. ConduitIO/conduit on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases