Fluentd: what the Ruby log collector actually does, and how to install it

What Fluentd collects, and who ends up running it

Fluentd is a log collector with a plugin model. The README states that it "collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on." That sentence is the whole product thesis: instead of every application shipping its own log shipper, you run one agent that speaks to everything on both ends.

The people who end up running it are the ones with a mismatch problem. A service writes JSON to stdout, a legacy daemon writes syslog, a database writes to a file, and the destination is a managed log service. Fluentd sits in the middle and normalizes the stream. The repository leans into this with an example directory that is effectively a catalogue of the input and output plugins: in_tail, in_http, in_forward, in_syslog, in_tcp, in_udp, out_file, out_forward, out_copy, out_exec_filter. Each file is a working config fragment rather than documentation prose, which is a reasonable way to learn the syntax.

It is not a log search product. There is no query language, no index, no retention policy of its own. Fluentd moves and reshapes events; whatever you point it at stores them. Teams that expect a UI will be disappointed, and that is a design boundary rather than a gap.

The forward protocol and the buffer are the actual mechanism

Two things define how Fluentd behaves in production, and neither is obvious from the tagline.

The first is the forward protocol. Inputs attach a tag to each event, and outputs match tags with a pattern. The example files show the shape: in_forward receives events over the network, out_forward sends them to another Fluentd, and in_forward_shared_key, in_forward_tls and in_forward_users add authentication and transport security to that same channel. This is why Fluentd deployments often look like a tree: edge agents forward to an aggregator, and the aggregator writes to storage. The forward plugin is the seam between those layers.

The second is buffering. The example set includes out_forward_buf_file and out_forward_client, which is a hint about where the interesting failure modes live. When a destination is slow or unreachable, the output buffer is what decides whether events are retried, spilled to disk, or dropped. The README does not document retry or buffer tuning, so treat the example configs and docs.fluentd.org as the starting point rather than the README.

Filters sit between input and output. example/multi_filters.conf and example/filter_stdout.conf show the chaining model: events arrive tagged, filters rewrite or drop them, outputs route by tag. That is the entire data flow, and it is small enough to hold in your head once you have read one config.

Installing Fluentd and sending your first event

The README's Quick Start is four lines and assumes a Ruby toolchain. The stated prerequisite for development is Ruby 3.3 or later, so check that first.

gem install fluentd

Next, generate a skeleton configuration directory. The README uses `-s conf`, which creates the directory and a starter fluent.conf inside it.

fluentd -s conf

Start the daemon against that config. The trailing ampersand in the README runs it in the background.

fluentd -c conf/fluent.conf &

Now push an event through the forward input. fluent-cat is the client that ships with the gem, and the README pipes a JSON payload into it with the tag debug.test.

echo '{"json":"message"}' | fluent-cat debug.test

What you should see depends on what the generated fluent.conf routes debug.test to. If it writes to stdout, the event appears in the terminal. If you want a known destination, the repository's example/out_file.conf and example/in_forward.conf are the two files to read side by side, since they show the input and the output halves of the same pipeline. The README does not document what the generated config contains, so open conf/fluent.conf after generating it rather than assuming.

Running Fluentd from source, and what the test suite tells you

If you are patching Fluentd or building against a fork, the README gives the bundler path. It installs bundler, then dependencies into vendor/bundle rather than the system gem path.

gem install bundler
bundle install --path vendor/bundle

The test command is a single rake task, and the README shows how to narrow it with the TEST environment variable, including glob patterns.

bundle exec rake test TEST=test/test_*.rb

That glob form is worth noting because it is the practical way to work on one plugin without waiting for the full suite. The README also mentions that git must be on PATH, and that on Windows Github for Windows and GitShell are the suggested setup. That is the only Windows-specific guidance in the README, and it is about development rather than running the daemon.

What the README does not cover is packaging. There is no Dockerfile in the top-level repository entries, and no container instructions in the Quick Start. If you want Fluentd in a container, the README points you at docs.fluentd.org and the website instead of describing an image itself.

Where Fluentd is the wrong tool

The clearest limitation is the runtime. Fluentd is a Ruby process, and the README lists Ruby 3.3 or later as a prerequisite. On a node where you are trying to keep the footprint tiny, a Ruby runtime plus a gem dependency tree is a real cost, and it is the reason Fluent Bit exists as a separate project. The README does not compare the two, so the honest framing is that the choice is about plugin breadth versus per-node weight, not about one being strictly better.

A second limitation is that the README is thin on operations. It documents installation, development and where to find more information, but not buffer sizing, retry behaviour, backpressure, or what happens when the output buffer fills. Those are exactly the questions that decide whether a log pipeline loses data during an incident. The example configs hint at the surface area (out_forward_buf_file exists for a reason) but the README does not explain it.

A third case: if you need to query logs, Fluentd is the wrong layer. It has no storage or search. Pointing Fluentd at a destination and then expecting to investigate incidents in Fluentd itself is a category error.

Finally, version support is not open-ended. The README states that v0.12 is deprecated and that support was stopped, with a link to a 2019 announcement. Anyone still on that line is running an unsupported branch.

Fluentd versus Fluent Bit, and versus Logstash

Fluent Bit is the comparison that matters most, because both come from the same community and both are log processors. The difference in approach is the runtime and the resulting footprint: Fluentd is a Ruby process with a plugin ecosystem built around gems, while Fluent Bit is the lighter agent that the ecosystem positions for constrained nodes. In a typical deployment that difference becomes an architecture: Fluent Bit on every node doing minimal collection, Fluentd as the aggregator doing the heavier routing and enrichment. The README does not spell this out, so treat it as the community framing rather than a documented recommendation.

Logstash is the other common comparison. The approaches differ in where the processing language lives: Fluentd expresses pipelines as tagged inputs, filters and outputs in fluent.conf, and extends behaviour through plugins written in Ruby. That makes the config declarative and the extension path a real programming language, which is either an advantage or a barrier depending on whether your team writes Ruby.

If you are already invested in the Elastic stack, Logstash is the path of least resistance. If you want one collector feeding many unrelated destinations through a plugin per destination, Fluentd's tag-matching model is the more direct fit. Neither is a drop-in for the other; the config formats do not translate.

Licence, releases and the cost of staying current

Fluentd is Apache-2.0, and the README states the copyright as 2011-2021 Fluentd Authors. Apache-2.0 is a permissive licence with an explicit patent grant, which is the usual reason it is acceptable in commercial products, but the specifics of your situation are for your own legal review rather than something this article can settle.

The release cadence visible in the repository is uneven in a way worth planning for. v1.19.3 was released on 2026-06-25, v1.19.2 on 2026-02-19, and v1.16.11 on 2025-12-10. That last one matters: a 1.16 line was still receiving releases while 1.19 was current, which tells you older lines get backports for a while. There is a backport workflow in .github and a Backport Pull Requests badge in the README, so the mechanism is automated rather than manual.

The upgrade cost is mostly in plugins. Fluentd itself is a gem, so `gem install fluentd` or a bundler update is the mechanical part. The risk is that a third-party output plugin has not caught up with the version you are moving to, and the README gives no compatibility matrix. The last push to the repository was on 2026-09-20, and master is described as the branch for v1 development.

Editorial conclusion

Adopt Fluentd when you need many input and output plugins in one process and you are willing to run Ruby 3.3 or later. Skip it when a single small binary per node matters more than plugin breadth, which is the case Fluent Bit is built for. Before committing, check the v1.19.x release notes against the plugins you plan to use and confirm your Ruby version meets the README's stated prerequisite.

Frequently asked questions

Official sources

1. fluent/fluentd on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases