bpftune: a daemon that re-tunes Linux sysctls while it watches them

The problem is 1,624 sysctls and nobody who is allowed to log in

The project starts by counting. The Linux kernel contains a large number of tunables, usually sysctl parameters, and they are typically introduced for situations where there is no single right answer for a configuration choice. On a 6.2 kernel the documentation quotes the count directly:

# sysctl --all 2>/dev/null|wc -l
1624

Sixteen hundred and twenty four tunables is the shape of the problem. An administrator cannot reason about that surface area, and most of them exist because the right value depends on the workload.

The second half of the problem is the operating model. Individual systems get a lot less care and administrator attention than they used to, and the documentation uses the phrase cattle not pets to describe it. Given modern cloud architectures used for most deployments, most systems never have any human administrator interaction after initial provisioning, and given the scale requirements, that is often an explicit design goal rather than a side effect. The README puts it bluntly: no ssh'ing in.

These two observations are related, and the documentation says so. In an earlier era of fewer, larger systems, tuning by administrators was more feasible.

What follows is a useful straw man, because it makes the implicit assumption visible. The caricature of the traditional approach is finding the set of magic numbers that will work for the system forever. The documentation calls it a caricature, and it is right to, but the point it highlights is the real one: that approach assumes systems are static. And that is the assumption bpftune is designed to break, because if you can observe the system, you can tune it, observe the effect of the tuning, and re-tune if necessary.

That loop is the project. Everything else is machinery for it.

Observability at fine grain is what makes per socket tuning possible

The key benefit the project claims is stated as two consequences of the same mechanism. By using BPF observability features, the system can be continuously monitored and adjusted. And because behaviour can be observed at a fine grain, rather than using coarse system-wide statistics, tuning can be done at a finer grain too, such as individual socket policies and individual device policies.

That second point is the one that separates this from the sysctl-tuning tools that came before it. A coarse view of a system tells you that the TCP buffers are too small, and the only lever is a global value that applies to every socket on the machine. A fine-grained view tells you which sockets are actually constrained, so the tool can raise the limit for the connections that need it.

The cost of that granularity is stated in the design principles, and the first principle is minimize overhead. Use observability features sparingly, and do not trace very high frequency events. This is a real engineering constraint rather than a slogan, because BPF programs run on hot paths. A program attached to every packet event is a program that costs you on every packet.

The principles list is worth reading as a whole because it explains the behaviour you will observe. Minimize overhead, as above. Be explicit about policy changes, providing both a what, meaning what change was made, and a why, meaning how does it help, with syslog logging making policy actions explicit with explanations. Get out of the way of the administrator, since the tool can use BPF observability to see if the admin sets tunable values that it is auto-tuning, and if they do it disables auto-tuning of the related feature set. And do not replace tunables with more tunables, since bpftune is designed to be zero configuration, with no options, and an effort to avoid magic numbers where possible.

That third principle deserves emphasis. A tool that tunes your kernel parameters and then needs its own set of parameters is not solving the problem, it is moving it. The zero configuration stance is the reason the tool can be dropped onto a host nobody administers.

The push-pull loop that raises and lowers the same buffer

The most concrete example in the documentation is the TCP buffer sizing case, and it is worth following in full because it is where the design earns its credibility.

The push side: with TCP buffer sizing, the tool often wants to get out of the way of applications and bump up tcp sndbuf and rcvbuf. But at a certain point, that risks exhausting TCP memory.

The pull side: the tool monitors whether it is approaching TCP memory pressure, and if so it tunes down the values it tuned up. In that way the system can find a balance between providing resources and exhausting them. The documentation is honest that in some cases the values do not need tuning up at all and may be fine as they are, and that in other cases those limits block optimal performance, and if they are raised safely, with awareness of global memory limits, the tool can get out of the way of improved performance.

Then there is the latency problem, and this is the part that is specific to TCP. Increasing buffer size leads to latency. To handle that, the tool correlates buffer size changes with TCP smoothed round-trip time, and if the correlation between these exceeds a threshold of 0.7, it stops increasing buffer size.

That correlation threshold is the only hard number in the whole design section, and it is worth noticing what it means. A buffer increase that correlates strongly with rising round-trip time is a buffer increase that is buying throughput with latency. A correlation below 0.7 means the latency change is not tracking the buffer change, so the buffers are probably filling a real gap rather than just adding delay. The design principle calls for push-pull approaches, and this is a concrete instance of it: the tool does not only push, it watches its own effect and pulls back.

The documentation also frames the tunables as options rather than as magic constants, which is the point of a tunable that has no single right answer.

Tuners, strategies, and events are the moving parts

The concepts section names four components, and they are worth understanding as a data flow.

Tuners each manage tunables and handle events sent from BPF programs to userspace via a shared ring buffer. Each tuner has an associated set of tunables that it manages.

Optional strategies let a tuner specify multiple schemes for auto-tuning the same set of tunables, where the choice is guided by an evaluation of their effectiveness. Each strategy specifies a name, a description, a timeout, an evaluation function, and a set of BPF program names in the tuner associated with the strategy. Strategies are set in the tuner init method via bpftune_strategies_add, and there is a coded example in test/strategy. When a strategy times out, the various evaluation functions are called and the highest-value evaluation dictates the next strategy.

That timeout mechanism is the interesting part. Rather than one tuning approach per tuner, the tool can try an approach, measure how well it worked, and switch to whichever alternative scored highest. It is a form of online experimentation, running against your production host.

Events specify a tuner id, which is which tuner the event is destined for, a scenario describing what happened, an associated netns if supported, and information about the event such as an IP address. The tuner then responds to the event guided by the active strategy, increasing or decreasing a tunable value, and describing the event in the log is key, because that is what lets an administrator understand what changed and why.

The architecture section shows how this maps onto the code. bpftune is a daemon managing a set of shared object plugin tuners, each loaded on start-up. Tuners can be enabled or disabled, and a tuner is automatically disabled if the admin changes associated tunables manually. The tuners share a global BPF ring buffer that allows posting events from BPF programs to userspace, and the example given is the sysctl tuner seeing a sysctl being set and posting an event. Each tuner has an id set when it is loaded, and each has a BPF component built using a BPF skeleton plus a userspace component with init, fini and event_handler entrypoints loaded from the shared object.

Four tuners ship, and the fourth is the one that watches for you

The supported tuners list is short, and each entry names a manual page in the docs directory.

The TCP connection tuner auto-tunes the choice of congestion control algorithm, documented as bpftune-tcp-conn. The IP fragmentation tuner auto-tunes IP fragmentation memory limits to support fragment reassembly, documented as bpftune-ip-frag. The neighbour table tuner auto-tunes neighbour table sizes by growing tables when approaching full, documented as bpftune-neigh. And the sysctl tuner monitors sysctl settings and, if the administrator sets a value that collides with an auto-tuned value, does something about it.

Three of these are straightforward: they adjust a value in response to an observed condition, growing a table when it fills or raising a limit when something is constrained.

The sysctl tuner is the interesting one, because it implements the get out of the way of the administrator principle directly. If a human sets a sysctl that the tool is also managing, the tool has to notice and back off, otherwise it will overwrite the administrator a moment later and the administrator will think the system is haunted. The mechanism for noticing is the same BPF observability used for everything else, which is a neat bit of design: the same hook that lets the tool tune also lets it detect interference.

The repository layout is consistent with a project that expects tuners to be written rather than only used. There is a src directory, an include directory, a docs directory with the manual pages, a sample_tuner directory, and a test directory. Alongside the code are a set of documents that tell you what a serious operational tool should ship: CASE_STUDIES.md, FEATURE_PLAN.md, TESTING.md, TROUBLESHOOTING.md, SECURITY.md, CONTRIBUTING.md and a THIRD_PARTY_LICENSES.txt. The presence of case studies and a troubleshooting guide next to the source is a good sign for a tool that changes kernel settings on machines you cannot log into.

Building it and packaging it for RPM

There is no install command in the documentation to copy, because this is a system daemon that has to be built and packaged, and the packaging machinery is in the repository.

The Makefile is an RPM-oriented one. It derives the package name and archive name by querying the spec file with rpmspec, appending the output of uname -m to get the package name, and it uses a PKG_DIR under the home directory by default with rpmbuild layout directories for sources, build and buildroot. The default install prefix is /usr, and DESTDIR is respected.

The build targets are small and clear. The all target depends on srcdir, docdir and testdir, and the srcdir target simply runs make in the src directory, the docdir target runs make man in docs, and the testdir target runs make in test. The test target forces a rebuild and runs the test suite in the test directory. The pkg target assembles an RPM: it creates the source, build and licensed directories, removes any previous archive, exports the current git HEAD as a tar with a prefix, compresses it with bzip2, copies the licence files into the licensed directory, and finally invokes rpmbuild with the spec file.

That last detail is worth noting. The package is built from git archive of HEAD, so the tarball is exactly the committed tree, which is the behaviour you want for a system package that has to be reproducible and auditable.

The Makefile header carries an SPDX identifier of GPL-2.0 WITH Linux-syscall-note and a copyright line for 2023 Oracle and its affiliates. The repository also has a buildrpm directory, a debian directory, and a LICENSE.txt, and the project record lists the licence as a custom one that GitHub cannot classify, which is consistent with the Linux-syscall-note variant, an additional permission note used for kernel-facing code.

For the licence question specifically, the header states the GNU General Public License version 2 terms and points at the GPL in the source tree. The custom classification and the syscall note are worth confirming against LICENSE.txt and THIRD_PARTY_LICENSES.txt for your own compliance review rather than taking a summary from this page.

Where auto-tuning is the wrong answer

Three situations rule it out, and they are the mirror image of the design.

The first is a host whose configuration is meant to be declarative and reviewable. The whole premise of bpftune is that the correct value is not knowable in advance, so the tool changes it over time. A platform team that wants sysctls in a configuration management repository, diffed in a pull request and applied identically to every host, has a different requirement, and this tool does not meet it. The values it sets are, by design, not in your repository and not in a reviewable form.

The second is a machine where you cannot accept surprise. The tool will change kernel parameters while the system is running, on a timer, based on observed conditions. The mitigation is the log, and the documentation insists that describing the event in the log is key so that an administrator can understand what changed and why. That is a workable arrangement, but it is an arrangement where a change to system behaviour is explained after the fact rather than reviewed before.

The third is a kernel without the required BPF support. The project is built on BPF observability features, and a tuner is a shared object with a BPF component built using a BPF skeleton. The tools for the sysctl, the neighbour table and the IP fragmentation limits all depend on the kernel exposing the right hooks. The documentation names a 6.2 kernel for the sysctl count, and the repository has a TROUBLESHOOTING.md, which is where a kernel version mismatch will be diagnosed.

There is also a risk worth naming without exaggerating it. The sysctl tuner automatically disables itself when an administrator changes an associated tunable, and the strategies mechanism deliberately tries one tuning approach, measures it, and switches to whichever alternative evaluates highest. Both are good design for a system that has to keep running unattended, and both mean the tool's behaviour on your host is not fully predictable from configuration alone. That is acceptable when the alternative is nobody tuning the host at all, and it is not acceptable when the host is already well tuned and the operator needs to know it will stay that way.

Editorial conclusion

bpftune fits a fleet operator running Linux hosts that nobody logs into after provisioning, where a static sysctl file is a guess and the workload varies over time, and it fits someone who wants per socket or per device granularity rather than one system wide value. It is the wrong choice for a machine where you want the configuration to be declarative and reviewable, because the tool's whole premise is that the right values change and cannot be written down in advance, and it is the wrong choice for a kernel without the BPF features it relies on. Before deploying it, read the case studies and troubleshooting notes in the repository, confirm the kernel version on the target host supports the BPF features the tuners need, and decide in advance how you will audit the syslog output, because the tool is explicit that the log describing what changed and why is the audit trail.

Frequently asked questions

Official sources

1. Issues
2. oracle/bpftune on GitHub
3. README
4. Releases