# dial9, an always-on profiler for Tokio that records instead of interrupts

> dial9 is a Rust profiler that records large numbers of events cheaply and analyses them later, combining Tokio runtime hooks, CPU profiling and application data into a trace file you open in a static HTML viewer. It is built to run in production, which means the cost of enabling it is a design question rather than an afterthought.

**dial9-rs/dial9** — Always-on profiler for Rust designed for production

- Repository: https://github.com/dial9-rs/dial9
- Stars: 584 · Forks: 62
- Language: Rust
- License: Apache-2.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/dial9-rs-dial9

## A microscope, not a stop-the-world profiler

dial9 is described as an always-on profiler for Rust designed for production, and the framing in the project documentation is a microscope for Tokio and Rust applications in general. It lets you record a large number of events cheaply and analyze them later.

The reason that framing matters is the alternative. A sampling profiler works by interrupting a thread periodically and capturing a stack. It is cheap enough to leave on, but it gives you a statistical picture of where CPU time goes, and it says nothing about what the async runtime was doing. The question dial9 is built to answer is the one in the project description: what is Tokio actually doing, and it becomes readily apparent once you have the data.

Three sources feed the trace, and the documentation is explicit that you enable as many as you need to debug, or as few as you can tolerate the overhead of in production. Most applications will want Tokio events, CPU profiling information, and a handful of application events.

Two paths exist for getting value out of the result. The `dial9` crate provides an HTML static site that can view the trace files, and the viewer is also hosted as a site. And there is a second path through the agent toolkit: `dial9` ships skill documentation and scripts that allow agents to perform scripted analysis of dial9 traces.

That second path is the unusual one, and it is the most interesting thing about the project. A trace of a scheduler problem is a large structured artefact, and asking a model to reason over it is a different proposition from asking it to reason over a flame graph. The repository layout backs the claim up, with an `AGENTS.md`, a `CLAUDE.md` and a `CONTEXT.md` at the root.

The audience is a Rust engineer debugging concurrency or scheduler behaviour in a service they cannot easily reproduce under load, and the last push to the repository was on 2026-10-19.

## Tokio's unstable flag is a prerequisite, not a detail

The instrumentation has a hard prerequisite, and the documentation puts it plainly. dial9's Tokio instrumentation is built on runtime hooks that Tokio only exposes under `tokio_unstable`. With the flag set, dial9 sees every task on the runtime, which means poll spans, task spawn and terminate, and per-worker queue depth.

The flag goes in your Cargo build configuration, for example `.cargo/config.toml` or the `RUSTFLAGS` environment variable, and explicitly not in `Cargo.toml`.

```toml
# .cargo/config.toml
[build]
rustflags = [
  "--cfg", "tokio_unstable",
  # Required by the cpu-profiling and memory-profiling features:
  "-C", "force-frame-pointers=yes"
]
```

The second flag, `force-frame-pointers=yes`, is for the `cpu-profiling` and `memory-profiling` features, which capture stacks with a frame-pointer unwinder. Frame pointers cost a little performance and buy you reliable stack walks, which is the trade the library is making.

The fallback behaviour is the part worth understanding. Without the flag, the Tokio instrumentation has narrower task coverage: poll events cover only tasks started through `dial9::spawn`, `spawn_in`, `block_on` or `spawn_with`, and there are no task spawn or terminate events and no per-worker queue depth. Worker timelines and wake causality are unaffected.

So the cost of not setting the flag is specific. You lose the visibility into tasks your application spawned through a path dial9 does not know about, which is most of them in a real service, and you lose the queue depth signal, which is the number that tells you a worker is starved. You keep the worker timeline, which is the more commonly used view.

This is a genuine trade-off rather than an oversight. `tokio_unstable` is a name that exists precisely to signal that the API behind it can change without a semver bump, and a profiler that depends on it is depending on internals. That is acceptable for a diagnostic tool you turn on when investigating, and less acceptable as a permanent part of a release build.

## The recorder, the writer and the attach point

The configuration model is what tells you whether this is a library or a framework, and it is worth reading the shape of the API.

The example builds a `DiskBuffer` writer, then a recorder, then attaches a Tokio runtime to it. The writer is where the retention policy lives.

```rust
use std::io;
use dial9::{AttachedRuntime, Dial9HandleTokioExt, Dial9TokioHandle, DiskBuffer, TokioAttachOptions};

fn my_config() -> io::Result<AttachedRuntime> {
    let writer = DiskBuffer::builder()
        .base_path("/tmp/my_traces")
        .max_total_size(5 * 1024 * 1024)   // keep at most 5 MiB on disk
        .max_file_size(1024 * 1024)     // optional: defaults to min(100 MiB, max_total_size / 4)
        .rotation_period(std::time::Duration::from_secs(300)) // optional: rotate every 5 min (default: 60 s)
        .build();
```

Every knob is on the writer, and the defaults are the interesting part. The per-file segment size defaults to the smaller of 100 MiB and a quarter of the total budget, and the rotation period defaults to 60 seconds. The example raises the rotation to 300 seconds and the total to 5 MiB, which tells you the intended use is a small ring of recent segments rather than a large archive.

The error handling on the writer is the part with a lesson in it. The comment in the example says that the builder downgrades to a disabled recorder if the writer cannot be created, and that you should use `dial9::recorder(writer?)` instead to surface writer errors explicitly. So the default is a profiler that silently turns itself off rather than a service that fails to start because it cannot write a trace file, and there is a stricter variant for people who would rather know.

That is the right default for something you enable in production, and it is also a trap. A profiler that silently disables itself is a profiler that can appear to be working when it is not, and the way to tell is whether the trace directory has segments in it.

The recorder is built with `segment_metadata`, so you can tag each segment with key-value pairs such as the service name, and the runtime is attached through the recorder's handle with a `TokioAttachOptions` builder that sets the runtime name and turns on task tracking.

## Zero-code configuration through environment variables

For production use, the documentation points at `dial9::recorder_from_env`, which removes the need to write any configuration code at all.

```rust
use dial9::Dial9TokioHandle;

#[dial9::main(config = dial9::recorder_from_env)]
async fn main() {
    let handle = Dial9TokioHandle::current();
    handle.spawn(async { /* wake events tracked when enabled */ }).await.unwrap();
}
```

The `#[dial9::main(config = ...)]` attribute is the integration point, and it accepts either a function or a closure. There is a `recorder_from_env_with` variant for keeping the env-driven recording while configuring the runtime it builds, and every `config` returns a `std::io::Result<dial9::AttachedRuntime>`, which is the recorder plus a runtime attached to it.

The environment variables are the knobs, and the tables in the documentation are unusually clear about the defaults.

The local trace writer settings are `DIAL9_ENABLED` for the master switch, off by default, `DIAL9_TRACE_DIR` defaulting to `/tmp/dial9-traces`, `DIAL9_ROTATION_SECS` defaulting to 60 seconds measured monotonically from writer start, `DIAL9_MAX_DISK_USAGE_MB` defaulting to 1024 MiB as the total on-disk budget, and `DIAL9_MAX_FILE_SIZE_MB` defaulting to the smaller of 100 MiB and a quarter of the total.

Each process writes into its own `{boot_id}/` subdirectory, where the boot id is a four-character alpha string followed by the process id, which is a detail worth knowing when you are correlating a trace with a log line.

The runtime knobs are `DIAL9_TASK_TRACKING_ENABLED` and `DIAL9_TOKIO_INSTRUMENTATION_ENABLED`, both on by default, and `DIAL9_RUNTIME_NAME`, unset by default, which is the human-readable runtime name in the trace metadata.

There is also an S3 upload path, gated behind the `worker-s3` feature, with `DIAL9_S3_BUCKET` and other settings. The feature gate is the thing to notice, because uploading traces to object storage is a different risk decision from writing them to a local disk, and it is opt-in by feature rather than by configuration.

## A workspace, a viewer and a self-hosting story

The repository is a large Cargo workspace, and the member list in the manifest explains the shape of the project.

```toml
[workspace]
resolver = "3"
members = [
    "dial9-core",
    "dial9-tokio-telemetry",
    "dial9-trace-format",
    "dial9-trace-format-derive",
    "dial9-macro",
    "dial9-metrique",
    "dial9-viewer",
    "dial9",
    "dial9-utils",
    "dial9-destinations-s3",
    "perf-self-profile",
    "examples/metrics-service",
    "examples/simple-local",
    "examples/memory-local",
    "examples/rayon-fanout",
    "examples/telemetry-test-app",
    "tests-build",
]
```

There is a core, a separate Tokio telemetry crate, a trace format and its derive macro, a metrics crate called `dial9-metrique`, the viewer, the S3 destination, a self-profiling crate and seven example applications, each of which is a workspace member.

The examples are the clearest guide to the intended uses: `simple-local` is the minimal case, `memory-local` is memory profiling, `metrics-service` is a service that reports metrics, `rayon-fanout` is a workload that fans out across a Rayon thread pool, `cuda-profiling` is GPU profiling and it is excluded from the workspace, as is the fuzz target under `dial9-trace-format/fuzz` and the minimum-version test set.

The viewer is a static site. The `dial9-viewer` member is depended on with `default-features = false`, and the manifest comment explains why: a workspace member cannot override an inherited dependency's default features, so consumers opt back into S3 by enabling `dial9-viewer/s3`, as the `dial9` crate's `cli` feature does.

For local development there is a compose file. Its header explains that the `regenerate` service rebuilds the demo trace file `dial9-viewer/ui/public/demo-trace.bin` with no host-side Rust, AWS or Java setup, using `docker compose run --rm --build regenerate` and `docker compose down` to tear down the DynamoDB sidecar. The compose file also spins up `amazon/dynamodb-local` for the S3 destination tests, and the `regenerate` service is marked `privileged` because the metrics service enables `cpu-profiling` on Linux, which calls `perf_event_open`.

That last detail is the one to remember if you want CPU profiles in a container, because the usual Docker defaults will not give you perf events.

## What the always-on design costs you

A profiler that is meant to stay on in production has a different set of trade-offs from one you run for five minutes, and dial9 is honest about most of them.

The first cost is disk. The writer's defaults are a 1 GiB total budget with 60 second rotation, and each process writes into its own directory. On a machine running twenty instances that is a set of directories to monitor, and the trace budget is a configuration value you have to decide rather than a number the tool picks for you.

The second cost is the dependency on unstable APIs. The whole Tokio story rests on `tokio_unstable`, and the documentation is explicit that the flag is what buys you full task coverage. Building your production binary with that flag set means your binary depends on Tokio internals that are not covered by semver, and it means upgrading Tokio can break your build.

The third cost is the `tokio_unstable` requirement itself for the frame pointers. The `cpu-profiling` and `memory-profiling` features need `-C force-frame-pointers=yes`, which is a compiler flag on your release build, and it has a cost in generated code. There is also the container constraint: capturing CPU profiles in Docker needs `perf_event_open`, which the compose file handles with a privileged service.

The fourth is the possibility of a silent no-op. The writer failure path downgrades to a disabled recorder rather than failing the service, which is the right default for production and the wrong one if you are relying on the profiler and do not know it is off. The stricter `dial9::recorder(writer?)` variant exists precisely for that case.

What you get in exchange is the ability to answer a question that conventional profiling cannot. A wake-up storm, a task that blocks a worker, a queue depth that grows without anyone noticing, an interaction between your own application events and what the runtime was doing at that moment. That is a class of bug that reproduces in production and not in a benchmark, and having the trace from the moment it happened is worth the flags and the disk.

## Conclusion

dial9 suits a team that has hit a Tokio problem they cannot reproduce under a sampling profiler, a scheduler question, or a wake-up storm, and that is willing to run with the `tokio_unstable` flag set. It does not suit a service where the trace volume on disk is unacceptable, and it does not suit a team that wants a conventional flame graph without a new toolchain. Verify first that the tracing flags you need are set in your Cargo build configuration rather than in your manifest, that your disk budget for the trace directory matches your retention expectations, and that the viewer you use can open the trace file your build produces.

## FAQ

### What does dial9 do?

It is an always-on profiler for Rust and Tokio applications that records large numbers of events cheaply and analyses them later. It combines data from Tokio, the operating system and your application, so questions about what the runtime is actually doing can be answered from a recorded trace.

### Why do I need the tokio_unstable flag for dial9?

dial9's Tokio instrumentation is built on runtime hooks that Tokio only exposes under tokio_unstable. With the flag set you get every task on the runtime, including poll spans, task spawn and terminate events and per-worker queue depth; without it, poll events cover only tasks started through dial9::spawn, spawn_in, block_on or spawn_with.

### How do I configure dial9 without writing any code?

Use dial9::recorder_from_env as the config argument of the dial9::main attribute. The environment knobs include DIAL9_ENABLED, DIAL9_TRACE_DIR, DIAL9_ROTATION_SECS, DIAL9_MAX_DISK_USAGE_MB and DIAL9_MAX_FILE_SIZE_MB, with runtime knobs for task tracking, Tokio instrumentation and the runtime name.

### What happens if the trace writer cannot be created?

The documented default downgrades to a disabled recorder rather than failing, so the service keeps running. The documentation notes that you can use dial9::recorder(writer?) instead to surface writer errors explicitly.

### How do I view a dial9 trace?

The dial9 crate provides an HTML static site that can view the trace files, and the viewer is also hosted as a site. The repository additionally ships skill documentation and scripts that let an agent perform scripted analysis of dial9 traces.

## Sources

- [dial9-rs/dial9 on GitHub](https://github.com/dial9-rs/dial9)
- [Issues](https://github.com/dial9-rs/dial9/issues)
- [License: Apache-2.0](https://github.com/dial9-rs/dial9/blob/main/LICENSE)
- [README](https://github.com/dial9-rs/dial9/blob/main/README.md)
- [Releases](https://github.com/dial9-rs/dial9/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/dial9-rs-dial9