# fastapi-observability wires a FastAPI app into Tempo, Prometheus and Loki

> This repository is a working reference stack that sends a FastAPI application's traces to Tempo, its metrics to Prometheus and its logs to Loki, then joins the three in one Grafana dashboard. It is a demo, not a library: the value is in the configuration files, and the last push to the main branch was on 2026-04-12.

**blueswen/fastapi-observability** — Observe FastAPI app with three pillars of observability: Traces (Tempo), Metrics (Prometheus), Logs (Loki) on Grafana through OpenTelemetry.

- Repository: https://github.com/blueswen/fastapi-observability
- Stars: 1,122 · Forks: 143
- Language: Python
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/blueswen-fastapi-observability

## Three signals, one Grafana dashboard, joined by trace ID

The repository's premise is that a FastAPI service should be observable through all three pillars at once: traces with Tempo and the OpenTelemetry Python SDK, metrics with Prometheus and the Prometheus Python client, and logs with Loki. What makes the demo more than three separate setups is the correlation. Grafana links them through the trace ID and exemplars, and the README describes three navigation paths. From metrics to traces, you read a trace ID out of an exemplar attached to a histogram and query it in Tempo. From traces to logs, you take the trace ID and the service name tag from a span in the Tempo data source and query Loki. From logs to traces, you extract the trace ID from the log line using a regex defined in the Loki data source. Each direction is a different click, but all three rely on the same identifiers, which is the actual design work in this project.

## The Loki Docker driver is the first install step

Logging is wired in at the container level rather than in the application, and the first thing the quick start asks for is the Loki Docker driver as a Docker plugin, with a different image tag for each CPU architecture:

```bash
# For ARM64
docker plugin install grafana/loki-docker-driver:3.3.2-arm64 --alias loki --grant-all-permissions
# For AMD64
docker plugin install grafana/loki-docker-driver:3.3.2-amd64 --alias loki --grant-all-permissions
```

The driver is what gives every container a `loki` logging option, so the compose file can attach it to the FastAPI services. The plugin has to be installed before the stack starts, and the README calls out a specific failure mode: if Docker reports that the logging plugin was found but disabled, the fix is to enable it:

```bash
docker plugin enable loki
```

That detail is the kind of thing a reference stack is for. Note that the plugin version is pinned in the command while the application images in the compose file are not, so the pinning discipline is inconsistent across the project.

## One compose file starts Loki, Prometheus, Tempo and three apps

The compose file at the repository root is the centre of the project, and it is where the version choices are visible. Loki runs as `grafana/loki:3.5.0` on port 3100, Prometheus as `prom/prometheus:v3.3.1` on port 9090, and Tempo as `grafana/tempo:2.7.2` on the OTLP ports 4317 and 4318. Prometheus is started with a specific feature flag that the whole metrics-to-traces path depends on:

```yaml
command:
  - --config.file=/workspace/prometheus.yml
  - --enable-feature=exemplar-storage
```

Three identical application services are defined, `app-a`, `app-b` and `app-c`, all running the same image `ghcr.io/blueswen/fastapi-observability/app:latest` and distinguished only by an `APP_NAME` environment variable and the port they publish on, 8000, 8001 and 8002. The demo has a cross-service action on a `/chain` endpoint, so you can see a trace span that crosses application boundaries rather than a single lonely request.

## The log driver injects a multiline stage and a timestamp regex

Every service in the compose file shares one logging anchor, and the pipeline it defines is the piece that makes log lines parseable. It includes a multiline stage keyed on a timestamp pattern, with a three second maximum wait time, and a regex stage that separates the time from the message:

```yaml
loki-pipeline-stages: |
  - multiline:
      firstline: '^\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}'
      max_wait_time: 3s
  - regex:
      expression: '^(?P<time>\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2},\d{3}) (?P<message>(?s:.*))$$'
```

The firstline pattern is the traceable Python log prefix, so an exception whose message spans several lines is folded into one entry rather than split into fragments. The regex then names the time and the message, and it is this stage that the log-to-traces direction depends on, because the trace ID has to survive into the line Loki stores.

## LoggingInstrumentor puts the trace ID into the log line

On the application side, the OpenTelemetry logging instrumentation overrides the logger format so that every line carries the trace and span identifiers. The format string in the README is the whole mechanism:

```txt
%(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d] [trace_id=%(otelTraceID)s span_id=%(otelSpanID)s resource.service.name=%(otelServiceName)s] - %(message)s
```

The `otelTraceID` and `otelSpanID` fields are the bridge between the logging world and the tracing world. The helper that sets all of this up is a single function, and its `log_correlation` flag lets you leave the format alone if you do not want the change:

```python
def setting_otlp(app: ASGIApp, app_name: str, endpoint: str, log_correlation: bool = True) -> None:
    resource = Resource.create(attributes={
        "service.name": app_name
    })
    tracer = TracerProvider(resource=resource)
    trace.set_tracer_provider(tracer)
    tracer.add_span_processor(BatchSpanProcessor(
        OTLPSpanExporter(endpoint=endpoint)))
    if log_correlation:
        LoggingInstrumentor().instrument(set_logging_format=True)
    FastAPIInstrumentor.instrument_app(app, tracer_provider=tracer)
```

The service name is set as a resource attribute, and the README notes that this is what lets Tempo distinguish one application from another. Without it the three services in the demo are indistinguishable in the trace view.

## Internal ASGI spans make traces noisy

The README is candid about a side effect of instrumentation. Each request span contains additional child spans, because the instrumentation catches every internal ASGI interaction, and the documentation links the discussion in the opentelemetry-python-contrib issue tracker. If the internal spans are not wanted, the README points at a workaround in the same issue, built on a new OpenTelemetry middleware with two overridden methods for span processing. That is a real operational detail: default FastAPI instrumentation produces a trace tree with more depth than most people expect, and the fix is not a configuration flag but a custom middleware. There is a second trap in the same area. The FastAPI instrumentation only covers the ASGI app's own request, so if another service should join the same trace, the current span has to be injected into the outgoing header, which the README describes as a separate step.

## Generating load and reading the shipped dashboard

The quick start gives three ways to put traffic against the app. Two shell scripts in the repository root send requests with siege and curl, and there are also a Locust file and a k6 script:

```bash
bash request-script.sh
bash trace.sh
```

```bash
locust -f locustfile.py --headless --users 10 --spawn-rate 1 -H http://localhost:8000
```

```bash
k6 run --vus 1 --duration 300s k6-script.js
```

The k6 script is a single virtual user for five minutes, which is enough to populate the latency histograms and give the exemplars something to attach to. Once the data is in, the README points to a predefined dashboard named `FastAPI Observability` on Grafana at http://localhost:3000/, and it says to log in with `admin:admin`. The dashboard is also published on the Grafana Dashboards site under the number 16110. Those default credentials and the `latest` image tags are the two things you would change first if this stack were anything other than a local demo.

## Conclusion

Adopt this repository as a reference stack when you want a working FastAPI to Grafana wiring to copy, especially the Loki Docker driver pipeline that carries trace IDs into the log line and the Prometheus exemplar storage that links a histogram to a trace. Do not adopt it as a production deployment: the compose file ships Grafana with default credentials, the images are tagged latest, and the project has no GitHub releases to pin against. Verify first that the Docker plugin version and the Prometheus `--enable-feature=exemplar-storage` flag match the versions in the compose file, and that your own log format carries the trace ID fields the Loki regex expects.

## FAQ

### How do I start the fastapi-observability stack?

Install the Loki Docker driver as a Docker plugin first, then run `docker-compose up -d` from the repository root. The compose file starts Loki on 3100, Prometheus on 9090, Tempo on 4317 and 4318, three FastAPI services and Grafana on 3000.

### Why do I get 'plugin loki found but disabled' when starting the stack?

The Loki Docker driver plugin was installed but not enabled. The README gives the fix as running `docker plugin enable loki`.

### How does the fastapi-observability demo link logs to traces?

The OpenTelemetry logging instrumentation overrides the logger format so every line includes trace_id and span_id fields, and Loki stores the lines. A regex defined in the Loki data source extracts the trace ID so a log line can lead back to the trace in Tempo.

### Why do my FastAPI traces have extra spans?

The OpenTelemetry FastAPI instrumentation catches each internal ASGI interaction, so a request span contains additional child spans. The README links an issue in opentelemetry-python-contrib with a workaround based on a middleware that overrides two span processing methods.

### What licence does fastapi-observability use?

The repository is published under the MIT licence, with the licence file in the root of the repository alongside the compose file and the application source.

## Sources

- [blueswen/fastapi-observability on GitHub](https://github.com/blueswen/fastapi-observability)
- [Issues](https://github.com/blueswen/fastapi-observability/issues)
- [License: MIT](https://github.com/blueswen/fastapi-observability/blob/main/LICENSE)
- [README](https://github.com/blueswen/fastapi-observability/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/blueswen-fastapi-observability