dd-trace-go: tracing, profiling and runtime security for Go services

Three products in one module

This repository contains Go packages for the client-side components of the Datadog product suite for Application Performance Monitoring, Continuous Profiling and Application Security Monitoring of Go applications. The APM side traces requests as they flow across web servers, databases and microservices, and the tracer package allows you to trace any piece of your Go code, with commonly used Go libraries automatically traced through out-of-the-box integrations in the contrib package. The profiler periodically collects and sends Go profiles to the Datadog API, broken down by function name and line number, to reduce end-user latency and infrastructure costs. Application Security Monitoring is the third piece, and its placement is the notable design decision: ASM is not a standalone Go package but is transparently integrated into the APM tracer, so it is switched on with an environment variable and inherits the trace context the tracer already has.

What application security monitoring adds

The security piece provides in-app monitoring and protection against application-level attacks that aim to exploit code-level vulnerabilities, such as Server-Side-Request-Forgery, SQL injection, or Reflected Cross-Site-Scripting. ASM identifies services exposed to application attacks and uses in-app security rules to detect and protect against threats in your application environment. The integration is what makes it cheap to adopt: because the security rules run inside the same tracer that already knows which service, which method and which route a request belongs to, the rules have context that an out-of-band web application firewall does not. The cost of that integration is that the library depends on the Datadog agent and the agent-side security component, so the feature is not usable against a plain collector.

Installing the tracer and the profiler

The module contains many packages, but most users should install the two below.

go get github.com/DataDog/dd-trace-go/v2/ddtrace/tracer
go get github.com/DataDog/dd-trace-go/v2/profiler

The contrib packages are published as nested modules and are installed separately, one for the library you want instrumented.

go get github.com/DataDog/dd-trace-go/contrib/gorilla/mux/v2

The named examples are net/http, gorilla/mux and database/sql, and each one is its own module path, which is why a contrib dependency can be versioned independently of the tracer. The README also warns about the other failure mode: if you installed more packages than you intended, you can use go mod tidy to remove any unused packages. That advice matters more here than in most libraries, because the contrib tree is large and it is easy to pull in several by accident while following different examples.

Contrib modules are separate modules with separate versions

The directory listing at the repository root shows how the module is divided. There is a v2 directory, which is the Go module major version 2 of the tracer, and inside the module path is /v2, so the import path carries the major version. Alongside it are ddtrace for the tracer, profiler, appsec, contrib, orchestrion, otelc, civisibility, datastreams, llmobs, openfeature, internal, instrumentation and several more, each a package or a sub-module of the same repository. The nested module structure is what lets the library support many third-party packages without forcing every user to take a dependency on all of them, and it is the reason the module file lists a long tail of dependencies that most applications never touch. The go.mod also requires Go 1.26.0 and sets a godebug directive for x509 negative serial numbers, which is the kind of detail that comes from a library supporting a very wide range of deployment environments.

The Go support policy is two releases wide

Datadog APM for Go is built upon dependencies defined in specific versions of the host operating system, Go releases, and the Datadog Agent and API, and dd-trace-go supports the two latest releases of Go, matching the official Go policy. The library also only officially supports first class ports of Go. That is a narrow support statement, and it is worth reading against the fact that the repository runs a GitHub Actions workflow for main branch and release tests, a separate system tests workflow, CodeQL analysis, and a parametric tests workflow for APM. The parametric tests are the interesting one: they run the same tracing scenarios against a matrix of configurations, which is what keeps the contrib packages behaving the same way across the library versions they instrument.

A Makefile that lints more than Go

The build targets are listed in the README and they say more about the project than a normal library's would. Beyond the obvious all, generate, lint, format and test, there are lint targets per language: lint/go and lint/go/fix, lint/shell for shell scripts, lint/misc which covers copyright headers and Makefiles, and lint/action for GitHub Actions workflows. There is also a specific target, lint/errlog, described as running SDK logging safety analyzers for constant messages, SafeError and LogValuer telemetry scrubbing, and unsafe format verbs, which is a check on the telemetry payloads themselves rather than on the code that produces them. The test targets are similarly layered: test runs core, integration and contrib tests, test/unit and test/contrib narrow them, test/appsec runs the suite with application security enabled, and test-deadlock and test-debug-deadlock add deadlock detection with and without the debug build.

Databases in the compose file, and what they are for

The docker-compose file is the test infrastructure and it lists the databases the integration tests run against: Aerospike, Cassandra, MySQL, Postgres and SQL Server, each pinned to an exact image digest rather than a tag, with health checks, retry counts and start periods tuned for each because they have very different startup times, Cassandra being given ninety seconds and MySQL thirty. The MySQL and Postgres services carry test credentials, and the Cassandra one is given a cluster name and datacenter in the same naming scheme the test suite expects. Pinning by digest is the detail that matters for a library whose tests run in CI: a floating tag would let an upstream image change under the tests, and the digest makes the run reproducible.

Where dd-trace-go is the wrong choice

Four limits apply. It is bound to Datadog: even with the OpenTelemetry dependencies and the OTLP exporters in the module file, the tracer, the profiler and the security rules all assume the Datadog agent and API, so a team standardising on a different collector is adding a dependency it will not fully use. The support policy is narrow, since only the two latest Go releases and first class ports are officially supported, and an application stuck on an older Go has to pin an older tracer. The dependency surface is wide, with the contrib tree and the agent-side packages in the same module, so a service that wants tracing and nothing else still pulls a large graph. And the profiler and the security layer are not free: the profiler ships CPU and memory profiles of production traffic, and the application security rules make live decisions about requests in your application, which is an operational commitment rather than a library flag.

Editorial conclusion

Use dd-trace-go when your Go service runs on Datadog, because the tracer, the profiler and the application security rules are wired together and the security layer is enabled with a single environment variable rather than a separate integration. Do not adopt it to get generic OpenTelemetry tracing, because the library is bound to the Datadog agent and API, even though it also depends on the OpenTelemetry SDK and exports OTLP. Verify first which Go versions you are on, since the project supports only the two latest Go releases and first class ports, that you install only the packages you need and run go mod tidy afterwards, and how the contrib modules nest, because they are published as separate modules and a mismatched version between the tracer and a contrib package is a dependency problem rather than a configuration one.

Frequently asked questions

Official sources

1. DataDog/dd-trace-go on GitHub
2. Issues
3. Project website
4. README
5. Releases