uber-common/jvm-profiler: a Java agent that profiles Spark executors and arbitrary methods

What uber-common/jvm-profiler solves for distributed JVM workloads

A single Spark application can run as dozens or hundreds of JVM processes spread across machines. Standard profilers assume you attach to one process and watch it. Uber's profiler takes the opposite approach: it is a Java Agent that runs inside every process and pushes metrics outward, so you can correlate the same metric across all executors of one application. The README states this was the original motivation, profiling Spark applications where per-machine data is useless without a common tag. The agent is generic, so the same jar works for a plain Java application, an executable jar, Tomcat and Spring Boot. The audience is therefore platform and data engineers who own the cluster, not an individual developer chasing a slow method on a laptop.

How the agent collects and ships metrics

The mechanism is the standard JVM instrumentation hook. You pass -javaagent:jvm-profiler-1.0.0.jar=param1=value1,param2=value2, and the agent reads its parameters from that single string. A reporter class decides where data goes: ConsoleOutputReporter, FileOutputReporter and KafkaOutputReporter are bundled by default, while RedisOutputReporter and InfluxDBOutputReporter must be selected at build time through a maven profile. A configProvider can replace the inline parameters; YamlConfigProvider reads a configFile that may be a local path or an HTTP URL, which matters when you want one configuration shared by many executors. metricInterval controls how often metrics are collected and reported, and sampleInterval controls stacktrace sampling. If sampleInterval is unset or zero, the README says the profiler does not do stacktrace sampling at all. Duration profiling and argument profiling are separate: durationProfiling takes a class and method (with wildcard support for the method name), while argumentProfiling appends an index such as .1 to capture the first argument's value. That argument capture is the part most teams underestimate, because it sends user data, not just counters, to the reporter.

Installing jvm-profiler and a first run against a sample application

The README requires JDK 8 or newer and maven. Building produces jvm-profiler.jar with the default reporters inside it. The command is a single maven invocation, and the custom reporter profiles are chosen at this step, not at runtime.

mvn clean package

To bundle a custom reporter such as the Redis one, pass its profile id. The README points at pom.xml for the full list of profiles.

mvn -P redis clean package

The quickest first use is the bundled example application, which reports to standard output. Note the reporter, tag, metricInterval, durationProfiling, argumentProfiling and sampleInterval parameters in one string.

java -javaagent:target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,tag=mytag,metricInterval=5000,durationProfiling=com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod,argumentProfiling=com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod.1,sampleInterval=100 -cp target/jvm-profiler-1.0.0.jar com.uber.profiling.examples.HelloWorldApplication

You should see metric lines on the console every five seconds. For a Spring Boot application the README uses the same agent string through the maven plugin; Spring Boot 1.x needs -Drun.arguments instead of -Dspring-boot.run.jvmArguments.

mvn spring-boot:run -Dspring-boot.run.jvmArguments="-javaagent:/opt/jvm-profiler/target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,metricInterval=5000,durationProfiling=foo.bar.FooController.barMethod,sampleInterval=5000"

Sending metrics to Kafka and what arrives on the topic

Kafka is the reporter that makes the multi-process story work. You supply brokerList and topicPrefix, and the README says metrics go to a topic named by combining the prefix with the metric group, giving profiler_CpuAndMemory for the example. The agent string is otherwise the same as the console case.

java -javaagent:target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.KafkaOutputReporter,metricInterval=5000,brokerList=localhost:9092,topicPrefix=profiler_ -cp target/jvm-profiler-1.0.0.jar com.uber.profiling.examples.HelloWorldApplication

The tag parameter is what lets you group executors belonging to one application, since the same jar is running in every process. The README does not document topic creation, partition counts or serialization format beyond pointing at the metrics example at the bottom of the document, so verify those against your broker before relying on the stream in production.

Where jvm-profiler is the wrong tool

This is an agent for headless, long-running, multi-process JVMs. It has no user interface, no flamegraph viewer and no interactive drill-down; stacktrace profiling produces data that you still have to turn into a flamegraph yourself, which is why the repository ships a stackcollapse.py script at the top level. If you need to click through call trees in a running IDE session, this project will not do it. Two further constraints are visible. First, argument profiling captures method arguments and sends them to the reporter, so profiling a method that handles credentials, tokens or personal data will export that data to Kafka, a file or the console; the README describes the feature but does not discuss redaction. Second, the licence is not a recognised SPDX identifier: the repository marks it NOASSERTION, and the file list includes both LICENSE and epl-v10.txt, so the exact terms need reading before redistribution. There are no recent releases retrieved, so the jar version in every example stays 1.0.0 and you are building from source.

jvm-profiler compared with desktop profilers such as VisualVM or IntelliJ

The difference is architectural, not a matter of feature checklists. A desktop tool like Java VisualVM or the IntelliJ profiler attaches to one JVM, usually on your machine or over a debug port, and gives you an interactive view of that process. jvm-profiler runs unattended inside every process and emits time series. That makes it weaker for a single-process investigation and stronger for questions like which executor is spending time in a specific method, or how much disk read each Spark application performs. The practical consequence: with jvm-profiler you must already know which metric and which class or method you care about, because you specify them in the agent string or the YAML config. You pay for that with configuration discipline and a downstream consumer for the emitted data.

Maintenance status, build cost and licence questions

The repository is not archived, and the last push was on 2026-05-21, so it is not abandoned. It is also not a project with frequent tagged releases; no recent releases were retrieved, and the README still refers to version 1.0.0 throughout. Upgrading therefore means rebuilding from master with mvn clean package rather than pulling a newer artifact, and if you use a custom reporter you must remember its maven profile each time. The licence situation deserves attention before you embed the agent in a product: the repository reports NOASSERTION for the licence, and the top-level files include LICENSE, NOTICE and epl-v10.txt. That combination suggests a permissive licence with a notice obligation, but the only reliable answer comes from reading those files. This is not legal advice; it is a note that the metadata alone does not settle the question.

Editorial conclusion

Adopt jvm-profiler if you run Spark or other multi-process JVM workloads and need per-executor CPU, memory, IO and method metrics in a central topic or file. Do not adopt it if you want an interactive UI for analysing a single process; the project ships reporters, not a viewer, and the README documents no rollback or uninstall path beyond removing the -javaagent flag. Before rolling it out, verify that your JDK is 8 or newer, that the reporter class you plan to use is bundled in the jar you built (custom reporters such as RedisOutputReporter need a maven profile), and that the Kafka brokerList and topicPrefix you pass are reachable from every executor.

Frequently asked questions

Official sources

1. Issues
2. README
3. uber-common/jvm-profiler on GitHub