Uber JVM Profiler: A Java Agent for Spark Fleet Metrics and Method Tracing

The problem: metrics from a hundred JVMs that nobody can line up

A single Spark application can span dozens or hundreds of processes on different machines. Each process has its own heap, its own garbage collection pauses, its own thread counts. Standard JVM tooling observes one process at a time, so the operator ends up with a hundred disconnected views and no way to say which executor was reading the hot HDFS file or which one spent its time in a name node call. The README states the profiler was created for exactly this case: Spark applications with many processes per application, so that metrics from those processes can be correlated. The tag parameter is the correlation key. Whatever string you pass as tag is reported alongside every metric, which means a downstream consumer can group by application rather than by host. The second problem is code you do not own. Duration profiling and argument profiling let you observe how often a method runs, how long it takes, and what value one of its arguments held, without modifying the target class. The README describes tracing HDFS name node call latency per Spark application and tracing which HDFS paths each application reads or writes.

The mechanism: a Java agent, a reporter, and a config provider

The project ships as a Java agent jar. You attach it with the standard -javaagent flag and pass parameters after an equals sign, comma separated. Three extension points are named in the parameter list. The reporter decides where metrics go: ConsoleOutputReporter, FileOutputReporter and KafkaOutputReporter are bundled by default, and the README says you can implement your own and name the class in the reporter parameter. The config provider decides where settings come from: YamlConfigProvider is the built-in option, and configFile points it at a local path or an HTTP URL. The third piece is the sampling and instrumentation configuration, which is where the profiling actually happens. metricInterval sets how often metrics are collected and reported, in milliseconds. sampleInterval sets how often stacktrace sampling runs, and the README is explicit that if it is unset or zero, stacktrace sampling does not happen at all. durationProfiling takes a fully qualified class and method, and it accepts a wildcard for the method name, for example com.uber.profiling.examples.HelloWorldApplication.*. argumentProfiling takes the same class and method plus a numeric suffix, so HelloWorldApplication.publicSleepMethod.1 means the first argument's value is captured and sent to the reporter. The data flow is therefore: agent hooks the target methods and the sampling timer, collects JVM and OS level values on the metric interval, hands everything to the reporter, and the reporter serialises to console, file or Kafka. Nothing in the README describes a server side component, so correlation happens in whatever consumes the output.

Getting it running: build profiles, agent flags, and the Kafka topic layout

Building requires JDK 8 or later and Maven. The README gives mvn clean package, which produces jvm-profiler.jar with ConsoleOutputReporter, FileOutputReporter and KafkaOutputReporter inside. Custom reporters such as RedisOutputReporter and InfluxDBOutputReporter are not in the default jar; you select them with a Maven profile, for example mvn -P redis clean package, and the README points at pom.xml for the full list of profiles and their ids. That detail matters more than it looks: the reporter you want may require a different build than the one your CI produces by default. For Spark, the README shows uploading the jar to HDFS and adding two configuration lines, spark.jars pointing at the HDFS path and spark.executor.extraJavaOptions containing -javaagent:jvm-profiler-1.0.0.jar. For a plain Java application the example is a single java command with -javaagent:target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,tag=mytag,metricInterval=5000,durationProfiling=...,argumentProfiling=...,sampleInterval=100 followed by -cp and the main class. For Tomcat the agent goes into CATALINA_OPTS and output lands in logs/catalina.out. For Spring Boot 2.x the README uses mvn spring-boot:run -Dspring-boot.run.jvmArguments="-javaagent:...", noting that Spring Boot 1.x needs -Drun.arguments instead. The Kafka example adds brokerList=localhost:9092 and topicPrefix=profiler_, and the README states metrics go to a topic named profiler_CpuAndMemory. So the topic name is the prefix concatenated with a metric category, which means your topic count grows with the categories you enable.

What it collects, and what the parameter list does not cover

The feature list covers heap, non-heap and native memory including VmRSS and VmHWM, memory pools and buffer pools for direct and mapped buffers; CPU usage and garbage collection time; duration profiling and argument profiling; stacktrace profiling to generate flamegraphs; IO metrics covering disk read and write bytes for the application plus CPU iowait for the machine; and thread metrics including total, peak, live, active and newly created threads. That is a broad surface for a single agent, and the breadth is the point: one attachment gives you the whole picture for a process. The gaps are in the parameter list rather than the feature list. The README truncates at ioProfiling, so the full set of IO related keys and their accepted values cannot be confirmed from the supplied material. There is no description of how the Kafka reporter handles serialisation format, partitioning, or delivery failure, and no statement about what happens to buffered metrics when the broker is unreachable. The reporter is described as pluggable, which is the honest answer for production: if the bundled behaviour is not what you need, the intended path is to implement your own reporter class and name it in the reporter parameter. Treat the bundled Kafka reporter as a starting point rather than a guaranteed delivery channel.

Where it is the wrong tool: sampling cost, attachment scope, and licence ambiguity

Three limitations are worth stating plainly. First, stacktrace sampling is optional and off when sampleInterval is zero or absent, and when it is on, the README gives 100 milliseconds in one example and 5000 in another. That is a wide range, and the cost of sampling every 100 milliseconds across hundreds of executors is not quantified anywhere in the material. If you enable it broadly, you are choosing to pay an unmeasured price. Second, the agent attaches at JVM start. The Spark example puts the agent in spark.executor.extraJavaOptions, which applies to executors launched after the configuration is set. There is no described mechanism for attaching to already running processes, so retrofitting the profiler onto a long-lived cluster means a restart or a rolling replacement of the affected processes. Third, the licence field is NOASSERTION. That is not a licence, it is the absence of a machine readable one, and the repository metadata does not resolve it. For an internal tool this may not matter. For anything redistributed or bundled into a product, it is the first thing to check in the repository itself, because no release artefacts were retrieved and therefore no licence file attached to a release could be inspected either. I am not giving legal advice; the point is that the metadata does not answer the question and the repository does.

Compared with async-profiler: sampling a single JVM versus reporting a fleet

async-profiler is the natural alternative for JVM profiling, and the difference in approach is structural rather than a matter of features. async-profiler attaches to one JVM, typically on demand, and produces a profile of that process: a flamegraph of CPU time, allocation profiles, lock contention. Its unit of work is the process and the session. Uber JVM Profiler's unit of work is the fleet. It is designed to be configured once through spark.executor.extraJavaOptions or CATALINA_OPTS and then run continuously across every process that starts with that configuration, emitting metrics on a timer to a reporter that a downstream system consumes. If your question is why this one request is slow, and you can attach to the process, a session-oriented profiler answers it faster and with more detail. If your question is which of four hundred executors are spending time in the name node, and you want that answer every five seconds without logging into anything, the fleet-oriented agent is the shape you need. The two are not substitutes, and the README's feature list reads as an argument for the fleet case: memory across all executors, IO per application, thread counts per process, all reported together with a tag.

Maintenance cost and what the repository does not tell you

There are no retrieved releases, so there is no changelog to read and no versioned artefact to pin against a known set of fixes. The last push timestamp is recent, but a push is not a release and does not tell you what changed. For an agent that gets attached to production JVMs, that is a real operational gap: upgrading means rebuilding from a commit and accepting whatever has changed since the last build you made. The build itself is Maven with profiles, which means the jar you produce depends on the profile flags you pass, so two teams building from the same commit can end up with different reporters inside the jar. That is a reproducibility problem worth solving with a pinned build command rather than a default mvn clean package. The licence question compounds this. If the repository does not carry an explicit licence, the safe assumption for a redistributed artefact is that you do not have one, and the only way to settle it is to read the repository directly. There is also no described compatibility policy for JDK versions beyond the JDK 8+ build requirement, and no statement about which Spark versions the executor configuration was validated against.

Editorial conclusion

Adopt it if you run Spark or Hadoop jobs across many JVMs and need per-application CPU, memory, IO, thread and stacktrace data correlated by a shared tag, or if you need method duration and argument values from code you cannot edit. Do not adopt it if you want a single-process interactive profiler, allocation-level flamegraphs, or a maintained dependency with clear release notes, since no releases were retrieved and the licence is NOASSERTION. Verify three things before committing: the actual licence text in the repository, whether the Kafka reporter's topicPrefix layout matches your ingestion pipeline, and whether the durationProfiling and argumentProfiling targets survive your build's class and method naming.

Official sources

1. Issues
2. README
3. uber-common/jvm-profiler on GitHub