# Dataproc initialization actions are scripts you run before the cluster is yours

> The GoogleCloudDataproc initialization-actions repository is a catalogue of shell scripts that run on every node of a Dataproc cluster right after it is created, installing things like Kafka, Flink, Alluxio, conda and Presto. The documentation is unusually clear that the samples come without support and that production users should copy them into their own bucket first.

**GoogleCloudDataproc/initialization-actions** — Run in all nodes of your cluster before the cluster starts - lets you customize your cluster

- Repository: https://github.com/GoogleCloudDataproc/initialization-actions
- Website: https://cloud.google.com/dataproc/init-actions
- Stars: 597 · Forks: 521
- Language: Shell
- License: Apache-2.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/googleclouddataproc-initialization-actions

## Scripts that run on every node the moment the cluster exists

When you create a Dataproc cluster you can specify initialization actions, in executables and scripts, that Dataproc runs on all nodes immediately after the cluster is set up. The documentation explains the intent: initialization actions often set up job dependencies, such as installing Python packages, so that jobs can be submitted without installing dependencies at job time.

That is the whole mechanism. There is no package manager, no image build and no declarative dependency file. You are handing the platform a list of shell scripts and it runs them on every node, and anything you wanted installed has to be installed by the script.

The practical consequence is that the script runs everywhere, on the master and on every worker, whether or not that node needs the component. Any action that only makes sense on one role has to detect its own role and branch, which the repository addresses directly through cluster metadata.

The repository is a catalogue rather than a framework. The top-level entries show the shape of it: directories for alluxio, drill, flink, kafka, presto, conda, rstudio, hue, hbase, oozie, livy, gobblin, hive-hcatalog, solr, dask, horovod, rapids, spark-rapids, gangli, stackdriver, prometheus, otel, cloud-sql-proxy, connectors, beam, bigdl, h2o, mlvm, dr-elephant, ranger, tony, toree, and more, alongside a requirements.txt with the Python packages the scripts rely on.

The audience is a Dataproc operator who knows exactly which extra piece of the Hadoop ecosystem their pipeline needs.

## The scripts must live in Cloud Storage, which shapes the whole workflow

There is one hard constraint that shapes every other decision: initialization actions must be stored in a Cloud Storage bucket and can be passed as a parameter to the gcloud command or the clusters.create API.

The generic form of the cluster creation command takes the bucket URIs and a timeout:

```bash
gcloud dataproc clusters create <CLUSTER_NAME> \
    [--initialization-actions [GCS_URI,...]] \
    [--initialization-action-timeout TIMEOUT]
```

During development you can point at the Dataproc-provided regional buckets, for example goog-dataproc-initialization-actions-us-east1, and the documentation gives the Presto action as the example:

```bash
REGION=<region>
CLUSTER=<cluster_name>
gcloud dataproc clusters create ${CLUSTER} \
    --region ${REGION} \
    --initialization-actions gs://goog-dataproc-initialization-actions-${REGION}/presto/presto.sh
```

The production recommendation is the part to internalise. The documentation says that before creating clusters it is strongly recommended that you copy initialization actions to your own Cloud Storage bucket, to guarantee consistent use of the same initialization action code across all cluster nodes and to prevent unintended upgrades from upstream in the cluster:

```bash
BUCKET=<your_init_actions_bucket>
CLUSTER=<cluster_name>
gsutil cp presto/presto.sh gs://${BUCKET}/
gcloud dataproc clusters create ${CLUSTER} --initialization-actions gs://${BUCKET}/presto.sh
```

The stated reason is consistency. If the bucket URI points at Google's copy, a change upstream silently changes what runs on your next cluster, and a cluster that worked yesterday can fail today with no diff in your own repository.

You also decide when to sync your copy with changes in the GitHub repository, which the documentation notes is also useful if you want to modify an action to meet your needs. Copy first, then treat your copy as the source of truth.

## Detecting the node role from instance metadata

Because every action runs on every node, the actions that behave differently per role have to ask the instance what it is. Dataproc sets special metadata values for the instances in the cluster, and the documented pattern is to read the dataproc-role attribute:

```bash
ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
  ... master specific actions ...
else
  ... worker specific actions ...
fi
```

The helper is a script on the instance at /usr/share/google/get_metadata_value, so the pattern is portable across every node without extra installation. That is the mechanism to use if you write your own action, and it is also how the shipped actions stay sane on a mixed cluster.

You can also feed your own metadata in at cluster creation, which is how a parameterised action gets its configuration without editing the script:

```bash
gcloud dataproc clusters create cluster-name \
    --initialization-actions ... \
    --metadata name1=value1,name2=value2,... \
    ... other flags ...
```

This is the cleaner way to make an action configurable. A script that reads its settings from metadata can be reused across clusters, while a script with values edited into it becomes a file you have to fork and keep in sync with everything else you have forked.

## What you can install, from Kafka and Flink to conda and RStudio

The catalogue is grouped by what you are trying to achieve, which is a useful way to think about it.

The first group is the Hadoop ecosystem: Alluxio, Apache Drill, Apache Flink, Apache Gobblin, Apache Hive HCatalog, Apache Kafka, Apache Livy, Apache Oozie, Apache ZooKeeper and Presto. These are components that exist as separate Apache or vendor projects and would otherwise need a build or a tarball dropped onto every node.

The second group is for data science and interactive work: Miniconda, Apache Zeppelin, RStudio Server, Intel BigDL and Hue. The documentation also notes you can switch to Python 3 using the conda initialization action, and offers a separate action to configure a nicer shell environment.

The third group connects to Google Cloud services. It installs alternate versions of the Cloud Storage and BigQuery connectors, with specific versions of these connectors coming pre-installed on Cloud Dataproc clusters. There is also an action to share a Cloud SQL Hive Metastore, or simply to read and write data from Cloud SQL.

The fourth group is monitoring: Stackdriver and Ganglia. The Ganglia entry is the older of the two approaches, while Stackdriver is the Google Cloud monitoring service, and both have a corresponding directory in the repository.

The repository also contains directories the README does not list among the actions, including atlas, beam, bigtable, bigdl, datasketches, disk, docker, gpu, h2o, hbase, hive-lineage, hive-llap, knox, logging, mlvm, opsagent, post-init, python, ranger, solr, spark-rapids, sqoop, starburst-presto, tony, toree and user-environment. The directory listing is broader than the documented list, which is a good sign of what people actually run.

## Actions that were removed because Dataproc now ships them

The removed actions section is the most useful part of the documentation for deciding whether to use this repository at all, because it maps exactly the overlap between the catalogue and the platform itself.

Apache Tez was removed because it is now pre-installed in all current Dataproc image versions. Datalab was removed because it has been replaced by Vertex AI Workbench, which integrates with Dataproc. Jupyter was removed because it has been replaced with the Jupyter Optional Component.

The pattern is that the catalogue shrinks whenever the platform absorbs a component. That is worth internalising before you reach for an init action: the list of things you need to install manually is getting shorter over time, and the correct first step is to check the Dataproc image version components rather than to look for a script.

The same logic applies to the connector actions. The documentation states that specific versions of the Cloud Storage and BigQuery connectors come pre-installed on Cloud Dataproc clusters, so the init action exists to install an alternate version. If the pre-installed version meets your requirement, the action is unnecessary complexity.

The connector versions are tied to Dataproc image versions, which the documentation links to a specific versions page. A script pinned to one image version can behave differently on another, another reason to copy and version your own copy rather than tracking the upstream bucket.

## Single node clusters break four of the actions

Single Node clusters have dataproc-role set to Master and dataproc-worker-count set to 0, which means the script runs exactly once, on the master. Most actions work out of the box for that reason, and the documentation gives two examples: notebooks such as Apache Zeppelin, and libraries such as Apache Tez, plus actions that run on all nodes of a cluster such as cloud-sql-proxy, which also work.

But some initialization actions are known not to work on Single Node clusters, and all of them expect to have daemons on multiple nodes. The list is short and specific: Apache Drill, Apache Flink, Apache Kafka and Apache ZooKeeper.

That is a coherent limitation rather than a scattered one. All four are distributed systems whose components talk to each other, and with a single node there is no second node to talk to. The daemon starts and then sits there doing nothing useful, or fails, and either way you do not get the service you asked for.

If you are experimenting on a single node cluster to keep costs down, this is the list to check before you build a plan around Kafka or Flink. The documentation invites pull requests or issues if you have a good use case for running one of these on a Single Node cluster.

## Apache 2.0, no releases, and a repository kept current on main

The licence is Apache 2.0, which covers both the sample scripts and any modification you make to them. That matters more here than in most repositories, because the scripts are things you copy into your own bucket and then edit, and the permission grant travels with the copy.

The project publishes no GitHub releases, so there is no version to pin. What you get is the main branch, and the last push was on 2026-10-12. The practical way to get a stable copy is the one the documentation already recommends: copy the script into your own bucket with gsutil, and treat that copy as frozen until you choose to pull a new version.

The repository also carries the usual scaffolding: a LICENSE file, CONTRIBUTING.md, a BUILD and a WORKSPACE file at the root for Bazel users, a .gitignore, and a requirements.txt listing the Python packages the scripts depend on, which are bootstrapping 0.1.2, click 8.1.7, packaging, and setuptools pinned below 70.0.0. There is also a push-to-gcs.sh script at the root, which is the mechanism for getting a script into a bucket, and an integration_tests directory for the actions themselves.

The documentation is explicit about support. The samples are provided to show how various packages and components can be installed on Dataproc clusters, you should understand how they work before running them on your clusters, and they are provided without support and you use them at your own risk. That is a reasonable position for scripts that install arbitrary software onto every node of a cluster, and it is also why the copy-to-your-own-bucket advice matters.

## Conclusion

The initialization-actions repository suits a Dataproc user who needs a component that is not pre-installed on the image, such as Kafka, Flink or Alluxio, and who can maintain a copy of the script in their own bucket. It is the wrong choice for components Dataproc already ships, since Tez, Datalab and Jupyter have all been removed from this catalogue because the platform provides them directly. Before running anything in production, copy the script into your own Cloud Storage bucket as the documentation recommends, pin the version you copied, and check the single node cluster list, because Apache Drill, Flink, Kafka and ZooKeeper are documented as not working there.

## FAQ

### How do I pass an initialization action to a new Dataproc cluster?

Initialization actions must be stored in a Cloud Storage bucket and passed to the gcloud dataproc clusters create command with the --initialization-actions flag, for example pointing at gs://goog-dataproc-initialization-actions-${REGION}/presto/presto.sh during development. The --initialization-action-timeout option sets a limit on how long the action may run.

### Why should I copy Dataproc initialization actions into my own bucket?

The documentation strongly recommends it for production use, to guarantee consistent use of the same initialization action code across all cluster nodes and to prevent unintended upgrades from upstream. You also decide when to sync your copy with changes in the GitHub repository, which is useful if you want to modify an action.

### Which initialization actions do not work on Dataproc Single Node clusters?

Apache Drill, Apache Flink, Apache Kafka and Apache ZooKeeper are documented as not working on Single Node clusters, because they expect daemons on multiple nodes. Most other actions work out of the box there since they run only on the master.

## Sources

- [GoogleCloudDataproc/initialization-actions on GitHub](https://github.com/GoogleCloudDataproc/initialization-actions)
- [Issues](https://github.com/GoogleCloudDataproc/initialization-actions/issues)
- [License: Apache-2.0](https://github.com/GoogleCloudDataproc/initialization-actions/blob/main/LICENSE)
- [Project website](https://cloud.google.com/dataproc/init-actions)
- [README](https://github.com/GoogleCloudDataproc/initialization-actions/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/googleclouddataproc-initialization-actions