n8n-autoscaling: Scaling n8n Workers From the Redis Queue Length

Scaling from queue length instead of a Kubernetes operator

The n8n Autoscaling System is a Docker-based autoscaling solution for the n8n workflow automation platform, and it dynamically scales worker containers based on the Redis queue length. The pitch against the alternatives is stated plainly: no need to deal with Kubernetes or any other container scaling provider, because a simple script runs it all and is easily configurable. The reference point is the alternative approach, which is a bigger orchestration platform where you would express the same scaling rule as a HorizontalPodAutoscaler and accept that platform's footprint. The performance claim in the readme is that this was tested with hundreds of simultaneous executions running on an 8 core 16GB RAM VPS, which is a specific and useful piece of information, because it says the worker pool does not need a large cluster to be useful.

What n8n 2.0 broke, and why the sidecar matters

The reason this project has an autoscaler for two things rather than one is n8n 2.0, which introduced breaking changes for task runners. Task runners are now separate containers in external mode, each worker needs its own task runner sidecar, the main n8n instance does not need a task runner since all executions are offloaded to workers, and external packages must be configured in the task runner image. The build handles all of this automatically, because the autoscaler scales both workers and their task runners together. The note in the services table makes the same point from the other side: the main n8n instance does not need its own task runner because all executions, including manual runs, are offloaded to workers with OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS set to true, and each worker has its own task runner sidecar. A scaler that moved only the workers would leave runners behind, and the ratio is 1:1.

The service graph, and what each container is for

The architecture diagram and the service table describe a fairly complete stack. The n8n service is the main instance with the editor and API, n8n-webhook is a dedicated webhook processor, n8n-worker is the queue worker that gets scaled, and n8n-worker-runner is the task runner for the workers. Redis is the job queue, PostgreSQL is the database with pgvector, n8n-autoscaler monitors the queue and scales workers and runners, and redis-monitor does queue monitoring. Around those are the optional pieces: n8n-backup for scheduled backups to cloud storage, cloudflared for a Cloudflare tunnel, and three Instance AI components, an internal sandbox API, a sandbox runner that is either Sysbox or a privileged Docker-in-Docker, and a one-shot mTLS certificate bootstrap, plus SearXNG for internal web search. The compose file at the repository root defines a bridge network and an external network named shark, and the worker services join both.

The setup wizard, and the daemon identity it protects

The recommended installation is a clone and a script.

git clone https://github.com/conor-is-my-name/n8n-autoscaling.git
cd n8n-autoscaling
./n8n-setup.sh

The wizard guides you through creating the .env file from the template, generating secure random secrets, setting the timezone, URLs, the Cloudflare tunnel and Tailscale, choosing the autoscaling parameters, configuring backups with schedule, encryption, cloud storage and notifications, detecting your container runtime as Docker or Podman and rootless or rootful, choosing a self-hosted sandbox, Daytona or no Instance AI sandbox at all, generating and preserving the sandbox mTLS and API secrets and the complete compose file list, creating the external network, and starting everything with health checks. One detail in the environment file explains why the wizard exists at all. Container named volumes belong to one daemon identity and not just to an engine name, so Docker versus Podman, rootless versus rootful mode and different local API sockets can all point at separate data stores, and the wizard persists all three fields and blocks an identity change rather than starting n8n against an empty database.

Manual setup for those who prefer to read the compose file

The manual route is the same steps without the wizard. You clone, copy the example environment file, and edit the variables.

cp .env.example .env

The values that matter are the ones the wizard would otherwise generate: strong passwords for REDIS_PASSWORD, POSTGRES_ADMIN_PASSWORD, POSTGRES_PASSWORD, N8N_ENCRYPTION_KEY and N8N_RUNNERS_AUTH_TOKEN, the domain settings including N8N_HOST and N8N_WEBHOOK, a CLOUDFLARE_TUNNEL_TOKEN, and optionally TAILSCALE_IP for private access. Then you create the external network yourself.

docker network create shark

And start the stack.

docker compose up -d --build

The environment file is where the scaling behaviour is set, and the defaults are visible in it: MIN_REPLICAS at 1, MAX_REPLICAS at 5, a scale-up queue threshold of 5, a scale-down threshold of 1, a polling interval of 10 seconds, a cooldown period of 10 seconds, a separate monitor poll interval of 5 seconds, and graceful shutdown timeouts of 300 seconds for both the queue bull and the general case.

Scraping with a browser inside the worker image

The build includes Puppeteer and Playwright with Chromium built in for what the readme calls pro level scraping from the n8n code node, along with stealth plugins for bot detection evasion. The implementation detail that makes it work is in the Dockerfile, which is a multi-stage build that adds tools to the n8n image and works around apk being stripped from the official image. A builder stage on alpine installs ffmpeg, git, openssh-client, graphicsmagick, jq and curl, and a second stage starting from the official n8n image copies the binaries and the shared libraries across, including the ffmpeg, swscale, curl, jq and oniguruma libraries. It also exposes port 5679 for the task broker, which is what external runners in n8n 2.0 connect to, and it drops back to the node user at the end.

Backups, hardening and the optional pieces

The feature list covers more than scaling. There are health checks and centralized log rotation for all services, with the compose file setting a json-file driver with a 10 megabyte size and three files by default. The security posture is described as hardened, covering Redis auth, localhost port binding, PostgreSQL user separation and non-root containers, though the main n8n service in the compose file is explicitly run as root:root, which is a nuance worth noticing against that claim. Scheduled backups take in PostgreSQL and Redis and volume data, with GPG encryption and multi-cloud upload, and there is a separate backup/ directory in the repository. The examples folder holds ready to import workflows, including Playwright and Puppeteer scrape, screenshot and stealth variants and a PDF one, plus a test for external Python packages.

The release line and the shape of the project

The version history shows a project that is still finding its shape, with 2.2.0 in August 2026 adding Instance AI and self-hosted sandboxes, 2.1.2 in the same month preserving Compose overrides during autoscaling, and 2.1.1 in July fixing a runner build caused by a pnpm store mismatch. That last one is informative, because it means the worker and runner images are built with pnpm and the store had to be reconciled. The last push was on 2026-08-28, and the repository is not archived. The layout follows the compose approach, with an autoscaler/ directory, a monitor/ directory, a backup/ directory, a Dockerfile and a Dockerfile.runner, several compose override files for the Cloudflare, Podman, Daytona and sandbox variants, a generate-systemd.sh for service generation, a n8n-task-runners.json file, and init-postgres.sh and init-pgvector.sql for database setup. The multi-architecture support covers amd64, arm64 and armhf, and Podman rootless works for the core stack through a compose override, though not for the upstream self-hosted sandbox service.

Editorial conclusion

n8n-autoscaling fits a self-hoster running n8n in queue mode who wants workers to grow with the backlog and shrink again, and who does not want to operate Kubernetes to get it. It is the wrong choice if you need a managed platform, since this is a compose stack you run yourself, and the wrong choice if your workload is bursty beyond what a single VPS can hold, because the scaling decision is based on queue length and a host with fixed resources. Before you start, run the setup wizard rather than copying the environment file by hand, since it also guards against the container daemon identity changing underneath you.

Frequently asked questions

Official sources

1. conor-is-my-name/n8n-autoscaling on GitHub
2. Issues
3. License: MIT
4. README
5. Releases