Celery: Python Distributed Task Queue for Async and Scheduled Work

What Problem Celery Solves and Who Uses It

Celery addresses a common architectural constraint: a web request handler should not block while waiting for a slow operation such as sending email, processing a file, calling an external API, or generating a report. The solution is to hand the work off to a queue and let a separate worker process handle it. Celery provides the tooling to define tasks, send them to a broker, and execute them in worker processes that can run on the same machine or across a cluster of servers.

The README describes the task queue model clearly: worker processes monitor the queue continuously and pick up tasks as they arrive. Celery communicates via messages through a broker. The client puts a message on the queue, the broker delivers it to an available worker, and the worker executes the task. A Celery system can run multiple workers and multiple brokers simultaneously, enabling high availability and horizontal scaling without changing application code.

The README also notes that the Celery protocol is not Python-only. It lists client libraries for Node.js (node-celery), PHP (celery-php), Go (gocelery and gopher-celery), and Rust (rusty-celery).

Brokers, Workers, and the Message Architecture

Celery does not include its own message broker. It connects to an external one. The README states that RabbitMQ and Redis transports are feature-complete. Experimental support exists for SQLite, which is useful for local development without running a separate broker service. The setup.py extension list shows additional adapters: sqs (Amazon SQS), gcpubsub (Google Cloud Pub/Sub), consul, mongodb, couchdb, cosmosdbsql (Azure CosmosDB), cassandra, dynamodb, arangodb, and slmq (IBM MQ).

Tasks are Python functions decorated with the task decorator on a Celery application instance. The README shows the minimal structure:

from celery import Celery

Workers subscribe to queues on the broker and execute tasks as messages arrive. Each worker is a separate process, and multiple workers can run in parallel on the same or different machines. Adding more workers increases throughput without changing task definitions or broker configuration.

Installing Celery and Setting Up a First Application

Celery is available on PyPI at https://pypi.org/project/celery/. Install it with:

pip install celery

To use Redis as the broker, add the redis extra after the base install. The setup.py defines the complete set of optional extras: redis, librabbitmq, django, sqlalchemy, mongodb, sqs, gcpubsub, eventlet, gevent, msgpack, pydantic, pytest, tblib (for serializing stack traces across workers), yaml, zstd, brotli, and many more. Each extra pulls in the corresponding transport or integration package.

For Django projects, the repository ships a complete example in examples/django/. Periodic task scheduling (similar to cron) uses the celery beat scheduler; an example lives in examples/periodic-tasks/. Additional example directories cover pydantic validation, quorum queues, security, result graphs, and eventlet and gevent concurrency configurations.

Python Version Support and the v5.7 Boundary

Celery v5.6.x runs on Python 3.9, 3.10, 3.11, 3.12, and 3.13, as well as PyPy 3.9+ at version 7.3.12 or later. The README states explicitly: "This is the last version of Celery which will support Python 3.9. Celery v5.7.x will work on Python 3.10 or newer versions."

For older Python versions, the compatibility matrix is: Python 3.8 requires Celery 5.5 or earlier, Python 3.7 requires 5.2 or earlier, and the Python 2.x series requires the Celery 4.x line or earlier. Python 2.4 support ended with Celery 2.2.

The current stable release is v5.6.3, published on 2026-03-26. The v5.7.0a1 alpha appeared on 2026-09-22. Release notes live in Changelog.rst in the repository root, and the pyproject.toml records pytest configuration, mypy settings targeting a subset of the codebase, and codespell ignore rules.

Concurrency Models and Configuration Options

Celery's default concurrency model uses multiple processes (prefork). The extension list includes eventlet and gevent as alternatives for I/O-bound tasks. The tblib extension enables cross-worker stack trace serialization, which is useful for debugging failures that occur inside worker processes.

The README notes that Celery "does not need configuration files" for basic use; connection parameters are passed directly as arguments or through environment variables. For larger deployments, configuration files or configuration objects are supported, as documented in the getting-started tutorials linked from the README.

The pyproject.toml shows pytest markers used in the test suite: sleepdeprived_patched_module, masked_modules, patched_environ, patched_module, flaky, timeout, and amqp. These markers reflect the testing complexity of a library that must work across multiple brokers, Python versions, and concurrency backends.

Limitations: Windows, Broker Dependency, and Scaling Constraints

The README is direct about Windows: "Celery is a project with minimal funding, so we don't support Microsoft Windows but it should be working. Please don't open any issues related to that platform." Windows is not an officially supported environment, and Windows-specific issues are not handled.

Celery requires an external message broker for all production use cases. There is no embedded broker option. Running Celery in an isolated environment without network access to RabbitMQ or Redis requires additional infrastructure, which adds operational overhead compared to libraries that queue work within a single process.

The prefork concurrency model runs separate processes, so state cannot be shared in memory across workers. State must pass through the broker or a result backend. Mixing eventlet or gevent with libraries that are not compatible with cooperative multitasking can cause subtle bugs that are difficult to reproduce.

Celery vs. RQ: When a Simpler Queue Is Enough

RQ (Redis Queue) is a Python task queue that uses only Redis. Its API is smaller and its operational requirements are simpler: install Redis, install RQ, run a worker. The trade-off is scope. RQ does not support RabbitMQ or any broker other than Redis, does not include a built-in periodic task scheduler comparable to celery beat, and has fewer integrations.

Celery's advantage is its broker flexibility, the celery beat periodic scheduler, the cross-language protocol, and the extensive extension list. Applications that need only Redis-backed async tasks with a minimal setup often find RQ simpler to operate. Applications that need multiple broker options, periodic scheduling, or the broader Celery extension ecosystem benefit from Celery's additional complexity.

The README notes that Celery can run on a single machine or across multiple machines and datacenters, a deployment range that RQ does not cover without additional tooling.

Maintenance, Sponsorship, and License

The last push to celery/celery was on 2026-09-28. Release v5.6.3 shipped on 2026-03-26; the v5.7.0a1 alpha appeared on 2026-09-22. The project is funded through Open Collective and offers a Tidelift subscription for enterprise users who need commercial support and maintenance guarantees. The README lists commercial sponsors including Blacksmith and CloudAMQP.

The repository includes a CONTRIBUTORS.txt listing project contributors, a SECURITY.md for reporting security issues, a docker/ directory with Docker configuration, and a helm-chart/ directory for Kubernetes deployments. The pre-commit-config.yaml shows automated linting checks using Bandit (bandit.json in the root), flake8, and codespell.

GitHub's license detection returned NOASSERTION for this project, meaning the type was not automatically identified. A LICENSE file exists in the repository root. Consult it before deploying Celery in a commercial product.

Editorial conclusion

Celery is the established choice for Python applications that need async task processing at scale, periodic scheduling, and multiple broker backends. Applications on Python 3.9 should migrate before v5.7 removes that support. Check the LICENSE file for exact license terms before deploying in a commercial context; the license field shows NOASSERTION in GitHub's detection, meaning the type was not automatically identified.

Frequently asked questions

Official sources

1. celery/celery on GitHub
2. Issues
3. Project website
4. README
5. Releases