python-redis-lock: a Redis lock that behaves like threading.Lock

The interface is deliberately the threading.Lock interface

python-redis-lock is a lock context manager implemented via redis SETNX and BLPOP, and the design goal is stated up front: the interface is targeted to be exactly like threading.Lock. That single sentence explains most of the appeal. A developer who already knows how to use a threading lock can read the same code and understand the failure modes, and the fact that it is a distributed lock rather than an in-process one is an implementation detail at the call site. The usage pattern is the standard one, with a connection, a name, an acquire attempt, and a release.

    from redis import Redis
    conn = Redis()

    import redis_lock
    lock = redis_lock.Lock(conn, "name-of-the-lock")
    if lock.acquire(blocking=False):
        print("Got the lock.")
        lock.release()
    else:
        print("Someone else has the lock.")

The one design consequence the README calls out is that because the library does not want to require users to share the lock instance across processes, you have to give the locks names. A name is the coordination point, since every process that wants the same lock has to know the same string. The non-blocking acquire returning a boolean rather than raising is another threading-like choice, and it means you can check the state of a contended resource without waiting. The package is on PyPI as python-redis-lock, requires Python 3.10 or newer, and has no runtime dependencies of its own.

Two Redis keys per lock, one for the lock and one for the waiters

The implementation section explains the mechanism, and it is the part to understand before you rely on it. The library uses two keys for each lock named with the given name. The first is a string value for the actual lock, and the second is a list value for signalling the waiters when the lock is released. The first is the familiar SETNX with expiry pattern that everybody writes by hand. The second is the part that makes this library different from a hand-rolled version, and it is what removes the spin loop. Instead of a blocked process polling the lock key in a loop with a sleep, a waiter blocks on the list, and the holder pushes to the list on release to wake it. The feature list states it as no spinloops at acquire. That matters under contention, because a polling implementation turns every waiting process into a client that hammers Redis, and the library instead makes the waiters passive until the lock is actually handed over. The list also explains the two-key footprint, since the signal list has to exist for waiters to block on, and a library that used a single key would have to poll or use a pub/sub channel that does not survive a subscriber going away.

Expiry, timeouts and automatic renewal, and what each one is for

The features list names four things: optional expiry, optional timeout, optional lock renewal using a low expire but keeping the lock active, and no spinloops at acquire. The distinction between expiry and timeout is the one that matters in practice. Expiry is the lease on the lock, how long the key survives in Redis before it disappears on its own, and it is the safety net for the case where the holder dies. Timeout is how long the acquiring process is willing to wait for the lock, and it is what stops a request from blocking a worker thread indefinitely. Renewal is the mechanism that lets those two coexist, since a long job that holds the lock for longer than its expiry would otherwise lose the lock while still running. The renewal is used with a low expire value, with the library periodically resetting the expiry while your code runs. The dogpile section shows the same combination in a Django cache context, and the lock there is used as a cache lock around an expensive computation, which is the classic cache stampede problem, described in the README as the dogpile, also known as the thundering herd effect or cache stampede, and the pattern avoids serving stale data while ensuring the work is performed a single time and every client waits for the fresh data.

A Django cache backend with a lock method

The library also ships a Django integration, and it is a small one that solves a specific problem. To use it you need django-redis, and python-redis-lock provides a cache backend with a cache method for your convenience, installed with the django extra.

    pip install "python-redis-lock[django]"

The settings then point at the backend.

    CACHES = {
        'default': {
            'BACKEND': 'redis_lock.django_cache.RedisCache',
            'LOCATION': 'redis://127.0.0.1:6379/1',
            'OPTIONS': {
                'CLIENT_CLASS': 'django_redis.client.DefaultClient'
            }
        }
    }

The backend adds a convenient lock method to the django-redis cache backend, and the usage is the double-checked locking pattern where the value is read, and if it is missing the lock is taken, the value is read again, and only then is the expensive work done and the result stored. The README notes a version interaction worth knowing, that if you use a django-redis older than 3.8.x you will probably need the deprecated redis_cache module, and that the redis_cache module was removed in django-redis versions above 3.9.x. So the supported window is a specific band of django-redis versions, and pinning to the wrong side of it produces an import error rather than a warning.

The failure mode is a lock that outlives its holder

The troubleshooting section is the most valuable part of the documentation, because it names a failure that every Redis lock has. In some cases, the lock remains in Redis forever, for example after a server blackout, a Redis or application crash, or an unhandled exception, and in such cases the lock is not removed by restarting the application. That is the direct consequence of a lease-based lock with no watchdog, and it is the thing that will bite you in production. The README offers two remedies and then warns you about them. The first is to turn on the auto_renewal parameter in combination with expire, so the lock has a time-out but the Lock object automatically keeps resetting the expire time while your application code is executing.

    # Get a lock with a 60-second lifetime but keep renewing it automatically
    # to ensure the lock is held for as long as the Python process is running.
    with redis_lock.Lock(conn, name='my-lock', expire=60, auto_renewal=True):
        # Do work....

The second is to call reset_all when the application starts.

    # On application start/restart
    import redis_lock
    redis_lock.reset_all()

Individual locks can be reset with the reset method, and the README closes this section by saying to use these carefully, if you understand what you do. That warning is the point. reset_all is a blunt instrument, and a service that resets every lock on startup will happily steal a lock that another healthy instance is holding, so the two remedies suit different failure domains, one for a crashed holder and one for a full outage.

Ownership, release and the limits of the guarantee

The library supports an owner identifier, which is useful when the application needs to know who holds the lock. You associate an identifier with the lock so it can be retrieved later by the same process or by a different one, and the example uses the hostname to build one.

    import socket
    host_id = "owned-by-%s" % socket.gethostname()
    lock = redis_lock.Lock(conn, "name-of-the-lock", id=host_id)
    if lock.acquire(blocking=False):
        assert lock.locked() is True
        print("Got the lock.")
        lock.release()
    else:
        if lock.get_owner_id() == host_id:
            print("I already acquired this in another process.")
        else:
            print("The lock is held on another machine.")

The last two branches are the point of the feature, since they distinguish the case where this host already holds the lock from the case where another machine holds it, and a naive implementation would treat both as contention. What the feature does not give you is a fencing token, a monotonically increasing number that the holder presents to the resource it is protecting, and without one the protection is not safe against a holder that pauses long enough to lose its lease and then resumes and writes anyway. The library does not claim to solve that, and the renewal mechanism is a mitigation rather than a proof. The scope of the guarantee is mutual exclusion against a well-behaved holder that respects the expiry, which is most of what people need and not all of what distributed locking means.

Editorial conclusion

Adopt python-redis-lock if you run multiple Python processes that must serialise work, and you want a lock whose interface matches threading.Lock closely enough that a reader does not have to learn a new model. Skip it if your contention is heavy, because the implementation is built around blocking list operations rather than a spin loop, and skip it if you need a fencing token or a guarantee stronger than a lease. Verify first that your Redis version and deployment tolerate the two key per lock pattern, that you set an expiry and a timeout appropriate to the work being serialised, and that you understand the failure mode where a crashed holder leaves the lock in place until something clears it.

Frequently asked questions

Official sources

1. ionelmc/python-redis-lock on GitHub
2. Issues
3. License: BSD-2-Clause
4. Project website
5. README