Dalli, the Ruby memcached client: what its retry, drain and size rules do to your code

The memcached 1.6.27 floor exists because the meta protocol rejects unknown flags

Dalli requires Ruby 3.3 or later (JRuby is also supported) and memcached 1.6.27 or later. The server requirement is stricter than a typical version note because of how the meta protocol behaves. A server that does not recognize a flag does not ignore it. It rejects the request outright, so a feature added to Dalli after your server's release fails with CLIENT_ERROR invalid flag instead of quietly degrading to something simpler.

The consequence for operators is that an old memcached box in a mixed fleet is not a slower node, it is a broken one for whichever calls use the newer flags. Earlier 1.6.x servers are listed as unsupported, and the project states that it tests against two points: the minimum supported memcached version and the latest release. If you run a managed cache whose server version you do not control, check that number before upgrading Dalli, because the failure shows up at request time and not at connect time.

The repository also carries upgrade guides named 3.0-Upgrade.md, 4.0-Upgrade.md and 5.0-Upgrade.md at its top level, plus a 5_0_ROADMAP.md, which is where to look when crossing a major version.

A signal during a timed socket read can abort the whole Ruby process on 3.3.0 to 3.3.7

Dalli sets its socket timeouts with IO#timeout. Ruby 3.3.0 through 3.3.7 and 3.4.0 through 3.4.2 contain a bug, tracked as Ruby #21195, where a timed read that is interrupted by a signal kills the process. The message looks like this:

[BUG] rb_sys_fail_path_in(io_fillbuf, fd:N ) - errno == 0

The crash happens inside Ruby itself, so no rescue clause in your application or in Dalli can catch it. The fix landed in Ruby 3.3.8 and 3.4.3. The practical rule is to treat those patch levels as the real minimum for any service that sends signals to its workers, which includes process managers that deliver SIGTERM or SIGUSR1 while a cache read is in flight.

The stated requirement of Ruby 3.3 or later therefore hides a trap: 3.3.0 satisfies it and still carries the bug. If you see the BUG line above in a log, the remedy given is to upgrade to 3.3.8, 3.4.3 or anything newer.

Namespaces accept a Proc, so the prefix can change on every call

The namespace option partitions one memcached instance between applications or environments. A string prefixes every key. A Proc is evaluated on each operation, which makes per-tenant prefixes possible without building a client per tenant.

# All keys will be prefixed with "myapp:"
Dalli::Client.new('localhost:11211', namespace: 'myapp')

# Dynamic namespace using a Proc (evaluated on each operation)
Dalli::Client.new('localhost:11211', namespace: -> { "tenant:#{Thread.current[:tenant_id]}" })

The join character is a colon by default. The namespace_separator option replaces it, and the value must be a single non-alphanumeric character. The accepted examples are :, /, |, ., -, _ and #. So namespace: 'myapp' with namespace_separator: '/' yields keys that start with myapp/.

The cost of the Proc form is that correctness now depends on state outside the client. The example reads Thread.current[:tenant_id], so a thread that has not set it will compute whatever that expression returns at that moment, and the keys it writes land under that prefix. Dalli will not warn you about it.

value_max_bytes must match memcached -I, and the limit includes key length plus 63 bytes

memcached refuses items larger than its -I setting, which is 1MB by default. If you raise it on the server, you set value_max_bytes to the same figure on the client:

# memcached started with -I 2m
Dalli::Client.new('localhost:11211', value_max_bytes: 2 * 1024 * 1024)

The subtle part is what gets measured. Because memcached's limit covers the whole item, Dalli adds up the stored value (after serialization and compression), the key, and 63 bytes of per-item overhead, then compares that sum with value_max_bytes. Anything larger raises Dalli::ValueOverMaxSize before a single byte goes over the network. With the defaults, a 10-byte key leaves 1,048,576 - 10 - 63 = 1,048,503 bytes for the value.

Two consequences follow. A payload that is exactly 1MB of raw data will be rejected, because the key and overhead push it past the line. And compression changes the answer: the check applies to the compressed bytes, so a large but compressible object may pass where the same object stored uncompressed would not. If you leave the server at its default and set a larger client value, you get errors from memcached instead of the early Dalli exception.

defer_drain trades a round trip for replies left unread on the socket

A quiet block (or a multi block) normally ends by waiting for the replies that memcached still sends for its quiet requests, one round trip per server the block wrote to. Setting defer_drain: true makes the block return immediately.

dc = Dalli::Client.new('localhost:11211', defer_drain: true)

dc.quiet { dc.set('key', 'value') } # returns without a round trip
dc.get('key')                       # first drains that connection with a noop, then gets the value

The requests still go out right away. Only the reading of replies is postponed until the next non-quiet request to that server, which reads them first using one noop. Error replies to quiet requests are discarded in both modes, but with defer_drain they are discarded at that later point rather than at the end of the block.

The replies in question are real: NF for a quiet delete, incr or decr on a missing key, NS for an add or replace that did not store, and error replies. They wait on the socket until something drains them. That is a saving for code that interleaves quiet writes with reads, and a hazard for code that does not.

A worker that only makes quiet requests never drains and can stall its own connection

If a client mixes quiet writes with reads against the same servers, it drains as it goes and needs nothing extra. A client that only ever issues quiet requests, such as a job that only runs quiet deletes, never sends the next non-quiet request. Its unread replies keep growing. If they fill the connection's socket buffers, the connection can stall until a socket timeout closes it and Dalli reconnects.

The fix is to call drain_deferred_responses on a schedule that suits the workload, for example after each batch or job:

jobs.each do |job|
  job.keys.each { |key| dc.quiet { dc.delete(key) } }
  dc.drain_deferred_responses # reads and discards the replies waiting on each server
end

The call only sends a noop to servers that received quiet requests since they were last drained, so invoking it when nothing is pending costs nothing. That makes it safe to put in a loop without tracking state yourself. The failure mode is slow to appear: a delete-only worker can run fine for a long time and then start hitting socket timeouts under load, with the cause being buffers filled by replies it never read.

Dalli refuses to replay incr, add and CAS writes after a network error

On a refused connection, a reset, or a read that times out, Dalli reconnects and retries. A server that fails socket_max_failures times in a row is marked down, and requests to it raise Dalli::NetworkError, or go to another server when failover is on.

The catch is that a request which failed after part of it was sent may already have been applied by memcached even though the reply never arrived. So Dalli resends only when applying twice is harmless: reads, touch, set, replace and delete without a CAS, and the multi-key operations. It raises Dalli::NetworkError instead for incr, decr, append, prepend, add, the CAS writes (set_cas, replace_cas, delete_cas, cas, cas!), and get_with_metadata with vivify_ttl or recache_ttl, which fetch_with_lock uses.

For your code this means a counter increment can fail with an exception where a plain set would have been silently retried, and you must decide whether to re-issue it. Even a replayed delete has an edge: it can return false if the first attempt had already removed the key.

A connection idle for over a second is checked before the next request. If memcached (via idle_timeout), a load balancer or a firewall closed it, Dalli reconnects before sending and the request is not counted as a failure. A connection dropped without the other end closing it cannot be detected this way, and the request on it fails by the rules above.

Multi-key calls skip a down server at the start but raise if one dies mid-operation

get_multi, get_multi_cas, get_multi_with_metadata, set_multi and delete_multi treat two situations differently. If a server is already down when the operation begins, it is skipped: its keys are missing from the result, or they go to another server when failover is enabled.

A server that fails during the operation and cannot be reached on the retry is different. Without failover the operation raises Dalli::NetworkError, instead of reporting that server's keys as misses or leaving them unwritten. With failover its keys move to another server, or are left out when no server remains.

That split matters for callers who treat a missing key as a cache miss and recompute. A server that was already down looks like a miss, which is cheap but can cause a thundering recompute. A server that dies halfway surfaces as an exception, so a bulk read can fail where the same read a second earlier would have returned partial data.

Serialization is a separate concern. By default Dalli uses Ruby's Marshal, and the project carries a security note about that. Only data you control should be unmarshaled from a cache that other parties can write to, and the client exposes fine-grained settings for serialization and compression if the default does not fit.

Editorial conclusion

Dalli suits Ruby 3.3 or later applications talking to memcached 1.6.27 or later, where failover and tunable serialization matter. Skip it if your servers are older 1.6.x builds, and before shipping check your Ruby patch level, whether you rely on retried incr or add calls, and whether any worker only issues quiet requests.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. Release notes