Manifold: one send per node instead of one per recipient

The problem is one network send per recipient

Erlang and Elixir make it easy to send a message to any PID on any node, and the trap is that sending to N PIDs across a network copies the message N times. The README also states a plain cost figure: send calls run about 70 microseconds per operation, so a loop over recipients eventually becomes the bottleneck.

Discord's situation makes the numbers concrete. They run a single GenServer per Discord server, and some of those have around 100,000 PIDs connected to them from many different Erlang nodes. The GenServers were falling behind on their own message queues, and the cause was that 70 microseconds multiplied by connected sessions.

That is the shape Manifold addresses. It is not a queue, a buffer, or a batch scheduler. It changes who does the sending and how many times the message crosses the network.

Group by node, hash to workers, then send

The mechanism is described in three steps, and each one exists to bound a specific cost.

First, Manifold groups the target PIDs by their remote node. Second, it sends once per distinct remote node to a `Manifold.Partitioner` process on that node. Third, the partitioner hashes the PIDs with `:erlang.phash2/2`, groups them by number of cores, and sends to child workers, which finally send to the actual PIDs.

The guarantee this provides is the useful part. The sending process calls `send/2` at most once per involved remote node, no matter how many recipients there are. The partitioner does not become overloaded because the hashing and grouping spread across workers, and the linearizability guaranteed by `send/2` is preserved.

The README reports the outcome for Discord directly: packets per second dropped by half immediately after deploying, and the affected servers were able to keep up with their message queues. That is the project's own measurement on its own workload, which is the right frame for it, since the saving is proportional to how many recipients share a node.

A fanout to one PID per node gets you nothing from this design. The win scales with recipients per node, and it is bounded above by the number of nodes involved.

The API is send/2 with a list

Usage is deliberately thin. The dependency goes in your `mix.exs`:

defp deps do
  [{:manifold, "~> 1.0"}]
end

Then you call it like the builtin, except a list of PIDs is also accepted:

Manifold.send(self(), :hello)
Manifold.send([self(), self()], :hello)

The two forms are the whole surface. If you have a function that takes a single PID, you can pass a list without changing its contract, which is why the change is usually small at the call site and large at the network boundary.

The single-PID form is worth noting as the degenerate case. Manifold still does the grouping work, finds one node, sends to one partitioner, which sends to one PID. The overhead of the extra hop is real, and the README does not document a threshold below which plain `send/2` is better. Using Manifold for every send in an application, rather than for the fanout paths, is the mistake this API invites.

pack_mode: :binary pays off on large messages

Two performance options exist and both are opt-in. The first is `pack_mode`.

By default, and when `pack_mode` is unset, `nil`, or `:etf`, Manifold uses vanilla External Term Format. That is the default because it is the behaviour you already have.

Set `pack_mode: :binary` and Manifold ensures the term is converted into External Term Format exactly once, in the sending process, rather than once per receiving node:

Manifold.send(self(), :hello, pack_mode: :binary)

The rationale is arithmetic. Without the optimisation, a message sent to N nodes is translated into External Term Format N times. With large nested data structures that conversion is expensive, and doing it once instead of N times is the saving.

The README is precise about when it helps: the binary pack mode works best with large and deeply nested messages being sent to PIDs across multiple nodes. It does not claim a benefit for small messages, and for those the extra encoding step is at best neutral.

The pattern here is that each option targets a different cost. `pack_mode` targets encoding, which scales with message size. The next option targets who performs the send.

send_mode: :offload moves the work, and the README says it might backfire

The second option changes the process doing the sending. By default Manifold sends from the caller process. With `send_mode: :offload`, the send is offloaded from the calling process to a pool of `Manifold.Sender` processes.

Linearizability is maintained by consistent hashing of the calling process onto the same `Manifold.Sender`, so any given caller always offloads to the same sender. The pool size is configured separately:

config :manifold, senders: <size>

Manifold.send(self(), :hello, send_mode: :offload)

What makes this section worth reading carefully is the warning attached to it. The optimisation works best when the cost of sending to a local process is lower than the cost of sending over distribution, which the README says is most common for very large messages. And then plainly: for some workloads it might degrade overall performance, use with caution.

There is a second warning, and it is a correctness one rather than a performance one. Do not mix calls to Manifold with and without offloading. Mixed use of the two send modes against the same set of receiving nodes would break the linearizability guarantee.

That makes `:offload` the one option in this library you cannot enable incrementally across a codebase without auditing every call site first.

Four configuration keys, each with a cap

All application environment options are optional, and the README states that Manifold applies sensible runtime defaults when they are unset, so you only configure what you want to override.

config :manifold,
  gen_module: MyGenModule,
  partitioners: 1,
  workers_per_partitioner: 8,
  senders: 8

`gen_module` sets the module Manifold dispatches to in order to actually call send. The default is `GenServer`, and the configured module must define a `cast/2` function matching the types of `GenServer.cast/2`. That is the most interesting of the four, because it lets you replace the mechanism the partitioner uses to reach the real PIDs, which is where a cluster already running a different supervision or process registry would hook in.

`partitioners` is the number of `Manifold.Partitioner` processes per node, defaulting to `1` and capped at `32`. Raising it spreads partitioning work across more processes so the partitioner does not become the bottleneck.

`workers_per_partitioner` is the number of child workers each partitioner uses for the actual sends, defaulting to `System.schedulers_online()`, which is typically your core count.

`senders` sizes the `Manifold.Sender` pool used by `send_mode: :offload`, also defaulting to `System.schedulers_online()` and capped at `128`.

Two of the four have hard caps. That is worth noticing if you were planning to scale `partitioners` past 32 on a large node.

Where this fits, and what to weigh before using it

Manifold sits in a narrow gap. If you send to one remote PID, plain `send/2` is simpler and has one less hop. If you fan out to many PIDs on many nodes, plain `send/2` serialises encoding and repeated network copies, and that is the case Manifold was built for.

The alternative approaches each solve part of it. Buffering or batching at the application level defers work but does not reduce the copies, and it changes when the message is sent rather than how. A single dispatcher process per node that owns the recipient list reduces copies without touching the per-PID send, at the cost of that process becoming a single point of congestion. Manifold's design, grouping by node and hashing to workers, is essentially that idea applied at every hop rather than only the first.

The honest caveat is the maturity signal. The version history is thin: 1.3.0 in 2018, then 1.5.0 and 1.5.1 both in September 2022, and the last push was on 2026-07-07. There are no GitHub releases listed beyond those tags, and the build status badge still points at Travis CI. That is a library with a small API surface and a narrow purpose, which is not the same as an abandoned one, but it is a different thing from an actively developed dependency.

The repository carries a `bench/` directory alongside `lib/`, `priv/`, and `test/`, and the `priv/packets.png` image referenced in the README is the packets-per-second chart from the Discord deployment, which suggests benchmarking is part of the project's own practice.

The licence is MIT.

Editorial conclusion

Adopt Manifold when one process fans out to many remote PIDs and the send cost dominates, which is the pattern it was built for and the case where the win is structural. Do not enable send_mode: :offload without measuring, since the README states it can degrade performance depending on workload and that mixing offloaded and non-offloaded calls to the same receiving nodes breaks the linearizability guarantee. Verify first which of the two send modes your code already uses before adding the option.

Frequently asked questions

Official sources

1. discord/manifold on GitHub
2. Issues
3. License: MIT
4. README
5. Releases