megaref: Path-Level STM Transactions for Clojure's Associative Values

The Granularity Problem in STM

Clojure's STM uses refs for coordinated, synchronous change. A ref holds one value, and any transaction that alters it conflicts with any other transaction altering the same ref, even if they touch different parts of that value. For a large map or vector, that means one writer blocks all others, regardless of which keys they update. The README identifies this as the granularity problem. megaref attacks it directly by letting you update a path inside a ref without conflicting with updates on other paths. This matters for applications that keep a large shared state, like a game world or a configuration tree, where transactions frequently modify distinct branches. The library targets existing codebases: it does not require you to redesign your data flow, only to swap ref types and call sites.

Associative Refs and Subrefs: Two New Reference Types

megaref introduces two new ref types. The associative ref, called a megaref, stores an associative value and allows path-keyed updates. A subref is a scoped view onto another associative ref. The README states that subrefs exist to replace regular refs in existing code, easing migration. Instead of having several independent refs, you aggregate them into one megaref and replace each original ref with a subref that points to a path in that megaref. This is a key design choice: it lets you keep the same shape of code, with refs and transactions, while gaining finer-grained conflict detection. The library also provides drop-in replacements for alter, commute, ref-set, and ensure that work on both regular refs and the new types. That uniformity is what makes the migration path practical.

How alter-in and commute-in Resolve Conflicts

The core mechanism is path-level conflict detection. A transaction that calls (alter-in r [:a :b :c] f ...) will not conflict with another transaction that calls (alter-in r [:a :b :d] f ...), because the paths diverge at the last segment. However, the README warns that updating [:a :b] and [:a :b :c] will conflict, because one path is a prefix of the other. This prefix check is on by default and can be disabled with the :check-prefixes option set to false. The documentation is explicit: only disable it if you never update paths that are prefixes of each other. This is a real constraint. If you have a transaction that updates a whole subtree and another that updates a leaf within that subtree, you will still get conflicts, which is correct for consistency but limits the concurrency gain.

Getting It Running: Commands and Options

Installation is a standard Leiningen dependency: [net.cgrand/megaref "0.3.0"]. The README shows a usage example. You require the namespace, create a megaref with (mega/ref (vec (repeat 10 (vec (repeat 10 0))))), then use mega/alter-in inside dosync. The benchmark in the README compares a regular ref with a megaref, updating random paths. The output shows 3864 ms for refs versus 2170 ms for megarefs, a 44% reduction, though this is a single benchmark and not a guarantee. Two runtime options matter. :guards-count controls the number of stripes used for locking; higher values increase throughput until a plateau. The README advises that the optimal number depends on concurrency level, average number of paths modified, path length, and the :check-prefixes setting. You can query and set options at runtime via get-options and set-option!, which also cover validator functions and history settings.

The Migration Path for Existing Code

The README describes a concrete conversion strategy: aggregate several refs into one megaref, replace those refs with subrefs, and replace all calls to alter, commute, ref-set, and ensure with their homonymous replacements. It links to two commits that convert Rich Hickey's original STM ants demo to megarefs. This is a practical guide, not just a promise. The key benefit is that you do not have to change the shape of your code, meaning the same transaction logic works. However, the migration is not trivial: you must identify which refs can be safely aggregated, and you must ensure that no transaction updates a path that is a prefix of another path in the same transaction, unless you accept the conflict. The subref approach means that existing code that dereferences a ref will still work, because subrefs behave like refs from the caller's perspective.

Limitations: Prefix Conflicts and Tuning Complexity

The most obvious limitation is the prefix conflict rule. Even with :check-prefixes set to false, you risk silent data corruption if two transactions update paths where one is a prefix of the other, because the README says you should only disable it when you never do that. This is a sharp edge. Another limitation is that the library is old. The copyright dates are 2012-2013, and the last push is unknown, with no recent releases listed. That raises maintenance concerns: it may not be compatible with newer Clojure versions, and there is no evidence of active development. The tuning of :guards-count is also non-trivial. The README lists several factors but gives no formula, so you must benchmark your specific workload. This is not a set-and-forget library; it requires profiling to get the concurrency benefit.

Alternatives: Ref Types vs. Other Concurrency Approaches

The obvious alternative is to stick with regular Clojure refs and accept the granularity problem, or to redesign your data to use smaller refs. Another alternative is to use atoms with CAS loops, but those do not provide transaction semantics across multiple values. A more modern approach is to use immutable data structures with persistent vectors and maps, combined with core.async or agents, but those do not give you synchronous, coordinated transactions. The key difference is that megaref keeps STM's transaction guarantees while adding path-level granularity. If you do not need STM, you might not need megaref at all. The README itself acknowledges that the library is about solving the granularity problem within STM, not replacing it. For new code, you might prefer to design with smaller refs or use a database that supports row-level locking, but that changes the architecture.

Editorial conclusion

Adopt megaref if you have an existing Clojure codebase using STM refs with large associative values, and you need to reduce conflicts without restructuring your code. Skip it if your workloads rarely update the same ref concurrently or if you can redesign with immutable data structures. Before adopting, verify the :check-prefixes behavior matches your update patterns, tune :guards-count for your concurrency level, and confirm the EPL license is acceptable. The library is stable but old; test with your Clojure version, as the last push is unknown and no releases are listed beyond 0.3.0.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. README
4. Releases