# openacid/slim: A compact trie for indexing external data in Go

> Slim is a Go library for minimizing in-memory indexes over large external datasets. It targets read-heavy, mostly static data where retaining payloads in memory is less useful than retaining a compact key index.

**openacid/slim** — Surprisingly space efficient trie in Golang(11 bits/key; 100 ns/get).

- Repository: https://github.com/openacid/slim
- Website: https://openacid.github.io/
- Stars: 1,927 · Forks: 99
- Language: Go
- License: MIT
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/openacid-slim

## Why SlimIndex keeps payloads out of memory

Slim addresses a specific storage problem: a large collection may reside on disk or inside serialized records, while application memory is better spent on an index that can locate those records. The project describes this as the gap between growing data volume and limited memory capacity. Its response is not a general database that owns every byte of the dataset. It is a minimized index whose principal job is to map a key to the location or identity of an external value.

The repository exposes `SlimIndex` as the common index structure and `SlimTrie` as the underlying trie. In the README example, `SlimIndex` is initialized with `index.OffsetIndexItem` entries. Each item pairs a key with an `int64` offset. The containing data type implements `Read(offset int64, key string)`, allowing a successful lookup to lead back into the external data without storing every payload in the index itself.

This makes Slim relevant to static SSTable-like records, serialized files, and other datasets whose records can be read by position. It is also relevant when keys are unusually long. The README states that keys of 16 KB are supported and that memory consumption relates to key count rather than key length. That property matters when prefix compression would otherwise create substantial engineering and storage work, although the actual payload format and lookup process remain the caller's responsibility.

The intended users are Go engineers designing compact read paths for external data. Slim is less convincing for a conventional mutable key-value store. Its documentation presents an ordered static structure and does not document transactions, crash recovery, concurrent writes, or rollback.

## SlimTrie encoding, ordering, and lookup behavior

`SlimTrie` is the central data structure. The README describes it as an evolution of a trie and claims a minimized representation of 11 bits per key. It stores keys in order and assigns a `Get()` operation a documented complexity of `O(log(n) + k)`, where `n` is the key count and `k` is the key length. This complexity accounts for both index traversal and work associated with the requested key.

The exact bit-level encoding is not included in the README section, so it would be misleading to claim how bits are assigned to prefixes, suffixes, offsets, or branching information. The visible public design is still clear: `SlimTrie` holds index information, `SlimIndex` connects that structure to externally readable data, and ordered storage supports range-oriented use. The README advertises range scans, although one feature line also refers to readiness in `0.6.0`, creating an ambiguity that repository examples and package documentation would need to resolve before an application depends on the exact version.

Two operating modes are named in the README table of contents: filter mode and key-value mode. The performance section says its benchmark uses filter mode to maximize memory reduction and performance. That wording indicates that mode choice affects the representation, but the README does not document the full behavioral or capacity difference. The benchmark model stores `Elts []*KVElt` separately and uses `SlimTrie` as an index, which illustrates how filter mode can coexist with values held outside the compact structure.

Slim also uses Protocol Buffers for its serializable data structures. The README states that a single `proto.Marshal()` is sufficient to serialize a trie for transport or persistence. That is useful for immutable or reloadable indexes, but the documentation does not specify compatibility guarantees, concurrent mutation rules, or recovery procedures.

## First indexed lookup with the documented SlimIndex example

The README does not provide an explicit installation command. It points readers to its GoDoc and PkgGoDev pages, and `go.mod` identifies the module as `github.com/openacid/slim` with Go 1.17. Because no documented module version or release command is supplied, adding an invented version would be misleading. In a Go project, the first confirmed integration point should therefore be the `github.com/openacid/slim/index` package shown below.

The repository example defines external data and a reader that starts parsing at an offset. This callback keeps record access outside the index itself:

```go
func (d Data) Read(offset int64, key string) (string, bool) {
    kv := strings.Split(string(d)[offset:], ",")[0:2]
    if kv[0] == key {
        return kv[1], true
    }
    return "", false
}
```

The index is then constructed from key and offset pairs. The README calls this a prebuilt index because construction does not infer offsets from the source data:

```go
keyOffsets := []index.OffsetIndexItem{
    {Key: "Aaron", Offset: 0},
    {Key: "Agatha", Offset: 8},
    {Key: "Al", Offset: 17},
    {Key: "Albert", Offset: 22},
    {Key: "Alexander", Offset: 31},
    {Key: "Alison", Offset: 43},
}

st, err := index.NewSlimIndex(keyOffsets, data)
```

A lookup returns both a found flag and the external value:

```go
v, found := st.Get("Alison")
v, found = st.Get("foo")
```

According to the README output, `Alison` returns `found: true` and value `8`, while `foo` returns `found: false` and an empty string. A real adoption should generate or validate `OffsetIndexItem` values against its own record format. Copying the example without doing so would only index a toy string correctly.

## Sparse indexes and external-data access

The README also describes sparse indexes for datasets containing very large numbers of records. Instead of assigning one index item to every key, several adjacent keys can share an item. It gives an example of indexing every fourth key or at a wider chosen interval. The application can then use the sparse entry to locate a nearby region, inspect additional keys from the external data, and refine the target position.

This approach can reduce index memory when external reads are already coarse. The README uses the example of billions of 4 KB objects on a 4 TB disk and argues that reading either 4 KB or 1 MB may take about 20 ms for one disk I/O. That example is about choosing a sparse interval based on storage behavior, not a universal setting. Different record sizes, cache behavior, and latency profiles require different choices.

Sparse indexing introduces a clear failure boundary. An incorrect seed, offset interpretation, or record parser can cause a valid key to be missed even if the compact trie itself operates correctly. The repository describes the sparse-index example, but the README does not specify recovery after a failed coarse lookup or how overlap is handled across boundary records. Engineers should therefore treat record validation as part of the index contract rather than assuming the trie knows the external file format.

The structure fits a read path that repeatedly searches a mostly stable dataset. It does not by itself solve updates. If an external record changes length, its stored offset can move later records, making previously constructed offsets invalid. Rebuilding or otherwise reconciling the index would be necessary, although the README does not define an incremental update procedure.

## Limits of Slim compared with btree and binary search

The README uses `github.com/google/btree` and binary search as comparisons and reports relative performance and memory figures. Those figures belong to the repository's benchmark context and should not be treated as guarantees for another workload. The benchmark constructs a slice of key-value pairs, uses `SlimTrie` as an index, and runs the trie in filter mode.

The practical difference is where the data lives. A binary search approach operates over an ordered collection with suitable random access. `SlimIndex` instead keeps key-position metadata and delegates payload retrieval to a `Read(offset int64, key string)` callback. That separation is useful when values are serialized or stored on disk and should not occupy the main memory budget.

The btree comparison is less one-dimensional. The README calls the example storage structure a static btree-like key-value store and separately reports that Slim is faster in its benchmark. Slim's stated objective is to minimize index state and encode long keys economically, but the README does not explain the btree comparison sufficiently. The README does not explain the internal btree comparison sufficiently to make claims about mutation behavior, concurrency, or worst-case performance.

Slim is the wrong tool when immediate lookup requires updates to both keys and payloads. The ordered index focus, prebuilt offsets, and absent mutation documentation make it less suitable for a transactional store. It is also a weak candidate when callers require values to be returned from a fully in-memory map with no external reader, because that is exactly the coupling Slim is designed to avoid.

## Serialization, licensing, and maintenance constraints

Serialization is part of the library's design rather than an incidental helper. The repository contains `encode/`, includes Protocol Buffer dependencies in `go.mod`, and states that `proto.Marshal()` can serialize a trie for persistence or transport. An application can therefore keep its expensive external dataset elsewhere and reload a compact index representation, subject to compatibility rules in the generated protobuf definitions.

The repository is licensed under MIT. That permits use, modification, distribution, and sublicensing under the license's stated conditions, although software teams should review the actual `LICENSE` file and their own obligations. The MIT identifier alone does not answer questions about dependencies, generated code, or the legal effect of a particular deployment.

Maintenance status needs a narrow description. The repository is not archived, and its last push was on 2026-08-24. It has no GitHub releases. That absence matters for upgrades: the README and repository release page do not identify release tags, changelogs, compatibility promises, or a migration path. `go.mod` declares Go 1.17 and pins several dependencies, but a module path without a documented release tag does not establish which Slim version a downstream service should select.

The available maintenance information therefore supports using the repository as current source, but it does not support promising predictable release cadence. The last push date is recent, yet release planning remains undocumented in the README and repository release list.

## Conclusion

Slim is appropriate for Go services that need a read-optimized, mostly static index over disk or serialized records and can tolerate the assumptions behind its trie representation. It is a poor fit for workloads needing documented mutations, transactional updates, or release-tagged upgrades. Before adoption, verify that `SlimIndex` satisfies the required record ordering, retrieval, serialization, and update behavior of the actual dataset.

## FAQ

### What is openacid/slim used for?

It provides compact Go data structures for indexing large external datasets. `SlimIndex` can map keys to offsets and retrieve records through a reader callback without loading every payload into the index.

### How does openacid/slim compare with a B-tree?

The README reports that its benchmark is faster than `github.com/google/btree` and has an overhead of about 11 bits per key. It also compares Slim with binary search over in-memory data.

### Does openacid/slim support long keys and serialization?

The README states that `SlimTrie` supports keys up to 16 KB and that memory consumption relates to key count rather than key length. It also states that a trie can be serialized with `proto.Marshal()` for transport or disk persistence.

## Sources

- [Issues](https://github.com/openacid/slim/issues)
- [License: MIT](https://github.com/openacid/slim/blob/main/LICENSE)
- [openacid/slim on GitHub](https://github.com/openacid/slim)
- [Project website](https://openacid.github.io/)
- [README](https://github.com/openacid/slim/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/openacid-slim