FrostDB for embedded analytical workloads in Go

FrostDB's target workload and Go embedding model

FrostDB solves a specific storage problem: a Go process needs to receive many records, keep their dimensions flexible, and later scan or aggregate selected columns without operating a separate database server. The README positions the project for workloads dominated by writes with occasional analytical queries. It was built for Parca and for observability data, where records such as time-series samples can acquire label names that are not known when an application is compiled.

The project is an embeddable, wide-column, columnar database. "Wide" matters here because a field such as Prometheus labels can expand at runtime rather than requiring every label name to appear in a fixed Go struct or SQL schema. The database is aimed at developers who can include FrostDB directly in a Go application and who think in column scans, filters, and aggregations. It is not positioned as a conventional row database, and it does not expose a standalone server as its intended deployment model.

This focus creates a clear boundary. FrostDB expects timestamp, value, and other regularly shaped fields, plus dynamic string columns for labels. It does not expect frequent updates or deletions. A conventional transactional application that fetches a whole row by primary key is outside the documented fit. The README also says FrostDB is optimized for most interactions being writes, followed by analytical reads, which is different from a request path dominated by small point lookups.

Parquet storage, Arrow execution, and dynamic-column mechanics

FrostDB separates its persistent representation from its query-time representation. Apache Parquet is used for storage because the README cites its encodings for reducing memory and disk use. Apache Arrow is used during queries as the basis for vectorized execution. The query builder and optimizers operate through DataFrame-like APIs on top of Arrow, so column-oriented processing remains central to ingestion, compaction, and analysis rather than being limited to the on-disk format.

Dynamic columns solve the semi-structured schema problem without storing an entire label map as an indivisible row value. The README contrasts that approach with a map because values in a map remain grouped by row, preventing the database from gaining the same columnar benefits. A FrostDB schema can mark a field as dynamic. When ingestion encounters a previously unseen label name, FrostDB creates the corresponding prefixed column on the fly.

The documented Prometheus example declares labels as a dynamic `map[string]string` and applies the `rle_dict,asc(1),null_first` options. The timestamp is ordered ascending at sort position zero, while the value remains an ordinary `float64` field. Dynamically created label columns must be strings, and the README states that they remain dictionary and run-length encoded. This is a meaningful restriction. A dynamic field is not an unrestricted document value or an arbitrary typed object per field.

The write path through Parts and LSM compaction

FrostDB treats stored data as immutable. Its index is organized as lists of Parts in a structure similar to a log-structured merge-tree. A Part contains either an Apache Arrow record or a Parquet file. New records enter the first level, L0, in the form received from the insertion path.

When L0 reaches its configured maximum size, it is compacted into one Parquet file and moved into the next configured index level. Compaction continues through the levels until a file reaches the final level. This gives FrostDB an ingestion-oriented path: recent inserts remain in Arrow-backed parts, while older accumulated data becomes Parquet-backed as the index grows.

When the complete index reaches its configured maximum in-memory size, FrostDB rotates the index. The application can configure the rotated index to be dropped or written to storage selected by the application. That storage boundary is useful for an embedded library because it does not force one object-store vendor, but it also means the embedding application must understand lifecycle and retention decisions. The README excerpt ends while describing how queries scan Parts, so it does not provide a complete account of merge ordering, concurrent compaction behavior, or recovery for every component represented in the repository.

Using the repository's simple example without inventing setup steps

The README does not publish an installation command, versioned release, configuration file, or complete runnable quick-start program. It directs new users to the repository's `examples/simple` and `examples/aggregations` directories. Since there are no GitHub releases, the README provides no release-specific installation target. The repository is a Go module whose import path and required Go version are defined by the repository itself.

The schema shown in the README is the first concrete indication of what an application should model. In a Go file, the Prometheus-shaped structure from the design section is:

type Prometheus struct {
	Labels    map[string]string `frostdb:",rle_dict,asc(1),null_first"`
	Timestamp int64             `frostdb:",asc(0)"`
	Value     float64
}

This declares the expected fields and options, but it is not a complete program. A reader should use `examples/simple` as the authoritative first-use flow because the README says that example creates a database with a dynamic-column schema, inserts data, and queries it back. The linked example is more relevant than replacing its missing details with guessed constructor names or configuration flags.

The repository Makefile provides one documented verification command:

make test

The target invokes Go tests with assertion and debug tags and the race detector across all packages. A reader should expect this command to be the project-level check offered by the Makefile. The README does not say that this command represents a complete production validation, and its warning about possible correctness problems makes that distinction important.

Why FrostDB differs from DuckDB and mutable database servers

The README names DuckDB as a comparable embeddable columnar database, but FrostDB makes a narrower design choice. Its defining feature is not merely columnar storage. It combines write-oriented Parts and LSM-like compaction with schema-defined dynamic columns for dimensions that appear at runtime. DuckDB is only described as similar; the repository documentation does not provide a feature-by-feature comparison, so claims about DuckDB's exact behavior should not be inferred from FrostDB's README.

Compared with a standalone analytical server, FrostDB is linked into a Go application rather than deployed as a separate service according to the project's stated purpose. That avoids a client-server component for the intended architecture, but it also transfers storage rotation, memory limits, compaction, and persistence configuration into the embedding program. The README says a rotated index may be dropped or sent to storage of the application's choice, which is evidence that integration work remains with the caller.

For mutable workloads, the alternative is fundamentally different: choose a database that supports updates and deletes rather than assuming an immutable analytical design can be adapted safely. For row-oriented point reads, choose a row-oriented store. These are workload differences, not preferences between two implementations of the same interface.

Production warnings, correctness risk, and maintenance cost

The strongest limitation is stated by the project itself. The README calls FrostDB an infancy-stage project, says it should not be considered production-ready, warns of consistency and correctness problems, and says all APIs will change. That warning should dominate an adoption decision. Recent code movement does not remove this stated API and correctness boundary.

The repository is not archived, and its last push was on 2026-10-14. The repository has no GitHub releases, so there is no published release cadence or upgrade path identified in the repository documentation. An application may therefore have to track `main` or otherwise select a repository revision, but the README does not prescribe a release policy, compatibility promise, or migration procedure. The `go.mod` file requires Go 1.24.1 and pins a broad set of Arrow, Parquet, gRPC, storage, and observability dependencies. Changes in those dependencies can add upgrade work, although the documentation does not quantify it.

FrostDB uses the Apache-2.0 license. That permits use and modification under the license's conditions, but operators still need to review the license and dependency notices for their intended distribution. This article does not provide legal advice. Operationally, the larger concern is correctness: the project itself warns about it, so adopting FrostDB means accepting that its present behavior has not been presented as production-ready by its documentation.

Editorial conclusion

Go developers prototyping an embedded, immutable analytics store for observability-shaped data should examine FrostDB's dynamic-column model and vectorized query path. Applications needing stable APIs, established production guarantees, standalone operation, or mutable records should use another system. The first verification is the intended write and query workload against the repository's `examples/simple` program, followed by `go test -tags assert,debug -race ./...`.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. polarsignals/frostdb on GitHub
4. README