Apache Pegasus: what the gap between Redis and HBase costs in operational complexity

The design sits between Redis and HBase, and both comparisons are load-bearing

Redis is fast because data lives in memory. That keeps latency low, but it does not guarantee strong consistency across replicas by default. HBase is written in Java, built on HDFS, and writes to a distributed filesystem even for operations local to a single node. Apache Pegasus targets the space between those two designs.

Pegasus is written entirely in C++. Its write path relies on the local filesystem rather than HDFS. It provides horizontal scalability through hash-based partitioning, strong consistency through the PacificA consensus protocol, and high performance through RocksDB as the underlying storage engine. Each of those properties addresses one of the two systems it positions itself against.

Who this is actually for comes through in the listed requirements: multiple-level data safety, fast data migration between data centers, automatic load balancing, and online partition split. Those are the needs Redis cannot satisfy without external tooling and the ones HBase satisfies at higher latency and Java operational overhead. Teams already comfortable with HBase's ZooKeeper and HDFS dependencies have less incentive to migrate.

PacificA requires all three replicas to confirm a write before the client sees success

The consistency guarantee is specific: each write is replicated three times to different ReplicaServers before the system responds to the client. A write that returns success has already been acknowledged by three nodes, which limits data loss to the simultaneous failure of all three replicas.

PacificA is a consensus protocol from Microsoft Research, designed for log-based distributed storage systems. It governs replication and membership changes at the same time, where membership events include nodes joining, leaving, or failing. Handling those through a formal protocol rather than ad-hoc operator steps is what allows the cluster to survive node failures without manual replica reassignment.

The cost is availability under partition. Strong consistency means some write requests will fail when the cluster cannot confirm a quorum. Pegasus does not describe its partition-tolerance behavior in the README, and the specific conditions under which writes block or fail are not spelled out there. That is a gap worth understanding before sizing a cluster for production availability targets, and the documentation at pegasus.apache.org is where that behavior should be described.

RocksDB handles local storage, and its compaction behavior is part of the cost model

RocksDB is a log-structured merge-tree storage engine designed for write-heavy workloads on SSDs. Writes go first to a write-ahead log and then to a memtable, which is eventually flushed and compacted into sorted string table files on disk. That structure is efficient for sequential writes but introduces read amplification for random reads and write amplification during compaction.

Pegasus's use of RocksDB means its performance characteristics are tied to RocksDB's compaction behavior. A cluster under sustained write load will spend CPU and I/O on background compaction. On an SSD, that is manageable. On spinning disks, it can become a throughput bottleneck.

Tuning RocksDB for a specific workload is a known discipline, and Pegasus inherits both its benefits and its complexity. The project names RocksDB as the storage engine but does not document any Pegasus-specific configuration for it in the README. Tuning guidance belongs to the project website or the docs/ directory at the repository root.

The README sends you to the website for setup; the repository has a docker/ directory and run.sh

No install commands appear in the README. It points to pegasus.apache.org for documentation on getting started. The top-level repository entries include a docker/ directory, a run.sh script, and a build_tools/ directory, which suggests a container-based path and a shell-driven build process exist. None of those are described in the README itself.

Client drivers are organized as subdirectories inside the main repository. Java lives under java-client/, Go under go-client/, Python under python-client/, Node.js under nodejs-client/, and Scala under scala-client/. The C++ client header is at src/include/pegasus/client.h. Each client connects to a running cluster, so server setup precedes any client use.

For anyone evaluating Pegasus without a pre-existing cluster, the entry point is the official documentation site rather than the README. One additional note from the README is worth keeping: the master branch may be in an unstable or even broken state during development, and it advises using tagged releases from GitHub Releases for stable binaries.

MetaServer manages replica placement and can schedule an automated rebalance

Pegasus has two server roles with distinct responsibilities. ReplicaServers hold the data. MetaServer manages the distribution of replicas across ReplicaServers and owns the load balancing function.

When the cluster falls into an unbalanced state, an administrator invokes a rebalance command. That command schedules replica migration automatically rather than requiring the operator to move each replica by hand. Load balancing is described as a built-in function of MetaServer, though the README does not describe what triggers an imbalance or how migration interacts with in-flight reads and writes.

Partition split is also listed as an online operation. Pegasus uses hash-based partitioning, so the number of partitions determines how data is distributed across ReplicaServers. Splitting partitions without downtime is what makes horizontal scaling possible without a maintenance window. The mechanics of how a split progresses, whether it pauses writes, and how it interacts with the duplication feature are not addressed in the README.

Duplication replicates to a remote cluster asynchronously, and the lag is not quantified

Pegasus calls its cross-datacenter feature duplication. A change made to the local cluster becomes accessible from a remote cluster after a short time. That is asynchronous replication, which means the remote cluster can lag behind the local one. The stated purpose is higher availability and better performance from local reads: a service that reads from a nearby cluster avoids the latency of a cross-datacenter round trip.

The distinction from three-way replication matters here. Three-way replication within a single cluster is synchronous and required for every write. Duplication is asynchronous and optional, designed for multi-datacenter deployments where synchronous replication across a WAN would add too much write latency.

Two limits are absent from the README. The first is the maximum lag the duplication mechanism tolerates before it reports a problem or blocks. The second is the behavior when both clusters accept writes simultaneously during a failover, since conflict resolution under that condition is not described. Duplication's actual fault model depends on both answers, and the project website is where to look for them.

Cold backup targets HDFS or local filesystem and connects to pegasus-spark for analysis

Cold backup is a separate mechanism from the live three-way replication. Pegasus supports an extensible backup and restore mechanism where snapshots can be written to HDFS or to the local filesystem. A snapshot stored on HDFS can then be fed into pegasus-spark, a related project at github.com/pegasus-kv/pegasus-spark, for analytical workloads.

That combination serves two purposes that key-value storage alone does not cover. First, a snapshot on separate storage is the recovery option when all three live replicas are lost. Three-way replication protects against independent node failures but not against operator errors that affect all replicas at the same time. Second, an HDFS-based snapshot lets a team run analytical queries against a fixed point-in-time copy of the data without adding I/O load to the production cluster.

The backup model is described as extensible, which means behavior depends on which implementation is configured. The schedule, the restore procedure, and the consistency guarantees of a snapshot taken from a live cluster under active write load are not described in the README.

Client support spans six languages, but releases have not caught up to recent commits

Client drivers are available for six languages: Java, C++, Go, Python, Node.js and Scala. All six live as subdirectories inside the main repository, so client releases are bundled with server releases rather than published separately on language-specific registries.

The release timeline has a gap worth examining before committing to the project. Version 2.3.0 arrived on 2022-01-13, followed by 2.4.0 on 2022-10-31 and then 2.5.0 on 2023-12-11. The most recent push to the repository was on 2026-06-16, which means over two and a half years of commits exist outside any tagged release.

For teams evaluating Pegasus, v2.5.0 is the stable artifact to start with, because the README explicitly warns against using the master branch for stable deployments. What changed between v2.5.0 and the current commit history is documented in HISTORY.md at the repository root, and reviewing that file before adopting the project is a concrete first step.

Editorial conclusion

Pegasus makes sense for a team that has outgrown Redis's consistency model and finds HBase's HDFS dependency too heavy for its stack. The C++ server, PacificA replication, and RocksDB storage form a well-defined architecture, but none of the setup details are documented in the README. Before adopting, read the operational documentation at pegasus.apache.org, confirm that PacificA's failure behavior fits your availability requirements, and check the HISTORY.md file at the repository root to see what changed after v2.5.0, since those commits span over two years and have not been released.

Frequently asked questions

Official sources

1. apache/incubator-pegasus on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases