OpenTenBase: a distributed HTAP database you compile yourself

What OpenTenBase is, and what it inherited

OpenTenBase is described as an advanced enterprise-level database management system based on prior work of the Postgres-XL project. That lineage explains most of the design, and it is stated in the README rather than inferred.

On the SQL side, it supports an extended subset of the SQL standard, including transactions, foreign keys, user-defined types and functions. On top of that it adds parallel computing, security, management and audit functions. The repository is written in C, and the tree carries the artefacts of a long PostgreSQL derived history: configure.in and aclocal.m4, a GNUmakefile.in, doc/, contrib/, src/, a COPYRIGHT file, a Notice file and a stack of release note files going back through v2.2.0, v2.4.0, v2.5.0 and v2.6.0.

The interfaces are the reason this is worth a look. The README says OpenTenBase has many language interfaces similar to PostgreSQL, and points at the PostgreSQL download page for the list. In practice that means the drivers, the client libraries and the tooling you already use for PostgreSQL are the ones you would use here, which removes the usual objection to adopting a less common database.

The project is hosted at opentenbase.org, and the source and the latest version are both pointed at the GitHub repository.

The topology is coordinator, data nodes and a GTM

The cluster architecture is described in a short, dense paragraph, and it is the part to understand before anything else.

An OpenTenBase cluster consists of multiple CoordinateNodes, DataNodes and GTM nodes. All user data resides in the DataNodes. The CoordinateNode contains only metadata. The GTM is for global transaction management. The CoordinateNodes and DataNodes share the same schema.

And the access path is fixed. Users always connect to the CoordinateNodes, which divide up the query into fragments that are executed in the DataNodes, and collect the results.

That is the whole distributed model in three sentences, and it explains the two costs you inherit. First, a query is rewritten and scattered, so a coordinator becomes the entry point for everything and its behaviour matters for every workload. Second, the GTM exists because transactions spanning more than one data node need somewhere to coordinate, and that node is a piece of infrastructure that a single machine database does not have.

The configuration file documentation confirms the two operating modes. The instance type field says that distributed represents distributed mode, requiring a gtm, coordinator and data nodes, while centralized represents centralized mode. The same table describes a gtm with a single master and any number of slave IPs, and coordinators with master and slave IPs, where the number of slaves is an integer multiple of the masters and node names are generated automatically, deploying nodes-per-server nodes on each IP.

System requirements and the dependency install

The build requirements are stated before anything else, and they narrow who can realistically try this.

Memory is 8G RAM minimum. The supported operating systems are TencentOS 2, TencentOS 3, OpenCloudOS 8.x, CentOS 7, CentOS 8 and Ubuntu 18.04.

The dependencies come as a single package manager line, with a yum variant and an apt variant.

yum -y install git sudo gcc make readline-devel zlib-devel openssl-devel uuid-devel bison flex cmake postgresql-devel libssh2-devel sshpass  libcurl-devel libxml2-devel

apt install -y git sudo gcc make libreadline-dev zlib1g-dev libssl-dev libossp-uuid-dev bison flex cmake libssh2-1-dev sshpass libxml2-dev language-pack-zh-hans

The list tells you the shape of the thing. bison and flex mean the SQL grammar and the scanner are generated at build time, cmake is the build driver underneath the make wrapper, and postgresql-devel on the yum side pulls in the PostgreSQL server headers, which is the clearest single indicator of how closely this code tracks PostgreSQL internally.

Note also the last entry in the apt line, language-pack-zh-hans. This is a project with Chinese origins and a Chinese upstream, and the tooling reflects that.

A dedicated user, because the build wants one

The next documented step is to create a user called opentenbase, and the reason is visible in the sequence: the build is done as that user, in that user's home directory, with a data directory under it.

The steps are a directory, a user, a password, a group and a sudo rule.

# 1. Make directory /data
mkdir -p /data

# 2. Add user 
useradd -d /data/opentenbase -s /bin/bash -m opentenbase # add user opentenbase

# 3. Set password
passwd opentenbase # set password

# 4. Add users to the wheel group
# For RedHat
usermod -aG wheel opentenbase
# For Debian
usermod -aG sudo opentenbase

The fifth step is the one to notice. You enable sudo permissions for the wheel group through visudo, and then uncomment the line for the wheel group and save. The comment in the script is explicit about the RedHat and Debian difference, because the group name differs, and the visudo step is where the package management rights for the build user are granted.

Running the database as its own unprivileged user with its own home and data directory is standard practice, and doing it this early in the document means every later command, the compile, the installation, the cluster creation, runs in the same identity.

Compiling with configure and make

The build is the classic PostgreSQL shaped sequence, with the flags doing the interesting work.

su - opentenbase
cd /data/opentenbase/
git clone https://github.com/OpenTenBase/OpenTenBase

export SOURCECODE_PATH=/data/opentenbase/OpenTenBase
export INSTALL_PATH=/data/opentenbase/install/

The configure line enables user switching, the libxml and OpenSSL and ossp-uuid support, and compiles with debug symbols.

./configure --prefix=${INSTALL_PATH}/opentenbase_bin_v5.0 --enable-user-switch --with-libxml --disable-license --with-openssl --with-ossp-uuid CFLAGS="-g"
make clean
make -sj
make install

Then the contrib tree, which is where the cluster tooling and the signature helper live, and the helper is made executable before the contrib build because the build depends on it.

chmod +x contrib/pgxc_ctl/make_signature
cd contrib
make -sj
make install

The Makefile at the repository root is worth a mention because it is a compatibility shim with history in it. The comments explain that the Postgres-XL make files exploit features of GNU make that other makes do not have, and the file scans the PATH for gmake, gnumake or make, verifies that the candidate is GNU make, and refuses to continue with an error saying you must use GNU make to build Postgres-XL. The message still names Postgres-XL, which dates the code.

Packaging the install and describing the cluster

After the build, the installation directory is packaged into a tarball that the cluster tool can consume. The directory holds bin, include, lib and share, and the archive is created with tar zcf over the contents.

[opentenbase@VM-32-23-tencentos opentenbase]$ cd ${INSTALL_PATH}
[opentenbase@VM-32-23-tencentos opentenbase]$ ls
bin  include  lib  share
[opentenbase@VM-32-23-tencentos opentenbase]$ tar zcf opentenbase-5.21.8-i.x86_64.tar.gz *

If you obtained an RPM instead of building from source, the README gives the equivalent: rpm2cpio with cpio -idmv extracts the package into the current directory, and you package the resulting bin, include, lib and share tree into a tar.gz in the same way.

The environment setup before any of that is three exports.

PG_HOME=${INSTALL_PATH}/opentenbase_bin_v5.0
export PATH="$PATH:$PG_HOME/bin"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$PG_HOME/lib"
export LC_ALL=C

The cluster itself is created with the OPENTENBASE_CTL tool, and the example topology is one GTM, one coordinator and two data nodes. You generate the configuration template by running the tool and entering its prepare config command, which creates an opentenbase_ctl directory in your home with an opentenbase_config.ini template you fill in with the node information. The table of fields describes the instance name and its allowed characters, the instance type of distributed or centralized, and the software package path, which should be absolute.

One operational note deserves flagging. The preparation step offers disabling SELinux and firewalld as an optional step, with the commands given. That is convenient on a dedicated node and is not something to do on a host that is doing anything else.

Where OpenTenBase is the wrong tool

The first case is a workload that does not need distribution. If one PostgreSQL instance with its own extensions handles your volume, then a coordinator, data nodes and a GTM are three extra failure domains you now own, and the SQL surface you can use narrows to an extended subset of the standard rather than everything PostgreSQL offers. The README is explicit that it supports an extended subset, which is a limitation however generous the extension.

The second case is a team without a DBA or a platform engineer. The documented path is a source compile, a package tarball, a hand written configuration file describing master and slave IPs for the GTM and the coordinators, and a cluster started with a tool you invoke yourself. That is a database you operate like a product, not a service you install.

The third case is anything that needs a standard licence. The licence field is a custom one that GitHub cannot classify, and the terms live in LICENSE.txt in the repository root, alongside COPYRIGHT and Notice files. There is no MIT or Apache text to point at, and a procurement or compliance review will need to read that file rather than a standard identifier.

The alternative to weigh it against is plain PostgreSQL with the same drivers and the same SQL, and the difference in approach is the whole point of the project. PostgreSQL is one process owning one storage; OpenTenBase fragments a query across data nodes, coordinates it in a coordinator and manages distributed transactions through a separate node. You are choosing a distributed execution engine over a single node, and the question is whether your data and your traffic justify that.

Editorial conclusion

OpenTenBase suits an organisation that needs transactional and analytical queries across sharded data in one system, that already speaks the PostgreSQL wire protocol and driver ecosystem, and that has the operations capacity to run a multi node cluster. It does not suit a small team that wants a database to install in an afternoon, since the documented path is a source build followed by a hand written cluster configuration, and it does not suit anyone who needs a standard open source licence without reading LICENSE.txt. Verify first which version you will run against your PostgreSQL drivers, and read the opentenbase_config.ini field descriptions before you design the topology.

Frequently asked questions

Official sources

1. Issues
2. OpenTenBase/OpenTenBase on GitHub
3. Project website
4. README
5. Releases