Elixir: a cross referencer that indexes every release of a C project

Elixir indexes blobs, not trees

Elixir is described as a source code cross referencer inspired by LXR, and its stated purpose is to index every release of a C or C++ project, such as the Linux kernel, while keeping a minimal footprint.

The storage decision is the interesting part. It uses Git as the source code file store and Berkeley DB for the cross reference data, and internally it indexes Git blobs rather than trees of files. The reason given is to avoid duplicating work and data. The same file content recurs across thousands of tags, so a blob indexed once and referenced many times is what keeps the footprint down.

The data structure is described as straightforward and reminiscent of older LXR releases, chosen to keep queries simple and fast. That matters for the audience: this is a tool for looking things up constantly, and the design goal is a fast answer rather than a rich index.

The public instance at elixir.bootlin.com shows it running against real projects. The repository itself holds the Python implementation, a `script.sh` shell layer, a `projects/` directory, `samples/`, templates, static assets, tests in `t/`, and the wsgi.py entry point.

Three layers, and the documentation says to test each one

The architecture is a stack with a clear order, and the documentation makes a point of it: when installing the system you should test each layer manually and make sure it works correctly before moving on to the next one.

The bottom layer is `script.sh`, a shell script that provides commands to interact with Git and other Unix utilities. It is the layer that knows how to read a Git repository and hand back content.

Above it sit the Python commands, which use the shell script's services. `query.py` gives access to the annotated source code and identifier lists, and `update.py` creates and updates the databases.

On top of that sits `web.py`, the web interface, which uses the query interface to generate HTML pages and to answer REST queries. So the same query machinery serves both the human-facing pages and the API, and the shell script is the only component that talks to Git directly.

That layering is what makes the manual test sequence sensible. Listing tags through the shell script, building the database with the update command, and only then querying are three separate checkpoints, and a failure in any of them is much easier to localise when they are exercised in that order.

Installing Elixir is a sequence of small verified steps

The manual installation path is long but every stage is checkable. On Debian the dependencies come from apt:

sudo apt install python3-pip python3-venv libdb-dev python3-dev build-essential universal-ctags perl git apache2 libapache2-mod-wsgi-py3 libjansson4

The requirements list behind that command names Python 3.8 or later, Git 1.9 or later, the Jinja2 and Pygments libraries with Pygments 2.7 or later, Berkeley DB and its Python binding, Universal Ctags, Perl for non-greedy regexes and automated testing, and Falcon with mod_wsgi for the REST API.

Then the project is cloned and a virtual environment is created with the requirements installed:

git clone https://github.com/bootlin/elixir.git /usr/local/elixir/
python -m venv /usr/local/elixir/venv
. /usr/local/elixir/venv/bin/activate
pip install -r /usr/local/elixir/requirements.txt

Two environment variables tell Elixir where things live: `LXR_REPO_DIR` for the Git repository directory and `LXR_DATA_DIR` for the database directory, and the documentation suggests appending the exports to `/etc/profile` and sourcing it.

The first test is the cheapest one to run and worth running before anything expensive:

cd /usr/local/elixir/
./script.sh list-tags

The database build is where the real time goes

The update command takes a thread count, and the documentation is blunt about the cost: generating the full database can take a long time. It reports that indexing 1800 tags in the Linux kernel took about 15 hours on a Xeon E3-1245 v5.

That figure is the single most important planning input in the project. It means a full kernel index is an overnight job on one machine, and it means the tool is not interactive to set up.

The documentation offers a way out for testing: tweak the script, for example by limiting the number of tags with a head option, so the update and query commands can be tried cheaply. It also suggests an even smaller experiment, creating a new Git repository and adding a single tag instead of using the official kernel repository, which is described as very large.

The build command itself is short:

. ./venv/bin/activate
./update.py <number of threads>

Once the database exists, the queries are quick to check. The second test in the documentation runs an identifier lookup and a file lookup, and notes that the tag can be replaced with any other:

python3 -m utils.query ident v4.10 raw_spin_unlock_irq C
python3 -m utils.query file v4.10 /kernel/sched/clock.c

The last line of that section is a warning worth keeping: don't forget to activate the virtual environment.

Serving it means Apache, SELinux labels and a systemd tweak

The web interface is a CGI script meant to be called from a web server, and it supports indexing multiple projects, so it expects a different variable, `LXR_PROJ_DIR`, pointing at a directory laid out with one subdirectory per project, each containing `data` and `repo`. The other two variables are generated from that structure when the query command is called.

The Apache configuration is a file swap. On Debian and Ubuntu you replace `/etc/apache2/sites-enabled/000-default.conf` with `docker/000-default.conf`; on RedHat and Centos you edit `/etc/httpd/conf.d/elixir.conf` instead. Then the server restarts.

The SELinux section exists because of a specific failure. With systemd and SELinux enabled, the httpd server can only visit limited directories, and if your data directory is not among them you get a 500 status code. The fix is a context change:

chcon -R -t httpd_sys_rw_content_t /path/elixir-data/

You can confirm the labels with `ls -Z /path/elixir-data/` and read the audit trail with `audit2why -a | grep httpd | less`.

The systemd wrinkle is separate. By default the Elixir error log goes to `/tmp/elixir-errors`, but systemd enables PrivateTmp by default, so the real directory ends up under `/tmp/systemd-private-...`. Disabling it needs one attribute in the unit, `PrivateTmp=false`.

For other servers, the documentation notes that nginx and lighthttpd may not support WSGI and might need a separate WSGI server such as uWSGI, and it links a guide for that pairing.

A REST API on top of the same query engine

Once httpd is configured, the same index is reachable over HTTP, which is what makes Elixir useful in a scripted workflow rather than only in a browser.

The ident query takes the form `/api/ident/<Project>/<Ident>?version=<version>&family=<family>`, and the documentation gives a concrete call against the barebox project:

curl http://127.0.0.1/api/ident/barebox/cdev?version=latest&family=C

The response body is JSON, and the structure shown in the documentation starts with a `definitions` array whose entries carry a path, a line number and a type.

That request shape is worth pausing on. The project name and the identifier are path segments, while the version and the language family are query parameters, so a script can ask the same question against many tags and many projects without a session. The `family` parameter ties the query to a language, which is what lets the tool index a mixed C and C++ tree.

The alternative approach to the same problem is a code search engine over the current tree, and the difference in approach is exactly the tag axis. A search over HEAD answers what is true now. An Elixir query against a tag answers what was true at that release, across the whole history the Git mirror holds, which is why the project goes to the trouble of maintaining a stable remote and a history remote for the old Linux versions.

Editorial conclusion

Elixir is for maintainers of large C codebases who need identifier history across releases, for example to find when a function was introduced or which file touched a symbol in a given tag, and the tag-per-release model is what makes it useful. Skip it for a single project with a handful of commits, since the setup cost of a Git mirror, a Berkeley DB build and a full index run is not justified. Before you start, budget for the index: the documentation reports that generating the full database for the Linux kernel took about 15 hours on a Xeon E3-1245 v5 for 1800 tags, so use the head option to test the update and query commands first, and confirm your Git version is 1.9 or later and Python 3.8 or later.

Frequently asked questions

Official sources

1. bootlin/elixir on GitHub
2. Issues
3. License: AGPL-3.0
4. README