Fossilize: Recording Vulkan Pipeline State So It Can Be Replayed Without the App

Persistent Vulkan objects, recorded as CreateInfo

Fossilize is a library and a Vulkan layer for serializing various persistent Vulkan objects, which typically end up in hashmaps. The supported types are listed precisely: VkSampler for the immutable samplers in set layouts, VkDescriptorSetLayout, VkPipelineLayout, VkRenderPass, VkShaderModule and VkPipeline for both compute and graphics. The mechanism is that the CreateInfo structs for these objects can be recorded and replayed, which means the objects are described rather than recreated blindly. That distinction is the whole value. A pipeline cache stores a compiled binary keyed by a hash of the inputs, and it is opaque and device specific. Fossilize instead stores the inputs themselves, so the same recorded state can be used to build fresh pipelines on devices that have never seen it, and it can be replayed when the compiled form is not usable.

The three use cases the project sets out

The goals section lists three, and they explain the design. The first is internal engine use: extend the notion of VkPipelineCache to also include these persistent objects, so they can be created at load time rather than declared up front manually. The idea is that the serialized cache could be shipped, and applications can then assume all persistent objects already exist. The second is a Vulkan layer that can capture this cache for reproduction when errors occur before a conventional capture is possible. The third is to serialize state in the application once and replay it on N devices to build up VkPipelineCache objects without having to run the application. That third case is the interesting one for driver and hardware vendors, because it removes the need to run a game at all to produce a pipeline cache for a new device.

A .foz database built to survive a cut-off write

The serialization format is described as an overall binary database format containing deflated JSON or deflated varint-encoded SPIR-V with light compression, and the database is a bespoke format with the extension .foz. The design constraint that shapes it is survival of abrupt truncation. Writes to the database can be cut off abruptly because of external instability issues, which happens when capturing a real application in a layer that the application does not know about. That is a very specific threat model, and it explains why the format is bespoke rather than a sequence of files: a partial write has to leave a database that can still be opened and used. The details of the archive format live in fossilize_db.cpp. The JSON inside represents the various Vk*CreateInfo structures, and small data blobs such as specialization constant data are encoded in base64.

Handles become hashes, and the app resolves them

The reference scheme inside the JSON is worth understanding because it determines what your application has to supply. When one CreateInfo refers to another Vulkan handle type, such as pImmutableSamplers in a VkDescriptorSetLayout or a VkRenderPass in a VkPipeline, a hash is used rather than the handle itself, with 0 representing VK_NULL_HANDLE and anything else representing a key. When recording or replaying, a mapping from and to real Vulkan object handles must be provided by the application so that the key-based indexing scheme can be resolved to real handles. In other words, Fossilize does not own your object lifetimes. Your application records the structure, and your application decides which live handle each hash refers to, which is what allows the same archive to be replayed against a different device with different handle values.

Recording: a background thread and a database

The sample API shows the recording side, and the shape of it is that a recorder is initialised against a database and then asked to record individual objects as you create them.

#include "fossilize.hpp"
#include "fossilize_db.hpp"

void create_state()
{
    auto db = std::unique_ptr<Fossilize::DatabaseInterface>(
            Fossilize::create_database("/tmp/test.foz", Fossilize::DatabaseMode::OverWrite));
    if (!db)
        return;

    Fossilize::StateRecorder recorder;
    recorder.init_recording_thread(db.get());

The comment in the sample points at a rule that is easy to get wrong: if you use a database, you cannot serialize later with recorder.serialize(), because the recorder interface calls db->prepare() for you and the database method is what you want if you intend to replay the archive in fossilize-replay or something similar. Recording an individual object is a single call, and the sample shows the descriptor set layout being recorded with its create info, with the same done for render passes, pipelines, shader modules and samplers as necessary.

Replaying: you implement the creator interface

The replay side is a set of interface methods you implement, one per object type, each receiving a hash and the CreateInfo struct and returning a real handle. The sample shows the descriptor set layout case.

struct Device : Fossilize::StateCreatorInterface
{
    bool enqueue_create_descriptor_set_layout(Hash hash,
                                              const VkDescriptorSetLayoutCreateInfo *create_info,
                                              VkDescriptorSetLayout *layout) override
    {
        VkDescriptorSetLayout set_layout = populate_internal_hash_map(hash, create_info);
        *layout = set_layout;
        return true;
    }

The method is named enqueue rather than create, and the comment explains why: it can queue the work up for threaded creation, which is useful for pipelines, since pipeline compilation is the expensive part. The interface also carries notification and synchronisation hooks, notify_replayed_resources_for_type and sync_threads, which are where a threaded implementation joins its queued tasks. The docs note that create_info persists as long as the replayer exists, and that dummy values can be used in the layout output if the create info does not need to be used verbatim.

The varint encoding of SPIR-V

SPIR-V blobs are stored with a custom varint encoding rather than as raw words or as compressed archives. The scheme encodes every 32-bit SPIR-V word by encoding 7 bits at a time starting with the least significant bits, and the most significant bit in an encoded byte is set if another byte needs to be read for the same SPIR-V word. Each SPIR-V word therefore takes from one to five bytes. The reason this is worth doing is that SPIR-V is a word stream, so the low bytes of most instructions are small and compress poorly as fixed-width data, and the varint form exploits that while remaining deflatable afterwards. The implementation lives in varint.cpp and varint.hpp, and the deflate step sits on top of it, so the two compressions are complementary rather than competing.

Building it, and the submodule rules

The build supports GCC 4.8 or newer, Clang and MSVC 2013, 2015 and 2017 or newer. If rapidjson is not already bundled in your project, the submodules have to be checked out.

git submodule update --init

Otherwise you can set FOSSILIZE_RAPIDJSON_INCLUDE_PATH when building the library as part of your project, and FOSSILIZE_VULKAN_INCLUDE_PATH overrides the Vulkan header include paths. The distinction that saves time is between the CLI tools and the library: the CLI tools need the SPIRV-Tools and SPIRV-Cross submodules initialised, but if you are only building Fossilize as a library or a layer, one of the preset library-only options disables all those submodule requirements. The standalone build uses CMake workflows and presets.

cmake --list-presets=all .
cmake --workflow --preset <preset-name>

To link it into another project there is a two-line CMake snippet, add_subdirectory with EXCLUDE_FROM_ALL followed by target_link_library against the fossilize target, and an android_build.sh script builds the layer for armeabi-v7a and arm64-v8a.

Editorial conclusion

Fossilize fits engine and driver work where creating pipelines dominates startup and the same pipeline state has to exist on many devices, and where you need a reproducible capture of a crash. It is the wrong tool for a normal application with a handful of pipelines, since the recording and replay plumbing costs more than the problem. It is also the wrong tool if your objects change at runtime in ways the CreateInfo structs do not capture. Before you adopt it, confirm that your Vulkan headers and the rapidjson submodule are in place and try the library-only preset first.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. ValveSoftware/Fossilize on GitHub