VK-GL-CTS: the test suite your driver has to pass

A superproject that holds two test suites

VK-GL-CTS is the Khronos Conformance Testing Suite, and it originated from dEQP, the drawElements Quality Program. The repository contains tests for several graphics APIs: OpenGL, OpenGL ES, EGL, Vulkan and Vulkan SC. Structurally it is a superproject. The Vulkan CTS lives under the external/vulkancts directory with its own README, and the OpenGL and OpenGL ES CTS lives under external/openglcts with its own README as well. That distinction matters more than it looks, because a driver team working on Vulkan and a browser engine team working on OpenGL ES need different parts of this repository, and the shared scaffolding around them, the framework, the executor, the modules, the targets and the scripts, is what lets one project host both. The other top-level pieces are the android/ directory, the data/ directory, the doc/ directory and the execserver/ and executor/ directories.

Selecting what gets built, before the build eats your disk

Here is the practical warning the documentation leads with. When you configure the source for either the Vulkan or the OpenGL and OpenGL ES conformance tests, CMake generates build files that will, by default on desktop platforms, build every possible project binary, which may be undesirable because of the amount of time and disk space the build needs. The first way to avoid that is CMake's own target selection mechanism.

cmake --build BUILD_DIRECTORY --target deqp-vk

That builds only deqp-vk, the main Vulkan Conformance Tests binary. The second mechanism is a configuration option, SELECTED_BUILD_TARGETS, which if set to a non-empty value builds only the listed targets, separated by spaces, making a plain cmake --build behave as if the corresponding --target options had been passed. So a second configuration would list deqp-vk and deqp-vksc. The project then puts a caveat in bold: target subset selection may not have been thoroughly tested in all environments and situations, and it does not replace the instructions given for the purposes of creating a conformance submission. In other words, this is a local convenience, not a supported way to produce a submission.

The two wikis, and the Khronos member split

Documentation is split along the same member/non-member line as the rest of the Khronos process. There is a VK-GL-CTS wiki for Khronos members hosted on the Khronos GitLab tracker, and a VK-GL-CTS wiki for non-Khronos members on GitHub. The README points contributors at CONTRIBUTING.md for how to contribute. For anyone coming from a graphics driver background, the important thing to know is that the interesting material is often not in this repository at all, because the test descriptions, the specification references and the submission rules live in the wiki and in the specification documents themselves. The repository is the executable form of that specification, and the tests are generated partly from those documents, which is why the documentation and the code stay in step with the API revisions they cover.

Reading the results: qpa logs and embedded images

The output format has one feature worth knowing about in advance. The .qpa logs generated by the conformance tests may contain embedded PNG images of the results, and those can be viewed with scripts/qpa_image_viewer.html by opening the file with a web browser and following its instructions, or using the Cherry tool, which comes from the Android platform tree. This is a real convenience in practice, because a large share of graphics conformance failures are visual, a pipeline that produces the wrong colour or the wrong texture, and a failure that is purely a text diff is much harder to diagnose than one where you can look at the rendered image the driver produced. Being able to open the log in a browser and see the image, without installing a toolchain, is the difference between minutes and hours on that kind of bug.

Running the Vulkan video suites needs samples fetched first

The video decode and encode test suites need media that is not in the repository, and the documentation is explicit about the step. To run the Vulkan video decode test suite you need to run the script external/fetch_video_decode_samples.py before building and running any test suite such as the dEQP-VK.video.decode series, and it will download the video clips into external/vulkancts/data/vulkan/video. The encode side has its own equivalent, external/fetch_video_encode_samples.py, downloading into the same location, and it is needed before the dEQP-VK.video.encode series. The practical detail is the ordering. The download has to happen before the build, not before the run, because the test binaries are compiled against the presence of those sample files, and a build that happens first will not pick them up.

ANGLE for Android, and building the APK

For Android work the documentation covers building ANGLE for Android by following the instructions in the ANGLE repository, and then linking the resulting ANGLE shared object libraries into dEQP.apk with the --angle-path option, which causes the APK to use the ANGLE libraries for OpenGL ES calls rather than the native drivers. Two conditions apply. An ABI must be specified, and the directory structure containing the shared objects must match it so the build system can find the correct .so files. The example assumes the shared objects were generated into the Chromium out/Release directory and that the APK is generated with the arm64-v8a ABI, in which case you create that subdirectory and copy the lib*_angle.so files into it. The full build command then passes the SDK path, the NDK path, the ABI list and the angle-path to the Android APK build script. The repository carries the matching Android build files at the top level, an AndroidGen.bp and an AndroidKhronosCTSGen.bp.

Where this fits, and what it is not

The clear scope statement is that this is a conformance suite, not a library and not a driver. Nothing here is something you call from your application. The value of it is diagnostic and procedural: it is the definition of what correct behaviour is for these APIs, expressed as executable code, and it is the yardstick an implementation is measured against. For a driver developer, running the suite is the way to find the bugs that a conformance submission will find anyway, and finding them first is worth a lot. For an application developer, the suite is a source of test cases, and the qpa log format plus the embedded images make it usable even if you never intend to submit. What it is not is a testing framework for your own code. These tests exercise an implementation of a Khronos API, and they are enormous, and they assume a specific execution model with an executor and a test case tree that has no counterpart in a normal unit test suite.

Editorial conclusion

VK-GL-CTS fits a graphics driver engineer who needs to know whether an implementation is conformant, and a developer who wants to reproduce a conformance failure on their own stack, since the suite runs as ordinary command line binaries against a driver. It does not fit a developer who wants a graphics API to call, because nothing here is a library you link against, and it does not fit anyone planning to build the whole thing casually, because the default CMake configuration builds every possible project binary and the documentation suggests selecting a subset first. Before you start, read the README for the API you care about, because the Vulkan and OpenGL suites are separate trees with their own documentation, and treat the target selection mechanism as a convenience rather than a supported path, since the project warns it may not be thoroughly tested in all environments and does not replace the conformance submission instructions.

Frequently asked questions

Official sources

1. KhronosGroup/VK-GL-CTS on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases