Microsoft PICT for Pairwise Combinatorial Testing

Why PICT targets pairwise test suites

PICT solves the test design problem created by products with many independent configuration choices. A partition and volume creation interface, for example, may have Type, Size, File system, Format method, Cluster size, and Compression parameters. The README states that these values produce thousands of possible combinations, which can be impractical to test manually or exhaustively.

PICT focuses on pairwise coverage. Each generated row represents one test configuration, and the tool selects rows so that pairs of values across parameters are covered. A row can cover many pairs at once. The documented example includes values such as Type: Single, Span, Stripe, Mirror, RAID-5, Size: 10, 100, 500, 1000, 5000, 10000, 40000, and Compression: On, Off. The model is an input description rather than an executable test script. PICT therefore generates candidate configurations; it does not create assertions, invoke the application under test, or determine whether each configuration passes.

That boundary matters when choosing the tool. PICT is aimed at test designers and engineers who have already identified parameters and useful equivalence classes. It is less suitable for interfaces whose correctness depends on workflows, timing, state transitions, or interactions among more than two parameters, unless those requirements are represented separately in the test process.

PICT model files and generation flow

The documented workflow starts with a model file that lists interface parameters and their allowed values. Each parameter has a limited domain, either because the interface permits a fixed set of values or because the tester has applied equivalence partitioning. Once the model is prepared, PICT reads it and generates a compact set of parameter value choices.

The repository layout separates the implementation into api, cli, clidll, api-usage, and clidll-usage. The api directory contains the core combinatorial engine, including source files for combinations, derivation, exclusion, models, parameters, tasks, and work lists. The cli directory contains the command line front end and parsing code. The clidll project packages the command line client as a Windows DLL for in-process use. The usage projects show how the engine or DLL can be consumed.

This separation gives PICT more than one integration path. A team can use PICT.EXE as a command line tool, study the core API example, or use the Windows DLL example. The README does not document a Python binding, a REST service, or a hosted execution environment. The benchmark harness is also separate from normal generation. It repeatedly runs PICT with different random seeds and reports the smallest generated suite for each benchmark, which is useful for studying suite size but does not replace domain-specific validation.

PICT supports model constraints such as exclusions through its implementation, although the README excerpt does not provide a complete syntax reference for every model feature. The detailed documentation is located in doc/pict.md and the methodology is documented at pairwise.org.

Building PICT on Linux and running the tests

PICT documents a CMake-based Linux build. The commands assume CMake and a C++ toolchain are already installed. The build produces the project in the build directory, and the documented test sequence runs CTest with verbose output.

The following commands configure a Release build, compile it, and run the tests:

> cmake -DCMAKE_BUILD_TYPE=Release -S . -B build
> cmake --build build
> pushd build && ctest -V && popd

The configure command names the source directory as `.` and the output directory as `build`. The second command compiles the targets defined by the repository. The third command runs the CTest suite and returns to the original directory. The README describes Perl as a requirement for the test folder on Windows, but the Linux instructions shown here invoke CTest directly.

For a Windows build, the README directs users to open pict.sln in Visual Studio 2022 and install the VC++ build tools. The root `_build.cmd` script builds Debug and Release configurations from the command line. `_test.cmd` runs the tests, producing `dbg.log` or `rel.log` for the corresponding build. Those logs are intended to be compared with committed baselines. The documentation warns that randomized output can differ between runs, so a difference requires explanation rather than an automatic assumption of regression.

The repository Makefile exposes `pict`, `pict-benchmark`, and `libpict.so` as build targets. It also provides `all`, `test`, and `clean`. The README does not document a package-manager installation command, so building from the repository is the documented first-use path.

First generated configuration with PICT.EXE

The README identifies the most recent `pict.exe` as available from the GitHub releases page. For a first real use, a tester can prepare a small model containing a few interface parameters and run the executable against that file. The supplied documentation does not include a complete model-file example with every required formatting detail, so it points to `doc/pict.md` for detailed syntax and to `doc/sample-models` for sample models.

The repository's container workflow provides a documented way to package PICT and run a bundled sample. The image is built with the following Make target:

make image-build

That command creates an image tagged `pict:latest`. The documented sample invocation mounts the repository sample-model directory and runs a specific model:

make image-run

The Makefile expands this target to a Podman command that mounts `./doc/sample-models` at `/var/pict:Z` and runs `create_volume.txt`. For a local model, the README gives this form:

podman run -it --rm -v ./<local-dir>:/var/pict:Z pict:latest <your-model-file> [<pict-options>]

The angle-bracket placeholders in that command represent the reader's own directory, model file, and optional PICT options. After running a model, the meaningful output is the generated suite of rows. Review the rows for coverage and plausibility rather than treating generation as proof that the test model is complete.

PICT benchmark harness and reproducibility controls

The repository includes `pict-benchmark`, a native harness for running PICT repeatedly with different random seeds. By default, it runs the six standard pairwise.org scenarios referenced by the SigmaZone benchmark page for 1,000 runs:

./pict-benchmark --runs 1000

The harness can target built-in or custom benchmark notations. The documented examples use notation strings that describe parameter domains:

./pict-benchmark --runs 250 --benchmark 3^13 --benchmark "4^15 3^17 2^29"
./pict-benchmark --runs 500 --benchmark "6^20"

The harness also supports `--seed` to reproduce the randomized trial sequence and `--list` to print the built-in benchmark set. Every trial writes debug information to stderr, including the model shape, seed, and generated row count. This is useful for comparing repeated generation outcomes, but it is a benchmark utility rather than a general test-management interface. The documentation does not say that the smallest suite reported by the harness is always globally minimal.

There is also a practical maintenance detail. The repository is not archived and its last push was on 2026-10-14. However, the listed releases are v3.7.4 from 2022-03-31, v3.7.3 from 2022-02-18, and v3.7.2 from 2021-02-24. The source activity therefore does not align neatly with a recent sequence of published releases. Teams should inspect the current `main` branch, build scripts, and platform behavior rather than assuming that a recent commit guarantees a fresh release package.

Limits of PICT's pairwise coverage model

The central limitation is explicit in the method. Pairwise testing covers pairs of parameter values, so it does not guarantee coverage of every higher-order interaction. If failures occur only when Type, Size, and Compression have one particular combination, a pairwise suite may omit that row. PICT is appropriate when pairwise coverage is an accepted test objective, but it cannot replace analysis of requirements that depend on three or more interacting parameters.

The quality of the generated suite also depends on the model. A weak equivalence partition can make a broad-looking test set miss the values that matter operationally. The tester must decide whether a domain is complete, whether values are mutually appropriate, and whether any combinations are impossible or unsafe. The README points to detailed documentation for model syntax, but the excerpt does not establish how PICT handles application state, setup order, cleanup, assertions, or test data generation.

The tool should not be used as the only control for performance, security, or reliability testing. Its output is a set of parameter combinations, not evidence that a system behaves correctly. Likewise, random generation can produce different suites between runs. For controlled comparisons, the benchmark harness exposes a seed option, while ordinary test execution requires the team to manage generated files and review them according to its own test process.

A real alternative is exhaustive enumeration. For a small domain, generating every Cartesian-product row gives complete configuration coverage, including higher-order interactions, at the cost of a larger suite. PICT is the better fit when the domain is too large for exhaustive testing and pairwise coverage is the stated goal. The choice is therefore a coverage strategy decision, not simply a choice between two command line programs.

License, platform choices, and adoption boundaries

The repository identifies its license as Other and points to `LICENSE.TXT`. Because GitHub cannot classify the custom license in its standard categories, teams should read that file before embedding PICT in a build or distributing binaries. This review does not interpret the license terms or provide legal advice. The practical point is that the repository contains a project-specific license file rather than a license name that can be assumed from the topic or language.

PICT offers several platform paths, but they are not identical. `pict` and `pict-benchmark` are native command line targets in the Makefile. `libpict.so` is a shared library target. `pict.sln` and `_build.cmd` support the Windows development path, while `clidll` provides a Windows DLL packaging route. The container workflow uses `podman`, but the README does not claim that this is the only supported container engine or that every platform has an official prebuilt image.

The best adoption fit is a C++ project or a test team that already models interfaces as finite parameters and wants a generated pairwise suite. A team that needs orchestration, test execution, reporting, or higher-order coverage should check whether PICT can be one component of a larger toolchain. The first verification should be a small model reviewed against known requirements, followed by a build using the documented CMake or Windows commands and a comparison of generated rows with expected constraints.

Editorial conclusion

PICT is a practical choice for software teams that can express important interfaces as finite parameters and need a repeatable pairwise test suite. It is not a substitute for requirements analysis, higher-order interaction testing, or exhaustive testing where every combination is operationally necessary. Before adoption, verify that the model captures the right equivalence partitions, exclusions, and domain values, then inspect the generated rows against representative workflows. The project is not archived, and its last push was on 2026-10-14, although the listed releases are from 2022, so maintenance expectations should reflect that mixed history.

Frequently asked questions

Official sources

1. Issues
2. microsoft/pict on GitHub
3. README
4. Releases