aruba: testing command line programs from Ruby

A harness for programs you talk to over a pipe

The description is one line and it is the whole product: test command-line applications with Cucumber-Ruby, RSpec or Minitest.

The problem it solves is mundane and universal. A command line tool has a behaviour that people care about: it prints things, it writes files, it exits with a status, it takes flags and it reads standard input. Testing that usually means writing a shell script, and shell scripts do not have assertions, do not run on a developer's laptop and on the continuous integration server in the same way, and do not tell you which case failed.

aruba turns the command line program into something a test can drive. The tests run the program, capture its output, inspect the working directory afterwards, and make assertions about all of it, in the same test framework you already use for the rest of the project.

Three frameworks are supported, and that is the design decision worth noting. Most people reach for one of them, and a harness that only worked with Cucumber would be useless to the majority of Ruby developers. Supporting all three means the harness is a library rather than a plugin, and the readme's guidance reflects that, pointing newcomers at the documentation of whichever framework they use.

The project sits under the Cucumber organisation, with a support badge, a code of conduct and a contributing guide, and the maintainer list names a single person. The gem is MIT licensed, which is the right licence for a test harness, since almost every project that uses a command line tool eventually wants to use this one too.

Three levels of documentation, aimed at three different readers

The usage section of the readme is organised by who is reading it, which is a good sign for a project with a mixed audience.

For a user getting started, the documentation is the features directory in the repository itself, which is unusual and rather good. The features of a behaviour driven tool are executable documentation, so a directory of them is both the tutorial and the specification, and it cannot drift from the behaviour because it is the behaviour.

For a user of a Ruby testing framework, the readme points at the framework's own documentation and explains why. If you are new to the Cucumber ecosystem, visit the project documentation site, which also includes information about how to write feature files in the Gherkin language. If you want to use Aruba with RSpec, visit their documentation site. And for Minitest, the documentation is at the link given.

Each of those entries is short and exists to prevent a specific confusion, which is that you need to know the framework before you can use the harness. The readme is not trying to teach you RSpec, it is telling you where RSpec is taught.

For a developer extending Aruba, the full API documentation is on the RubyDoc site, and the repository carries the usual set of supporting material for a gem of this age: a gemspec, a manifest file, a Rakefile, a Rubocop configuration with a todo file beside it, a documentation folder, a simple coverage configuration, a yard options file for the generated API documentation, and an appraisals file for testing against multiple dependency versions.

Installing it is a Gemfile line

The install is the standard two ways, and the version constraint is worth copying exactly.

gem 'aruba', '~> 3.0'

bundle

Or installed directly:

gem install aruba

The constraint form allows any 3.x update below 4.0, which is what you want for a test harness, because a harness that changes behaviour between patch releases will break your build for no good reason.

The repository carries a gemfiles directory, which is the conventional place for the alternative dependency sets used to test a gem against different versions of what it depends on, and the appraisals file at the root is the tool that generates them. The combination means the project tests itself against a matrix of dependency versions rather than only against the newest, which is what you would hope from a harness that other people's projects depend on.

The Dockerfile and its Podman variant at the root are for running the test suite in a container, and the Dockerfile is more interesting than most test Dockerfiles because of how it is put together.

The version matrix, and an honest note about Windows

The support section is short, precise, and includes a sentence that will save you a wasted afternoon.

Aruba is supported on Ruby 3.2 and up, and tested against CRuby 3.2, 3.3, 3.4 and 4.0, and JRuby 10.0. It is supported on and tested with Cucumber version 11 and 12.

And on operating systems, the readme says Aruba is fully tested in CI on Linux and MacOS. On Windows, only RSpec tests and some of the Cucumber scenarios are run, so your mileage may vary, and full Windows support is a work in progress.

That last point deserves emphasis, because the honest framing is unusual. Testing a command line application on Windows is genuinely hard, since the shell, the path handling and the line endings all differ, and a harness that claims to test command line programs has to deal with all three. Saying that only part of the suite runs there, and saying it plainly, is better than shipping a green badge that means something different on one platform.

So the practical reading is: if your tool runs on Linux and macOS, aruba is fully supported. If it runs on Windows, the harness works but the suite is partial, and you should check what your own tests actually exercise there.

A release policy that deliberately breaks the rules

The release policy section is the most interesting paragraph in the readme, and it is short enough to quote almost in full.

The project uses Semantic Versioning 2.0.0. The gem depends on rubygems to ensure correct dependency versions, so dropping support for older dependencies and Ruby versions will not result in a major version bump.

That is a deliberate exception to the usual promise, and the reason given is mechanical rather than philosophical. Because the gem relies on rubygems to resolve the correct dependency versions, an application that installs a newer major version of aruba may still get older versions of the things aruba depends on, and the old code keeps working. So a major version bump is not needed to signal the change, and making one anyway would break applications that did not need to break.

For a user, that means you cannot rely on the major version to tell you whether a Ruby version you use is still supported. You have to read the release notes. For a maintainer of a project that depends on aruba, it means your automated upgrade will not warn you, so the readme's supported Ruby list is the thing to check when you plan a Ruby upgrade.

The branch policy is in the same spirit and equally explicit. Development happens in the main branch, which currently targets bug fixes on the 3.0 release. If necessary, maintenance of the 2.4.x releases takes place in a 2-4-stable branch. Older releases will not receive updates, and stable branches will not be created until absolutely necessary.

The Docker image, and why it is built the way it is

The Dockerfile is a small masterclass in keeping a test image fast, and the comments explain the reasoning.

It starts from a Ruby 4.0 base image, creates a user, makes a working directory owned by that user, and switches to it, so that the rest of the build runs as a non-root user. That matters for a test image, because a test that creates files and runs programs is exactly the sort of thing that behaves differently as root.

The dependency install is where the care is. Only the files needed for the dependency resolution are added first, which is the Gemfile, the gemspec and the manifest file, plus two specific files, the version file inside the library and the executable. Then the dependencies are installed. Only after that is the full source code added.

The point of the ordering is layer caching. The layer that installs the gems only changes when the Gemfile changes, so the expensive step is reused on every code change, and the cheap layer with the source is rebuilt every time. The comment at the top of the file shows the run command, and explains that mounting the working directory into the container makes it pick up changes to the code, so you can edit and rerun the tests without rebuilding the image.

docker run -v $PWD:/aruba --rm -it test-aruba-docker:latest bash

The Podman variant exists because the two tools differ in a few small ways, and shipping both is a courtesy to the people whose platform is the one you did not pick.

Editorial conclusion

aruba suits a team that ships a command line tool and wants its behaviour tested the way an application is tested, because it removes the shell scripting that usually stands in for a test harness. It is a poor fit if you want to test a program that only runs on one platform, since the Windows support is explicitly incomplete, and a poor fit if you need to pin an old Cucumber, because the supported range is narrow and moving. Before you upgrade a project that depends on it, read the release policy, since dropping an old Ruby or dependency version is deliberately not treated as a breaking change and will not arrive as a major version.

Frequently asked questions

Official sources

1. cucumber/aruba on GitHub
2. Issues
3. License: MIT
4. README
5. Releases