cypress-visual-regression turns Cypress screenshots into pixel diffs

A plugin, not a test runner

cypress-visual-regression is a module for adding visual regression testing to Cypress, and it is worth being precise about what that means. It does not run tests, it does not replace Cypress, and it does not know anything about your application. It gives you one additional command, compareSnapshot, that takes a Cypress screenshot, compares it against a stored baseline image, and produces a pass or fail.

The comparison is done with pixelmatch, and the dependency list in package.json makes the whole stack visible. There are four runtime dependencies: chalk 5.6.2 for coloured console output, pixelmatch 7.2.0 for the image comparison itself, pngjs 7.0.0 for reading and writing PNG buffers, and sanitize-filename 1.6.4, which is there because snapshot names become file paths and therefore need sanitising before they are written to disk.

That is a small, comprehensible dependency surface, and each entry has an obvious job. The repository itself is a TypeScript project with src/, tests/, an eslint config, a prettier config, a .nvmrc and a CHANGELOG.md. The package version in package.json is 6.0.1, published as a package named cypress-visual-regression under the MIT licence, authored by Michael Herman. The default branch is master, and although the repository has no GitHub releases, the npm badge in the README points at the published version, so npm is where the release history lives.

The people who benefit are teams already running Cypress end to end tests who want a cheap way to catch the class of regression that assertions on the DOM cannot see: a shifted layout, a lost font, a changed colour, a broken image.

Installing it and wiring it into the config

Installation is one npm install:

npm install cypress-visual-regression

Then the plugin has to be configured in your cypress.config.js file, through the configureVisualRegression function called in setupNodeEvents:

const { defineConfig } = require('cypress')
const { configureVisualRegression } = require('cypress-visual-regression')

module.exports = defineConfig({
  e2e: {
    expose: {
      visualRegressionType: 'regression'
    },
    screenshotsFolder: './cypress/snapshots/actual',
    setupNodeEvents(on, config) {
      configureVisualRegression(on)
    }
  }
})

Two things in that snippet do the real work. The screenshotsFolder points Cypress at where actual screenshots are written, and the expose block carries the plugin configuration, which Cypress passes to the plugin through its exposed values mechanism.

The command itself has to be registered in the support file, cypress/support/e2e.js:

const { addCompareSnapshotCommand } = require('cypress-visual-regression/dist/command')
addCompareSnapshotCommand()

The TypeScript path is the same idea with imports. The config becomes cypress.config.ts using import syntax, the support file becomes cypress/support/e2e.ts with the same addCompareSnapshotCommand call, and the README asks for a specific tsconfig.json so that ts-node transpiles without type checking and emits ES2015 modules:

{
  "ts-node": {
    "transpileOnly": true,
    "compilerOptions": {
      "module": "ES2015"
    }
  }
}

Note that the import path points at cypress-visual-regression/dist/command rather than the package root, which tells you the command is a separate entry point from the plugin configuration.

Base mode, regression mode, and updating baselines on demand

The most important option is visualRegressionType, and the README draws attention to it. The value 'base' generates baseline images, and 'regression' compares the current screenshot to the base screenshot. The default shown in the options table is empty, which is another way of saying you have to choose.

That split is the whole workflow of a visual regression suite. You run the suite once in base mode to create the reference images, you commit those images to version control, and from then on every run is in regression mode, comparing what the application looks like now against what it looked like when the baseline was taken.

The README also documents an escape hatch for the case where a difference is expected. When running in regression mode, you can enable automatic base snapshot updates on mismatch with visualRegressionUpdateSnapshots, and it can be set as an environment variable alongside the run command:

CYPRESS_visualRegressionType=regression CYPRESS_visualRegressionUpdateSnapshots=true cypress run --spec 'cypress/e2e/**/*.cy.ts'

The same value can be passed through the Cypress CLI expose mechanism with cypress run --expose visualRegressionUpdateSnapshots=true.

That option is the sharpest tool in the box and the easiest to misuse. A run with it enabled overwrites the baselines, so a test that was supposed to catch a regression instead accepts whatever the application currently renders. Used deliberately, for example after an intentional redesign, it is exactly what you want. Left on in CI, it silently disables the entire suite, because the diff is always empty when the base is rewritten from the same run.

The options table marks visualRegressionUpdateSnapshots as defaulting to false, and the fact that it has to be set through Cypress expose values rather than in the config file makes it a little less likely to be switched on by accident.

The option table decides where your images live

The rest of the configuration is a small table of six variables, all prefixed with visualRegression and all set under expose in the config file.

module.exports = defineConfig({
  e2e: {
    screenshotsFolder: './cypress/snapshots/actual',
    expose: {
      visualRegressionType: 'regression',
      visualRegressionBaseDirectory: 'cypress/snapshot/base',
      visualRegressionDiffDirectory: 'cypress/snapshot/diff',
      visualRegressionGenerateDiff: 'always',
      visualRegressionFailSilently: true
    }
  }
})

visualRegressionBaseDirectory defaults to cypress/snapshots/base and is where the base snapshots are stored, and visualRegressionDiffDirectory defaults to cypress/snapshots/diff and is where generated image differences are written. Note that the example above sets both to cypress/snapshot paths without the s, which is a small inconsistency in the README worth noticing rather than copying blindly.

visualRegressionGenerateDiff controls the artefact, not the verdict. It can be 'fail', 'never' or 'always', and the default is 'fail', so differences are only written when the test is about to fail. Setting it to 'always' means you get a diff image for every screenshot whether or not it differs, which is useful when you want to eyeball rendering drift over time and wasteful otherwise.

visualRegressionFailSilently defaults to false and decides whether an error found in regression is thrown or returned as part of the result. With it false, the test fails loudly, which is what you want in CI. With it true, the failure is reported but not raised, which is the setting to reach for when you are triaging a new suite and do not yet trust the baselines.

Setting these globally is not the only way. Global defaults for the command can be overridden by passing options to addCompareSnapshotCommand, and the README lists the three groups you can override: cypress screenshot arguments, pixelmatch options, and the plugin configuration of errorThreshold and failSilently.

compareSnapshot takes a name, a threshold and options

The command you actually call in a test has three forms:

cy.compareSnapshot(name)
cy.compareSnapshot(name, errorThreshold)
cy.compareSnapshot(name, options)

The name is the name of the base snapshot file the actual screenshot is compared against, and it is the key that ties a test to its image. The errorThreshold is the threshold under which any image difference is considered a failed test, expressed in percentages, and its default is 0, which means any measurable difference fails.

The options form is the same override mechanism described earlier, letting you pass cypress screenshot arguments, pixelmatch options and the failSilently and errorThreshold values for that one call.

The global form of the same thing is:

const { addCompareSnapshotCommand } = require('cypress-visual-regression/dist/command')
addCompareSnapshotCommand({
  capture: 'fullPage', // cypress screenshot option
  errorThreshold: 0.5, // plugin threshold option
  pixelmatchOptions: {
    threshold: 0 // pixelmatch threshold option
  }
})

Two thresholds appear here and they are not the same thing. errorThreshold at 0.5 is the plugin's own measure of how much of the image changed, counted in percent. The pixelmatch threshold of 0 is a per pixel colour difference sensitivity, where 0 means any change at all in a pixel counts. Setting the pixelmatch threshold to a small non zero value is the usual way to stop anti aliasing noise from failing a suite.

The command yields a Visual Regression Result object. The exact shape of that object is worth checking in the type definitions shipped in dist, since the result object is what a test would assert on when it wants to check more than pass or fail.

Where a pixel diff stops being the right tool

The hard part of visual regression testing is not the comparison, it is the decision about what should be considered stable. A pixel diff answers the question of whether the rendering changed, and that is a different question from whether the change is a bug.

The most common failure is environmental. Font rendering differs between operating systems and between browser versions, so a baseline captured on a developer laptop and compared in a Linux container will produce differences that are not regressions. The same applies to animations, time stamps, carousels and anything that renders a random value. A test suite built on raw pixel equality in that situation produces failures nobody can act on, and the usual result is that people stop reading the results.

The thresholds in the options table are the mitigation, and they only go so far. A global errorThreshold of 0.5 is a blunt instrument, and the per call override exists precisely because different components deserve different sensitivity. A component with a shadow or a transition needs a threshold; a static table of text does not.

There is a second structural limitation. The result is a picture, so a failure gives you a picture to look at. There is no assertion on semantics, so a change that swaps the order of two buttons, or changes the accessible name of a control, or moves a form field, can be invisible to a pixel diff if it happens to occupy the same space, and can be a serious functional regression. Behavioural assertions catch that class; pixels catch the other class. A suite that relies on images alone is testing rendering, not behaviour.

The third limitation is the size of the baseline set. Every image is a file in the repository, and a suite of a hundred visual assertions is a hundred binary files to review on every change, which is a review cost that grows quickly and does not scale the way a text diff does.

How the plugin is built and what to read in the source

The build setup tells you the shape of the project. The build script runs tsup over four entry points, src/plugin.ts, src/command.ts, src/utils/image.ts and src/utils/logger.ts, emitting both ESM and CommonJS formats with type declarations. The main field points at dist/plugin.js, the module field at dist/plugin.mjs, and the types field at dist/plugin.d.ts, and only the dist directory is published to npm, so the package is small.

The prebuild script is unusual and worth noticing: it removes dist, runs lint with fix, runs format with fix, and then runs the tests. The tests are vitest, and the test script is vitest run with the globals flag. The typecheck script is a separate tsc --noEmit over the project, and the preversion script runs lint, format check, test and build in sequence before a version bump. In other words, the repository treats formatting, linting, types and tests as a gate rather than as advice.

Four source entry points is a small enough codebase to read in an afternoon, and the split is informative. plugin.ts is the Cypress side, the thing configureVisualRegression installs into the test run. command.ts is the custom command, which is why the README's import path for it points into dist/command. utils/image.ts is where the pixel work lives, with pixelmatch and pngjs. utils/logger.ts is the chalk based reporting, which is how you get a coloured diff summary in the terminal rather than a raw exception.

The dev dependencies also suggest how the project is tested: faker for generating test images, esbuild and esbuild-node-externals for the build, eslint-config-love as the shared lint rules, and cross-env for setting environment variables across platforms in the test scripts.

The changelog is in the repository root, and with no GitHub releases the CHANGELOG.md and the npm version history are the record of what changed and when.

Editorial conclusion

Adopt cypress-visual-regression if your team already runs Cypress end to end tests and has a component or page whose rendering matters enough to pin with images, and you are willing to curate baselines. Do not adopt it as a general UI test strategy, since it compares pixels rather than behaviour, and a pixel diff tells you something changed without telling you whether the change is a regression. Before you switch it on, generate baselines on the platform you will run CI on, set the errorThreshold per test rather than globally, and leave visualRegressionFailSilently off until the baseline set is trustworthy.

Frequently asked questions

Official sources

1. cypress-visual-regression/cypress-visual-regression on GitHub
2. Issues
3. License: MIT
4. README