pytest-randomly: shuffling test order to expose the order-dependent test

A plugin that makes order dependence visible

The problem this solves is narrow and well known. Randomness in testing can be quite powerful to discover hidden flaws in the tests themselves, as well as giving a little more coverage of your system, and by randomly ordering the tests the risk of surprising inter-test dependencies is reduced. The README cites Google's C++ test runner googletest as a place where the same technique is used, and it cites research suggesting that dependent tests do exist in practice and that a random order of test executions can effectively detect such dependencies. The audience is therefore a Python team with a suite large enough for a hidden dependency to survive, and the value is not faster tests but tests that mean what they say. A suite that passes only because of the order it runs in is a suite whose failures will not reproduce, and a plugin that reorders the suite turns that latent problem into a visible one on the first run.

Shuffling happens in three levels, and the module level comes first

The reordering is not a flat shuffle of every test in the run. The plugin first shuffles at the level of modules, then at the level of test classes if you have them, and then at the level of functions. That ordering matters for how you debug a failure, because the README points out that since the ordering is by module and then by class, you can narrow down an inter-test pollution failure by rerunning just the module or class that failed. The same feature also works with things like doctests. The practical effect is that a failure caused by one test polluting another shows up as a module or class level problem, and the reproduction step is to rerun that subset with the same seed. This is also why the plugin is not a substitute for isolation. If a suite depends on a fixed global execution order, the shuffle will find that, but the fix still lives in your fixtures, not in the plugin.

One seed per test, derived rather than shared

The second mechanism is reseeding, and its detail is what makes the plugin usable in a suite that creates random data. At the start of the test run, and before each test setup, run and teardown, the plugin resets Python's global random seed to a fixed value using random.seed(). The fixed value is derived from the base random seed, the pytest test ID, and an offset for setup or teardown, which ensures each test gets a different but repeatable random seed. Different but repeatable is the key phrase. A shared seed would mean every test sees the same random numbers and would surface collisions between tests; a per-test seed derived from a stable identity means each test is deterministic on its own while the whole run still varies. The README connects this to factory boy's fuzzy values, where tests can create data based on random numbers and yet remain repeatable, and argues it also ensures the tested system is not affected by any data that is filled in randomly due to not being specified.

The libraries it reseeds, and the one it does not

Python's global random module is only half the problem, since most test data libraries keep their own random state. The plugin resets several libraries' random states at the start of every test if they are installed. factory boy is on the list, and so is Faker, where the faker pytest fixture is also affected because the plugin defines the faker_seed fixture itself. Model Bakery is covered, and NumPy is covered but only for its legacy random state, which is an explicit limitation rather than a full one. Polyfactory is covered only for the default random generator, so factories that have called seed_random() with their own seed are unaffected by the plugin. For anything else, there is an escape hatch: additional random generators can be registered under the pytest_randomly.random_seeder entry point and will have their seed reset at the start of every test, where you register a function that takes the current seed value. That entry point is the mechanism a library author or an in-house package would use to opt in, and it is worth checking for your own dependencies before you assume coverage.

Installing it and replaying a failure with the printed seed

Installation is a single pip command, and the supported Python versions are stated right after it:

    python -m pip install pytest-randomly

Pytest finds the plugin automatically, so there is nothing to enable. The output of a run gains one extra line at the start, which tells you the seed in use:

    $ pytest
    ...
    platform darwin -- Python ...
    Using --randomly-seed=1553614239
    ...

If the tests fail because of ordering or randomly created data, you restart them with that seed using the flag the output suggests:

    pytest --randomly-seed=1234

Or more conveniently with the special value last:

    pytest --randomly-seed=last

The README notes that last only works if pytest's cacheprovider plugin has not been disabled, which is a detail worth knowing if you have trimmed your plugin set. To narrow down a pollution failure you rerun just the module or class that failed with the same seed:

    pytest --randomly-seed=1234 tests/module_that_failed/

Since the ordering is by module then by class, that subset run reproduces the interaction without the rest of the suite.

Turning off what you do not want, including the whole plugin

Every behaviour the plugin adds is on by default, and the flags exist to switch pieces of it off. The --randomly-dont-reset-seed flag turns off the reset of random.seed() at the start of every test, and --randomly-dont-reorganize turns off the shuffling of the order of tests. That is the right pair of knobs for the common case where a test asserts on a specific value produced by the global random generator. For a full opt out, the plugin appears to pytest with the name randomly, and the -p argument disables it entirely:

    pytest -p no:randomly

The README also points to the pytest-order plugin for the finer-grained problem, saying to use it to fix the order of some tests and pointing at its documentation section on usage with pytest-randomly. So the intended path for a suite that needs a handful of tests in a fixed order is not to abandon the plugin but to combine it with pytest-order. The plugin also works with pytest-xdist, which matters because a parallel run is where order dependencies are most likely to surface as intermittent failures.

Where it is the wrong tool, and how the project is maintained

The limits are worth stating plainly. Any test that depends on a specific global random value, a fixed clock or a particular insertion order in a shared fixture will start failing, and the plugin offers no way to exempt a whole file from the reseeding other than disabling the plugin for that run. The reseeding is also broad by design: it resets the global seed before setup, run and teardown, so a fixture that deliberately seeds the global generator inside setup to get a known value will find that value overwritten. And the coverage of third-party libraries is partial, as the NumPy legacy-state and Polyfactory caveats show, so a suite using a random library the plugin does not know about keeps its order dependence. On maintenance, the project is a plugin in the pytest-dev organisation, the current source version is 5.0.0 in the pyproject file, and the last push to the repository was on 2026-10-16. The repository publishes no GitHub releases, so the changelog at CHANGELOG.rst and the older HISTORY.rst are the version record. The README is plain about the origin, noting that the plugin started life as the nose plugin nose-randomly, and the licence is MIT.

Editorial conclusion

pytest-randomly suits a team whose suite has grown large enough that hidden inter-test dependencies are plausible, and it costs almost nothing to try because the plugin can be disabled with a single flag or removed for one run. It is the wrong tool for a suite that deliberately asserts on a specific execution order, since the shuffling runs at the module level first, then the class level, then the function level and there is no built-in way to exempt a subset, and it is the wrong tool if your tests depend on a fixed random seed from elsewhere. Verify first that your dependencies are covered by the reseeding rules, because the plugin only resets the libraries it knows about and provides a pytest_randomly.random_seeder entry point for the rest, and note that the current source version is 5.0.0 with the last push on 2026-10-16 and no GitHub releases published.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. pytest-dev/pytest-randomly on GitHub
4. README