deprecation_toolkit: a recorded baseline of Ruby deprecations

Deprecations as a failing test instead of a background worry

Deprecation Toolkit is a Ruby gem that helps you get rid of deprecations in your codebase. The reasoning in the introduction is short and worth repeating in substance: having deprecations in your application usually means something will break whenever you upgrade a third-party dependency, and the sooner they are fixed the better.

What makes that actionable is the problem it admits. Fixing all deprecations at once is tough, depending on the size of the application and the number of deprecations, and you may have to resolve them progressively while making sure the team does not add new ones. The gem exists for exactly that middle state, where the backlog is large enough that a big bang cleanup is not going to happen and small enough that a list will do.

The project is from Shopify, is MIT licensed, and the repository was last pushed on 2026-10-19 with releases v2.4.0 on 2026-05-07, v2.3.0 on 2025-10-22 and v2.2.3 on 2025-04-16. The topic tags are deprecations, gem-upgrade and testing, which is the whole scope in three words. The repository is a normal gem layout with lib/, spec/, test/, a Rakefile, a gemspec, a Gemfile with a lock file, a gemfiles/ directory for testing against several dependency sets, a .devcontainer/ directory, a .rubocop.yml, a .ruby-version, a CHANGELOG.md and a RELEASE.md.

The baseline is a set of YAML files next to your tests

The mechanism is a baseline comparison. The gem records all existing deprecations into .yml files, and when a test encounters a deprecation that is not in the recorded set, the configured behaviour triggers. The default behaviour is to raise an error, which in a test run means a red build.

The comparison runs in both directions, and that is the detail that makes the tool useful over a long period. A deprecation that was recorded and has now disappeared is also an error, because the recorded list is out of date. In practice that means the baseline is a contract: it says these are the deprecations that exist right now, and both adding one and removing one is a change that someone has to look at.

There are two ways to record. You can set the behaviour to the record behaviour in configuration:

DeprecationToolkit::Configuration.behavior = DeprecationToolkit::Behaviors::Record

Or you can run your tests with the record flag, whose short form is -r:

rails test <path_to_my_test.rb> -r

The default location for the recorded files is the test/deprecations folder, and the location is configurable. The deprecation_path setting accepts either a string or a proc, and the proc is called with the path of the running test file, which is what lets you split the baseline by area of the application rather than keeping one list for everything.

Four behaviours, and a hook for writing your own

The behaviour setting decides what happens when a non-recorded deprecation is encountered, and the gem ships four of them.

Raise is the default and raises one of two errors. DeprecationIntroduced is raised when a new deprecation appears, and DeprecationRemoved is raised when a deprecation that was recorded has gone away. Record records the deprecations instead of raising, which is what you use when building the baseline. Disabled does nothing, and the documentation describes its purpose as temporarily turning the gem off without removing it from the Gemfile, which is a sensible way to unblock a hotfix without changing dependencies.

The fourth is CIRecordHelper, and it is the interesting one because it exists to solve a problem specific to modern CI.

The extension point is the fourth option. You can create your own behaviour object and perform whatever logic you want, as long as it responds to trigger:

class StatsdBehavior
  def trigger(test, deprecations, recorded_deprecations)
     # Could send some statsd event for example
  end
end

DeprecationToolkit::Configuration.behavior = StatsdBehavior.new

The trigger signature hands you the test, the deprecations found and the recorded ones, so a custom behaviour can count deprecations per test, send them to a metrics system, or open a ticket. The example in the documentation does exactly the first of those, sending a statsd event. This is the part of the gem that generalises: the same baseline mechanism can be a hard failure in CI and a measurement in a report.

Recording the baseline when CI splits your test files apart

The CIRecordHelper behaviour exists because of how CI systems run tests, and the explanation in the documentation is specific enough to be useful.

Imagine you have thousands of tests and need to record deprecations for each one on your own machine, which is going to take ages. Recording on CI with the ordinary record behaviour is not possible, because of the way CI parallelises tests in multiple containers: multiple tests from the same file run in parallel on different machines, the deprecation files that get created are then split, and regrouping them is described in the documentation as a nightmare.

The behaviour works around that by writing a JSON representation of the deprecations instead of separate files. Your CI needs a way to download the log generated from the test run, you download it on your local machine, and you then run a rake task with the log path passed in as FILEPATH, using the task named deprecation_toolkit:record_from_ci_output. That task parses the CI log, grabs the output the toolkit produced, and converts everything back into the YAML files.

So the workflow inverts. Instead of one machine recording the whole baseline, every CI worker records its own slice into a single log, and the log is reassembled afterwards into the committed baseline. The cost is a step in the release process that only exists while the baseline is being built, and the benefit is that a large suite can be baselined in one CI run rather than one local run that takes hours.

Configuration for the messy parts: paths, allow lists and plain warnings

Three configuration settings deal with the parts of deprecation handling that do not fit the default path.

The first is deprecation_path, which takes a string or a proc called with the path of the running test file:

DeprecationToolkit::Configuration.deprecation_path = 'test/deprecations'
DeprecationToolkit::Configuration.deprecation_path = -> (test_location) do
  if test_location == 'admin_test.rb'
    'test/deprecations/admin'
  else
    'test/deprecations/storefront'
  end
end

The second is allowed_deprecations, which takes an array of Regexp and Procs. A deprecation matching one of them is ignored, which is the escape hatch for a warning you have decided to live with:

DeprecationToolkit::Configuration.allowed_deprecations = [/Hello World/]

ActiveSupport::Deprecation.warn('Hello World') # ignored by Deprecation Toolkit

When you pass a proc, it receives the deprecation message and the callstack, which lets you allow a warning based on who raised it rather than on what it says.

The third is warnings_treated_as_deprecation, and it exists because most gems do not use ActiveSupport::Deprecation but just call Kernel#warn to print a message. The setting takes an array of regular expressions, and an empty regex matches every warning:

DeprecationToolkit::Configuration.warnings_treated_as_deprecation = [//]

Anything that responds to === can be a matcher, so a proc works as well. That setting is the one that widens the net the most, and it is also the one that will turn a noisy dependency into a failing build, so it belongs in the same commit as the baseline it needs.

There is also a setting for the file path format of the recorded files. For Minitest the default uses the class name, and for RSpec it is the file location with the spec part removed, and a custom format can be given as a proc that receives the test instance.

Minitest by default, RSpec by configuration, and what the baseline costs you

By default the gem uses Minitest as its test runner, and using it with RSpec requires configuration, which the documentation shows in a section whose example is cut short in the README. That is worth checking against the current documentation before you wire it into a RSpec suite, because the file naming defaults differ between the two runners and a RSpec setup that does not get the format right will write its baseline somewhere you did not expect.

The costs of the approach are real and they are structural rather than accidental. The baseline has to be committed and kept current, and because a removed deprecation is also an error, every fix to a deprecation requires a baseline update in the same change. That is the intended workflow, and it is the mechanism that stops a stale list from hiding a new problem, but it does mean the gem produces failing tests during a cleanup, not only during a regression.

The escape hatches exist for that. allowed_deprecations lets you whitelist a specific warning while you deal with it, the Disabled behaviour turns the whole thing off temporarily, and a custom behaviour lets you route the events somewhere other than an exception, for example into metrics. A team that wants visibility without blocking merges can implement a behaviour that counts and reports, and a team that wants enforcement uses the default.

What the gem does not do is tell you where the deprecation comes from or how to fix it. It finds warnings, records them, and compares them. Deciding that a deprecation is acceptable, scheduling the work to remove it, and keeping the list honest across a year of dependency upgrades is the part that stays with your team, and the value of the toolkit is that it makes that work visible instead of implicit.

Editorial conclusion

deprecation_toolkit fits a team that is tired of finding out about a broken dependency upgrade from a production incident, because it turns the deprecation backlog into a visible list and a failing test, and the custom behavior hook means the same mechanism can feed a metric instead of only an error. It is the wrong tool if your deprecations come from a source the gem does not see, and it is the wrong tool if you cannot accept that removing a deprecation is also a failure, since the recorded list has to be updated as you fix things. Before adopting it, run it in the record behavior once to build the baseline, commit those YAML files, then switch to the default raise behavior, and check whether your test runner is Minitest or RSpec, because the default is Minitest and the RSpec setup is a separate configuration step.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. Shopify/deprecation_toolkit on GitHub