# job-iteration: Making Rails Background Jobs Interruptible and Resumable

> Shopify's job-iteration is an ActiveJob extension that turns a long running job into a cursor based iteration, so a deploy, a worker restart or an evicted pod stops the job where it is and a later run picks it up from the checkpoint. The cursor comes from the enumerator you build, so the same mechanism handles ActiveRecord records, batches, relations, CSV files and nested enumerations.

**Shopify/job-iteration** — Makes your background jobs interruptible and resumable by design.

- Repository: https://github.com/Shopify/job-iteration
- Website: https://www.rubydoc.info/gems/job-iteration
- Stars: 1,318 · Forks: 64
- Language: Ruby
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/shopify-job-iteration

## The Job That Grows Past the Deploy Cycle

The README opens with a job that everybody who runs Rails in production has written at some point:

```ruby
class SimpleJob < ApplicationJob
  def perform
    User.find_each do |user|
      user.notify_about_something
    end
  end
end
```

The description of its failure modes is precise. The job runs quickly with a hundred User records, but as the number of records grows it takes longer, and eventually there are millions of records and the job ends up taking hours or even days. With frequent deploys and worker restarts, a job will be either lost or restarted from the beginning, and some records, especially those at the beginning of the relation, will be processed more than once.

The cloud part is the part that is hard to argue with. There is no way to guarantee that a single job will have reserved hardware to run for hours and days. What if AWS diagnosed the instance as unhealthy and restarted it in five minutes? What if a Kubernetes pod is evicted? Again, all job progress is lost.

And there is a third case, which is specific to running at scale. At Shopify, the library is also used to interrupt workloads safely when moving tenants between shards and to move shards between regions.

The principle the project is built on is the disposability idea from the twelve factor methodology: software designed for high availability must be resilient to interruptions that come from the infrastructure.

## Two Methods Replace perform

The change to a job is small and the shape of it is the whole API. You include the JobIteration::Iteration module and describe the job with two methods, build_enumerator and each_iteration, instead of perform:

```ruby
class NotifyUsersJob < ApplicationJob
  include JobIteration::Iteration

  def build_enumerator(cursor:)
    enumerator_builder.active_record_on_records(
      User.all,
      cursor: cursor,
    )
  end

  def each_iteration(user)
    user.notify_about_something
  end
end
```

The contract is stated in the API section: the job must respond to build_enumerator and each_iteration, and build_enumerator must return an Enumerator object that respects the cursor value.

The split is what makes the work resumable. The enumerator knows how to produce the sequence and where to start, and the iteration method is the body that runs once per item. The relation is ordered by primary key, exactly like find_each does, which is what makes the cursor meaningful, because the order is the same on every attempt.

So the two methods are not a stylistic preference. A job that cannot be described as an enumerator plus a body cannot be checkpointed, because there is nothing to resume from.

## The Cursor Is the Whole Mechanism

The cursor is the value that says how far the job got, and it is threaded through the enumerator builder on every call, which is why every example in the README passes it.

The enumerator builder is where the shape of the work is chosen, and there are several shapes. active_record_on_records iterates a relation record by record. active_record_on_batches works in batches of a given size, in this case a hundred comments at a time, and the batch is passed to the iteration method as an array of records. active_record_on_batch_relations is the variant where the batch is a relation rather than an array, and the example uses update_all on it, which is the point, because a relation batch can be updated in the database without loading it.

There are also enumerators that have nothing to do with Active Record. The array enumerator takes any array, and the csv enumerator takes a CSV file, so a job that imports a file can be checkpointed by line.

The nested enumerator composes them, taking a list of lambdas that each receive a cursor:

```ruby
def build_enumerator(cursor:)
  enumerator_builder.nested(
    [
      ->(cursor) { enumerator_builder.active_record_on_records(Shop.all, cursor: cursor) },
      ->(shop, cursor) { enumerator_builder.active_record_on_records(shop.products, cursor: cursor) },
      ->(_shop, product, cursor) { enumerator_builder.active_record_on_batch_relations(product.product_variants, cursor: cursor) }
    ],
    cursor: cursor
  )
end
```

The nested example is the one that shows the design scaling, since a cursor for a three level iteration is a composite of three cursors, and the lambdas receive the parts they need.

## Parallel Iteration and the Throttling Guide

Two of the features change the execution model rather than the enumeration, and both are documented in their own guides.

Parallel iteration runs the work in several jobs at once. The example asks for five instances, and the comment explains the effect: it runs in five separate jobs, with each job processing a subset of the users. The enumerator builder call is parallel_active_record_on_records with an instances argument, so the same records are split across jobs and the cursor has to track which instance and which position.

Throttling is the opposite adjustment, and the guide is listed alongside the parallel iteration one. A job that is checkpointed and resumable can be run at whatever rate the queue allows, and a job that hammers a database with five parallel instances is exactly the situation where you want to slow it down.

Both features point at the same property. Because progress is checkpointed, the job does not need to be scheduled once and left alone, and because the job can be interrupted, the rate at which it runs can be changed between attempts without losing the work already done.

## Interruption Comes From the Queue Adapter

The library hooks into most popular queue adapters out of the box to support graceful interruption, and no extra configuration is required. The supported adapters are listed in order of implementation: Resque, Sidekiq, GoodJob, Solid Queue, Amazon SQS and Delayed::Job.

That list is the interesting engineering detail. The library cannot interrupt a job by itself, because a job that is being executed inside a worker process is not something the library can reach unless the queue adapter cooperates. So each adapter has an implementation that knows how to ask a running job to stop, and the job then finishes its current iteration, saves the cursor and exits, leaving the work in the queue for the next attempt.

The platform requirements are short. Ruby 3.1 and later, and Rails 7.1 and later. The README adds a caveat that support for older platforms that have reached end of life may occasionally be dropped if maintaining backwards compatibility is cumbersome, which is an honest statement about the cost of supporting a wide adapter matrix.

The primary requirement is Active Job, and for iteration without Active Job in Sidekiq there is a separate Sidekiq Iteration project linked from the requirements section, which is the answer for the case where the job is a plain Sidekiq worker rather than an ActiveJob.

## What the Guides Cover and What the Repository Holds

The README points at a guides/ directory with six documents, and they map onto the decisions a user has to make.

Iteration: how it works is the one to read first, because the checkpoint mechanism is not obvious from the API. Job argument semantics covers how the job's arguments behave across attempts, which matters because the cursor has to survive a retry and the arguments are how it travels. Best practices is about how to structure a job so it is worth checkpointing. Writing custom enumerator is for the case where none of the built in shapes fit. And throttling and parallel iteration are the two guides for the operational questions.

The repository is a conventional Ruby gem. There is lib/ for the implementation, test/ for the suite, the guides/ directory, bin/, a gemfiles/ directory for testing against multiple dependency sets, a Rakefile, a gemspec, a .ruby-version, a .rubocop.yml and a .yardopts for the documentation generator. There is also a CHANGELOG.md, a CODE_OF_CONDUCT.md and a dev.yml, which is the development container definition.

The release history shows active maintenance, with v1.15.1 and v1.15.0 both published on 2026-09-18 and v1.14.0 on 2026-05-14, and the API documentation is published on rubydoc. The last push was on 2026-10-15.

## Conclusion

job-iteration fits a Rails service whose background jobs grow past the deploy cycle, since it is the difference between a job that restarts from zero and a job that continues from its checkpoint, and it fits teams that have to move tenants between shards or regions without losing work. It is a poor fit for short jobs, since the checkpoint machinery and the enumerator contract cost more than they save, and a poor fit if your work is not naturally a sequence, because the cursor is the mechanism and an unordered or non repeatable body cannot use it. Before adopting it, read the guide on how it works and the one on argument semantics, because the arguments of the job are what carry the cursor between attempts. The gem is MIT licensed, requires Ruby 3.1 and Rails 7.1 or later, the newest release is v1.15.1 from 2026-09-18, and the last push was on 2026-10-15.

## FAQ

### What problem does job-iteration solve?

It makes long running ActiveJob jobs interruptible and resumable. The README describes jobs that iterate over millions of records, taking hours or days, and being lost or restarted from the beginning when a deploy or a worker restart happens, with some records processed more than once, and it notes that cloud infrastructure can restart an instance or evict a pod at any time.

### How do I convert a job to use job-iteration?

Include the JobIteration::Iteration module and replace perform with two methods, build_enumerator and each_iteration. The job must respond to both, and build_enumerator must return an Enumerator object that respects the cursor value.

### Which enumerators does job-iteration provide?

There are builders for Active Record records, Active Record batches, batch relations, parallel iteration over records with an instances count, plain arrays, CSV files, and a nested builder that composes several enumerators with lambdas. Each takes the cursor so the sequence can be resumed.

### Which queue backends does job-iteration support?

Resque, Sidekiq, GoodJob, Solid Queue, Amazon SQS and Delayed::Job, in order of implementation. The library hooks into the popular queue adapters out of the box to support graceful interruption, with no extra configuration required.

### What are the platform requirements for job-iteration?

Ruby 3.1 and later, and Rails 7.1 and later. Active Job is the primary requirement, and the README notes that support for older platforms that have reached end of life may occasionally be dropped if maintaining backwards compatibility is cumbersome.

## Sources

- [License: MIT](https://github.com/Shopify/job-iteration/blob/main/LICENSE)
- [Project website](https://www.rubydoc.info/gems/job-iteration)
- [README](https://github.com/Shopify/job-iteration/blob/main/README.md)
- [Releases](https://github.com/Shopify/job-iteration/releases)
- [Shopify/job-iteration on GitHub](https://github.com/Shopify/job-iteration)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/shopify-job-iteration