Split: the Rack-based A/B testing framework for Ruby apps

What Split solves, and who it is for

Split exists so that a Ruby web application can serve different versions of a page or a code path to different visitors and later ask which version performed better. The README describes it as a rack based A/B testing framework designed to work with Rails, Sinatra or any other rack based app, and it credits Abingo, Vanity and Resque as influences. That lineage explains the shape of the thing: it is a library you drop into an existing application rather than a service you send events to.

The intended user is a developer who can edit the view or controller that renders the thing being tested. The README's first example is a login button image chosen by ab_test, and the controller example picks a number of free starter points for new users. Both are decisions made in application code, which is where Split expects to live. If your marketing team wants to build experiments without a deploy, this is not the workflow Split offers.

How ab_test and ab_finished move a visitor through an experiment

The mechanism is a single method call that both registers the experiment and returns an alternative. The README states that ab_test names the experiment with its first argument and takes the alternatives as the remaining arguments, and that a returning user gets the same alternative as before. That persistence is the core of the design: without it, a visitor would see a different button on every request and the results would be meaningless.

Conversion is a separate call. ab_finished marks a completion for a named experiment, and the README shows it in a controller after business logic and inline in a view after a signup message. The two calls are deliberately decoupled, so an experiment can be started on one page and finished several steps later.

Alternatives are not required to be equally weighted. The README shows three forms of the weight argument, including ab_test(:homepage_design, {'Old' => 18}, {'New' => 2}), which it says shows the new alternative to visitors 1 in 10 times, with a default weight of 1. That is the mechanism to use when an alternative is experimental or has not been load tested.

For development, the README documents a URL override: a request to a page with ab_test[button_color]=red forces the red alternative. The same paragraph notes that the override is not stored in the session and does not count toward results unless the store_override configuration option is set. That distinction matters, because an override used during manual testing will otherwise leave your numbers untouched, which is usually what you want.

Installing the gem and running a first experiment

Split is distributed as a RubyGem, and the README points at rubygems.org/gems/split as its home. The requirements section states that Split v4.0 and later is tested with Ruby 2.5 or greater and Rails 5.2 or greater, and that older Ruby or Rails projects should look at v3.0 or v0.8.0. Redis is the datastore, and Split only supports Redis 4.0 or greater. The README's macOS instructions use Homebrew and start the server on port 6379.

brew install redis
redis-server /usr/local/etc/redis.conf

Installing the gem itself is one command. In a Rails application the README says that adding the gem to the Gemfile autoloads it once Redis is configured, so no initializer is strictly required to begin.

gem install split

Sinatra needs two extra steps. The README says to enable sessions and mix in the helper methods at the top of the app, which is what makes ab_test available inside routes.

require 'split'

class MySinatraApp < Sinatra::Base
  enable :sessions
  helpers Split::Helper
end

With that in place, the smallest useful experiment is a view block. The README's example passes two image paths and yields the chosen one to the block, so the rendered page shows one of the two files and the assignment is recorded for that visitor.

<% ab_test(:login_button, "/images/button1.jpg", "/images/button2.jpg") do |button_file| %>
  <%= image_tag(button_file, alt: "Login!") %>
<% end %>

When the visitor completes the action you care about, call ab_finished with the same experiment name. The README shows this in a controller, which is the more reliable place because it runs regardless of which view rendered.

def buy_new_points
  # some business logic
  ab_finished(:new_user_free_points)
end

What you should see after this is a recorded alternative for the visitor and, once conversions arrive, a row for the experiment on the dashboard. The README does not document a command line interface for creating experiments or reading results, so the dashboard is the surface you work from.

Reading significance, and the sample size problem the README admits

The dashboard offers two ways to judge an experiment. The default uses a z test on the difference between the control and alternative conversion rates, and the README is explicit about its limits: it works only with more than 30 participants and 5 conversions per branch, it reports significance at the 90, 95 or 99 percent level, and it cannot tell you which alternative is best when an experiment has more than two branches.

The second option simulates from a beta distribution to estimate the probability that a given alternative is the winner across all alternatives, and the README recommends it precisely for experiments with more than one control and one alternative. It is also usable for a simple two-way test. The README warns that these simulations are slow for a large number of experiments, so results are cached, with recalculation defaulting to once per day.

The honest part is the caveat about false positives. The README cites a blog post on the pitfalls of A/B testing and states that determining the requisite sample size per branch before running the experiment is highly recommended, because otherwise the rate of false positives increases. That is not a footnote about statistics in general. It is a warning that the default z test will happily report significance on an underpowered test, which is the failure mode most teams hit in practice. Split gives you the number; it does not stop you from acting on a bad one.

Recalculation cost and the dashboard_calculate_winning_alternatives setting

The beta-distribution probabilities are computed during dashboard rendering. The README says the recalculation is triggered while rendering the dashboard and that this can make the dashboard slow when many experiments are due at once. On busy installations it recommends turning the on-dashboard calculation off and running it from a background job instead, with Split::ExperimentCatalog.all.each(&:calc_winning_alternatives) given as the job body.

The configuration for both the interval and the on-dashboard switch is shown in the README. Setting the interval to 3600 recalculates hourly rather than daily, and disabling dashboard calculation leaves the dashboard showing the most recently calculated probabilities.

Split.configure do |config|
  config.winning_alternative_recalculation_interval = 3600 # 1 hour
  config.dashboard_calculate_winning_alternatives = false
end

This is a real operational constraint rather than a tuning detail. A team that never moves the calculation off the dashboard will find that the page gets slower as experiments accumulate, and the fix requires a job runner that the README does not choose for you. Split leaves that decision open, which fits its stated goal of being hacker friendly, but it means the deployment work is yours.

Where Split is the wrong tool

Split assumes Redis is available and that assignments should live there. If your application has no Redis and you do not want to add one for experimentation, the gem has nothing to fall back on; the README names Redis as the datastore without describing an alternative backend. Redis 4.0 or greater is a hard floor, so an older managed instance is a blocker.

The framework also assumes the person running the experiment can change code. There is no documented way to create an experiment from a user interface, no segmentation by user attributes in the README, and no targeting rules beyond the URL override and alternative weights. A team whose experiments are authored by non-engineers, or which needs to target by geography or account tier, will find the model too narrow.

Finally, the statistical layer is deliberately minimal. The default z test does not rank more than two alternatives, and the README itself warns about false positives when sample size is chosen after the fact. Split tells you what happened to your conversion rate. It does not design the experiment for you, and it does not enforce the sample size calculation its own documentation recommends.

Split against a hosted experimentation platform

The alternative most teams weigh against Split is a hosted experimentation product, where assignment, event collection and analysis run outside the application and the SDK is a thin client. The difference in approach is where the logic lives. With Split, the experiment is a method call in your view or controller, the alternatives are arguments in Ruby, and the data sits in your Redis. With a hosted platform, the experiment is configured on someone else's servers and your code asks for a variant.

The trade is control against operational surface. Split keeps everything in your repository and your infrastructure, which means no third party sees your conversion data and no external service sits in the request path. The cost is that you own the Redis instance, you own the dashboard recalculation job described above, and you own the statistical interpretation. A hosted product absorbs that work and adds features the README does not describe here, such as targeting rules and experiment authoring outside the codebase.

There is also a lighter-weight path worth naming: writing the assignment yourself with a random number and a cookie. That avoids the dependency entirely, but you would then be building the persistence, the weighting, the override and the significance calculation that Split already provides. For a single throwaway test that is reasonable. For a programme of experiments it is not.

Maintenance, licence and upgrade cost

The repository is not archived, and the last push was on 2026-09-15. The most recent release listed is v4.0.5 from 2025-08-04, preceded by v4.0.3 in 2023-11-15 and v4.0.2 in 2022-12-02. Release cadence is therefore uneven: a gap of roughly two years separates v4.0.3 from v4.0.5, and the version numbering stays inside the 4.0 line. Anyone planning to depend on Split should read that cadence as a signal about how quickly fixes arrive rather than assuming a steady release train.

The version history also defines the upgrade boundary. The README states that Split v4.0 and later is tested with Ruby 2.5 or greater and Rails 5.2 or greater, and that projects needing Ruby 2.4 or older Rails should try v3.0 or v0.8.0. Upgrading from those older lines is the main compatibility decision, and the README does not describe a migration path between them.

The licence is MIT, which is permissive and places few obligations on how you redistribute or modify the gem. That is a statement about the licence identifier in the repository, not legal advice; if your organisation has licence policy, route it through whoever handles that.

Editorial conclusion

Adopt Split if your application is already a Rack app, you run Redis 4.0 or later, and you want experiment logic to live in the same repository as the code it changes. Do not adopt it if you need a hosted experimentation platform with its own SDKs and experiment governance, or if you cannot operate a Redis instance for this purpose. Before committing, verify the Ruby and Rails versions your application runs against the requirements in the README, and check how your team will handle the dashboard recalculation described in the configuration section, because on a busy installation that work can be moved to a background job.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. splitrb/split on GitHub