evil-seed: partial anonymised dumps built from your own model graph

The problem, stated as two separate ones

The motivation section is short and it is the clearest statement of what the tool is for.

Using production-like data in a staging environment could be very useful, especially when debugging production bugs that only show up with real data.

The easiest way to get it is to use production database backups. But that is not an option for rather large applications, for two reasons.

The first is size: a production dump can be extremely large, and it just cannot be dumped and restored in a reasonable time.

The second is that you should care about sensitive data, and the readme puts the word anonymization in parentheses as if it were the less important half. It is not. The two reasons are independent, and a tool that solved only the first would still be unusable, because a copy of your production database in a staging environment is a copy of your production database in a place with weaker controls and a different team.

So the shape of the solution follows from the shape of the problem. You do not want all the data, so take some of it. You want it to be the data your application would actually produce, so take it from the application's own models rather than from a generic table export. You want the sensitive values replaced, so have a mechanism for replacing them. And you want the result to be something you can put into a database, so write a plain SQL file.

That last point is worth holding on to, because it is a constraint that shapes everything else. The output is not a proprietary archive format that needs this tool to read back. It is a SQL file, restorable with any database client, and the readme says so and gives the one-line Ruby equivalent for people who would rather not leave their process.

The project is MIT licensed and is built by a design and engineering consultancy whose stated specialities are developer tools, artificial intelligence, and cybersecurity startups, which is a slightly unexpected home for a database tool and suggests the maintainers use it themselves.

Roots, and the constraint that bounds the dump

The configuration is a single block of code, and the first thing it asks for is the set of root models with their constraints.

The readme explains this in a way that is worth repeating, because it is the concept the whole tool turns on: you specify the root models and their constraints to limit the number of dumped records, and it is like a where clause on the model with no other conditions.

So a root is a model plus a condition, and the condition is expressed the way you would express it in a query. The example is a forum filtered to the featured ones, and the configuration comment gives the equivalent as a direct query on the model, which is a good way to say it: the root is a query, and the tool does the rest of the work from there.

From that root, everything else follows. The tool walks the associations of the root record, and it walks them in a deep-first fashion, which the caveats section comes back to, because that is the source of most surprises. Every record it reaches is included in the dump, and every record it reaches is anonymised according to the rules for its model, whether or not you thought of that model when you wrote the configuration.

The root block also carries the tools for shaping the result, and there are a lot of them. You can limit the number of root records. You can specify the order the records are selected in, which matters because a limit and an order together are how you get the newest hundred rather than an arbitrary hundred. And the readme is careful to say that anything you can pass to the where method works as a constraint, with an example that uses a raw SQL fragment with a bound parameter and a time, so the constraint language is the query language and not a separate thing.

Then there is the case the readme singles out: system-wide models, where you omit the constraints so that every record is dumped, and you exclude all the associations. A role table is the example. That is the dictionary of the application, and a staging environment needs it intact, and it needs nothing else attached to it.

The association path language

The most interesting part of the configuration is the language for saying which associations to follow and which to ignore, and it is worth understanding because it is the part that determines what actually ends up in the file.

The exclusion mechanism takes a pattern of an association path, and the readme defines what a path is: a dot-delimited string of an association chain starting from the model itself, with an example of three segments. And the pattern can be a regular expression, which is the detail that makes the mechanism usable. A fixed path would let you exclude exactly one chain and nothing else, which is not enough, because what you usually want to exclude is a family of chains: any association whose name matches a certain word, or any association that leads back up to the parent.

The configuration example uses exactly those two shapes: a pattern matching a particular word anywhere in the path, a fixed path for one named chain, and a pattern anchored at the start of the path for the parent association. The three together are the tool for cutting a graph at the places you choose.

The inclusion mechanism is the mirror image, and it takes a hash of association chains to include, with the readme noting that you can also express the same thing as a regular expression, and showing the two forms side by side. The hash form is the readable one and the regular expression form is the general one, and having both means the readable form is a convenience rather than a limitation.

There is a third option, custom scoping for associations, and this one is genuinely useful rather than merely convenient. You can give a block for a particular chain, and inside it you can call any query method. The example orders by creation time, descending, on a nested chain. So you are not limited to saying whether to follow a chain; you can say what part of it to follow.

And the readme's warning deserves emphasis: use the exclusion mechanism aggressively, because you will be amazed how much your application's model graph is connected, and because the traversal is deep-first, some records will get into the dump even if you do not want them there.

That is an honest warning about the tool's behaviour rather than a bug report, and it is the single most important thing for a user to internalise: the dump is a superset of what you nominated, and the exclusion patterns are how you make it not be.

Limits: size, depth, and the rules that are not limits

Two mechanisms keep the dump from growing without bound, and they work at different levels.

The first limits the number of records included for a collection or a single-record association, and it can be set globally or for particular chains. The configuration shows the global setting, then the same setting applied to one chain, then to a nested chain, and then a form that names the associations rather than the path. So the limit can be expressed either as a path from the root or as the association names themselves, and the two forms exist because the second is what you already know when you are looking at a model.

The second limits depth. The readme is careful about what counts: all traversals through the collection, the belongs-to, and the rest are counted, and it gives a worked example where a chain of forums, subforums, questions and answers is five levels deep.

And then there is the rule that is not a limit but sounds like one, and it is the most important sentence in that part of the documentation. The number of included collection and single-record records can be limited for every association, but the belongs-to records for all associations that are not excluded are always dumped, in order to keep referential integrity.

So the tool makes a deliberate choice about referential integrity over your limit. If a question references an author, and the author is not in the dump, the question has a foreign key pointing at nothing, and the database will either reject the insert or your application will explode on the missing record. So the parent rows come along whether you asked for them or not.

That has a consequence worth thinking about when you plan a dump. The belongs-to chain is how data leaks into a dump that you thought you had excluded. If your model has a belongs-to chain that reaches back to a user table, the users come with it, and the anonymisation rules for that model are what stand between you and a copy of your customers' details.

Transformations, anonymisation, and the switches

The configuration has two ways to change the data, and the difference between them is stated clearly.

The first is a customisation block, which takes a model and gives you a hash of the record's attributes. You can replace a value, merge into a nested attribute, or set a timestamp. The example resets the encrypted password to the same value for every user, with the reason given as ease of debugging on a developer's machine, and it does the same for a metadata attribute.

The second is an anonymisation block, which the readme calls a handy domain-specific language for transformations, and which is declarative: you name the attributes and give a block for each, and the value of the block is what gets written. The example uses the usual fake data library for a name and an email address, and a plain block for a login field that appends a suffix to the existing value.

And the warning attached to the anonymisation block is the one that matters: the model setters will not be called, and the results of the blocks will be assigned instead. So an attribute that has a setter doing validation, or normalisation, or hashing, will not be normalised and will not be validated. If your password column is hashed by a setter, a block that assigns a plaintext value writes plaintext, and the readme's example of setting the encrypted password to a known hash is exactly that case handled by hand.

There is also a note attached to the customisation block: you have the hash of the record's attributes, not the record itself. So the block operates on a plain hash and not on a model instance, which is a limitation and also a simplification that keeps the tool independent of your application's callbacks.

Then there are three switches that are about what the tool does rather than about the values. One removes columns from the dump for a model, and the readme notes that this is especially useful with encrypted columns, and that the column is removed even if the model is not a root and is reached through an association. One disables the nullification of foreign keys for records that are not in the dump, which is the setting you reach for when the dump is going into a database that already has the referenced rows. And one unscopes relations so that soft-deleted records and anything else the default scope hides are included.

Two more settings are about output rather than content: a verbose mode that prints the progress of the dump, and a separate switch for printing the SQL as it goes.

Output, compatibility, and restoring

Creating the dump is one call with a path.

require 'evil_seed'
EvilSeed.dump('path/to/new_dump.sql')

And the readme's third tip in the caveats list is to look at the resulting dump, because there are some useful debug comments in it. That is a small thing with a large practical value. A dump of a large database is a file nobody wants to read, but a file with comments that say what was selected and what was excluded turns the artefact into something you can inspect before you trust it, and the comments come for free because the tool knows why it included each record.

The compatibility story is short. PostgreSQL and MySQL are covered by saying that any version which works with the model layer should work. SQLite needs a specific version, three point seven and eleven or newer, and that version is required because of the support for inserting multiple rows at a time. That is the one place where the output format depends on a database feature, and it is worth knowing before you point the tool at a SQLite file.

Restoring is deliberately ordinary. The result is a plain SQL file, and you can restore it with any SQL client, or from Ruby in one line by reading the file and handing it to the connection.

ActiveRecord::Base.connection.execute(File.read('path/to/new_dump.sql'))

And the first restoration tip is the one that catches people out: reset the primary key sequences after restoration, so that default records can be generated afterwards and your application behaves as expected. A restored dump has explicit identifiers in it and the sequence is still where it was, so the next record your application creates collides with one you just restored. It is the difference between a staging environment that works and one that fails in a confusing way an hour later.

The release history shows the tool maturing in the right direction. The most recent version brought a new configuration syntax and custom scopes on associations, and the one before it brought association inclusion, depth limiting, unscoping, and the option to disable nullification. Each of those is a request that comes from using the tool on a real application, and the pattern is the same one the tool itself embodies: start from the application's own model graph, and cut it down to what you need.

Editorial conclusion

EvilSeed fits a team with a large database and a staging environment that needs to look like production for debugging, and that cannot justify copying real customer data into it. It does not fit an application whose associations are so tangled that a partial dump is not representative, because the traversal will pull in more than you expect. Verify first which columns hold the sensitive data, since a column you forget to transform is a column in your staging database, that the anonymisation rules cover the model and not only the root models, since associations bring in records you did not nominate, and that the resulting dump is restored into a database whose schema matches, because the file is plain SQL and will be applied to whatever is at the other end.

Frequently asked questions

Official sources

1. evilmartians/evil-seed on GitHub
2. Issues
3. License: MIT
4. README
5. Releases