# datafaker generates test data shaped like your table and writes it to Kafka or a database

> datafaker is a Python tool that reads table metadata, derives construction rules from it, generates data that matches the column types, and streams or batches the result into relational databases, Hive, Kafka, MongoDB and more. It is a data loader with a schema-aware generator attached, not a general purpose fake data library.

**gangly/datafaker** — Datafaker is a large-scale test data and flow test data generation tool. Datafaker fakes data and inserts to  varied data sources. 测试数据生成工具

- Repository: https://github.com/gangly/datafaker
- Stars: 644 · Forks: 164
- Language: Python
- License: not declared
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/gangly-datafaker

## The pain the project names: rows that obey the schema

The README lays out the motivation in a list of five problems with manual test data creation, and each one is a design constraint on the tool.

The first is that it is time consuming, because you have to construct different data for fields of different data types in a table. The second is limited data volume, since manual methods are impractical when you need a large amount. The third is inaccurate data, and the examples given are email addresses that must follow a certain format, phone numbers with a fixed number of digits, IP addresses with a fixed format, and age as a non-negative value in a reasonable range, where manual construction may not meet the range or format requirements and leads to backend program errors. The fourth is multi-table association, where the amount of manually created data is small and primary keys across tables may not be associated correctly or may have no associated data at all. The fifth is dynamic random writing, for streaming data written to Kafka at random intervals or rows inserted into MySQL at random, which is cumbersome to do by hand and hard to count.

That framing is what separates datafaker from a fake data library. The unit of work is a table, not a field. The stated gap in the market is that there is no open source test data generation tool that can generate data with a structure similar to MySQL tables, and the answer is to read the table's own metadata and generate rows that fit it.

The feature list follows from that. It covers multiple data types including common database field types and custom ones such as IP address, email and ID number. It simulates multi-table association by designating some fields as enumerated types, randomly selected from a specified data list, so that multiple tables can be associated with each other and queried correctly when generating a large amount of data. It supports batch and streaming generation with configurable streaming intervals, several output methods including screen printing, files and remote data sources, and configurable output format with text and JSON currently supported.

The three scenarios it targets are backend development after creating a new table, database performance testing, and streaming data testing for Kafka.

## Five modules from command line to inserted row

The architecture section describes five modules, and the sequence tells you how the tool thinks.

The parameter parser parses the commands entered by the user from the terminal command line. The metadata parser is where the schema-awareness lives: users can specify metadata from local files or remote data source tables, and after obtaining the file content, the parser processes the text content into table field metadata and data construction rules according to predefined rules.

The data construction engine then simulates the generation of different types of data based on those rules. The data routing module splits into batch and streaming generation according to the output type, and for streaming the generation frequency can be specified, with the data then converted to a user-specified format for output. Finally the data source adapter adapts to different data sources and imports the generated data into the corresponding one.

That division has a practical consequence for how you use it. The construction rules are a separate, inspectable artefact derived from the metadata, which means the mapping from column to rule is a thing you can review and adjust rather than something hidden in the generator. The rules themselves are documented separately in the Construction Rules page of the documentation, alongside usage examples, command parameters and notes.

The repository layout matches the pipeline: a datafaker package directory, a bin directory holding the console scripts, a doc directory with the per-topic documentation and the architecture diagram, a tests directory, a tox.ini, a requirements.txt and a setup.py.

## Installing the tool and the driver for your data source

Installation is documented as two methods, and the pip route is the one you would use.

From source, the README says to download the source code, unzip it, and install:

```bash
python setup.py install
```

The pip route is:

```bash
pip install datafaker
```

And there are explicit commands for the two operations you will actually repeat, upgrading and removing:

```bash
pip install datafaker --upgrade
```

```bash
pip uninstall datafaker
```

The driver situation is the part that catches people out, because the core install does not pull in a database client for the source you intend to write to. The README has a table of required packages per data source, and it is a long one. MySQL and TiDB need mysql-python or mysqlclient, with the note to use mysqlclient for Windows and Python 3. Oracle needs cx-Oracle, which requires some Oracle libraries. PostgreSQL and Redshift need psycopg2. SQL Server needs pyodbc with the connection string format mssql+pyodbc://mssql-v. HBase needs happybase and thrift. Elasticsearch needs the elasticsearch package. Hive needs pyhive. Kafka needs kafka-python. MongoDB needs pymongo.

So a realistic install for a Kafka pipeline is the base package plus kafka-python, and for a PostgreSQL target it is the base package plus psycopg2. The requirements.txt file in the repository lists a broad set of these, including mysqlclient, psycopg2-binary, pymongo, pyhive, happybase, thrift, sasl, kafka-python and elasticsearch, so installing everything is possible but heavy.

The core dependencies are more modest: faker at 19.1.0 or newer, which is the library that does the actual value generation, sqlparse for parsing SQL, configparser for configuration files, and sqlalchemy 2.0.0 or newer for database connections.

## The setup script shows the age of the codebase

The setup.py file in the repository is worth reading, because it documents the project's compatibility target in a way the README softens.

The README says datafaker is compatible with Python 2.7 and Python 3.4+. The setup file makes that concrete. There is a branch that appends pywin32 to the requirements on win32, and a branch for Python 2.6 that appends argparse, with a comment explaining that argparse was not in the standard library until 2.7. The extras_require section carries a conditional extra keyed on python_version==2.6, also pulling in argparse.

The version is not hard coded. It is read from the package with a find_version helper that searches the source for a __version__ assignment, and it raises a RuntimeError with the message that it is unable to find the version string if the pattern does not match. The scripts installed are bin/datafaker and bin/datafaker.cmd, the Windows variant included.

Two more details are useful. The package_data includes the data/*.json and data/*.ini files and the requirements.txt, which tells you the tool ships configuration and data tables as package resources rather than generating them at runtime from nothing. And the commented-out line reading requires from requirements.txt is replaced by an explicit list of four dependencies, so the runtime install is deliberately narrower than the development requirements file.

None of this makes the project unusable, but it does set an expectation. A codebase that still carries a Python 2.6 branch and a setup.py install path is one where the packaging conventions predate modern Python, and the README's statement that the current version has been released on PyPI is the instruction that matters for a new install.

## Where datafaker is the wrong tool

Three boundaries, and the first one is about the kind of data you need rather than the tool.

The first is unit test fixtures. If you need three records with a specific first name and a specific email so that an assertion in a test passes, datafaker is the wrong size of tool. The Faker library it depends on is already the standard answer for that, and it is smaller, faster to install, and designed for a single value rather than a table. datafaker's design assumes a schema, a volume, and a destination, and if your need is a value, all that machinery is overhead.

The second is that the metadata-driven design is a constraint, not only a convenience. The tool generates data that resembles your table because it read your table, and the enumerated type mechanism that makes multi-table association work means you are describing relationships by pointing fields at lists of values. That is a workable model, and the Construction Rules documentation exists because the mapping from column to rule is the part you will spend time on. A project whose data does not fit that shape, for example a table whose valid rows are not expressible as a rule per column, will find the tool's flexibility is bounded by its own model.

The third is operational, and it is the one to check before committing. The repository has no GitHub releases, the last push was on 2026-04-04, and the release notes live in a documentation page rather than in tags. The setup file records the licence as Apache License 2.0 and the README badge shows Apache 2.0, while the release history is absent, so nobody is publishing a versioned artefact for you to pin. If your pipeline needs a specific, auditable version, that is a gap you have to close yourself, by pinning a PyPI version and reviewing what changed.

The alternative approach, and it is a real one, is to write the loader yourself against SQLAlchemy, which is already one of the four core dependencies, and use Faker only for the value generation. That is what a small project often ends up doing, because the metadata parsing and rule derivation are the parts you understand best and the parts most likely to need to change. The difference in approach is that datafaker ships that layer as a finished, configurable pipeline with streaming and multiple output adapters, and you get the whole thing for the cost of one pip install plus a driver.

## Conclusion

Adopt datafaker if your bottleneck is filling a table with a million rows that respect the column types and the foreign key relationships, or if you need a continuous stream of synthetic records into Kafka for a pipeline test, because the metadata-driven construction rules are the part that manual scripting does badly. Do not adopt it for unit test fixtures, where the Faker library it depends on is simpler and better suited, and do not assume the project is currently in motion, since the last push was on 2026-04-04 and there are no GitHub releases. Verify first that your database driver is installed, since the drivers are separate packages listed per data source, that the metadata format you feed it matches the documented parser behaviour, and that the licence position is settled, because the setup file records an Apache License 2.0 while the release history is absent and the compatibility statement still mentions Python 2.7.

## FAQ

### What is datafaker and what problem does it solve?

It is a large-scale test data and streaming test data generation tool written in Python. The README states that no open source test data generation tool generates data with a structure similar to MySQL tables, so datafaker reads table metadata, derives construction rules from it, and generates rows that respect the column types and the relationships between tables.

### How do I install datafaker?

Run pip install datafaker, or download the source, unzip it and run python setup.py install. The commands for upgrading with pip install datafaker --upgrade and removing with pip uninstall datafaker are also documented.

### Which database drivers do I need for datafaker?

They are separate packages installed per data source: mysqlclient for MySQL and TiDB, cx-Oracle for Oracle, psycopg2 for PostgreSQL and Redshift, pyodbc for SQL Server, happybase and thrift for HBase, elasticsearch for Elasticsearch, pyhive for Hive, kafka-python for Kafka and pymongo for MongoDB. The four core dependencies are faker, sqlparse, configparser and sqlalchemy.

### How does datafaker handle relationships between tables?

By designating some fields as enumerated types, randomly selected from a specified data list, which ensures that multiple tables can be associated with each other and queried correctly when generating a large amount of data. The Construction Rules documentation covers how the mapping from table field metadata to rules is defined.

### Can datafaker write to Kafka continuously?

Yes. The data routing module splits into batch and streaming generation, and for streaming the generation frequency can be specified, with data converted to a user-specified format before being handed to the data source adapter. Streaming data testing for Kafka is one of the three scenarios the README lists as a target.

## Sources

- [gangly/datafaker on GitHub](https://github.com/gangly/datafaker)
- [Issues](https://github.com/gangly/datafaker/issues)
- [README](https://github.com/gangly/datafaker/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/gangly-datafaker