SDGym: a benchmark harness for synthetic tabular data generators
Benchmarking synthetic data generation methods.
At a glance
- What is it?
- SDGym runs SDV synthesizers, its own baselines, and custom models over public or private datasets and reports quality, privacy, time and memory. It is a measurement tool for people who already write synthesizers, not a generator you adopt on its own.
- Who is it for?
- Adopt SDGym if you already have a tabular synthesizer and need a repeatable way to compare it against SDV's models and baselines on the same datasets, with time, memory, quality and privacy recorded together. Do not adopt it if you only need synthetic data for a downstream task: SDGym scores generators, it does not hand you a production dataset, and the repository labels itself pre-alpha.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap SDGym fills: comparing generators on the same footing
Synthetic tabular data has a measurement problem. Two teams can each train a generator, each report that the output looks plausible, and have no way to compare the results because they used different tables, different row counts and different scoring. SDGym exists to remove that variation. The README describes it as a benchmarking framework for modeling and generating synthetic data, with the stated goal of measuring performance and memory usage across classical statistics, deep learning and other techniques. The scope is deliberately narrow: it is a harness that runs generators and records what happened, not a generator itself. The intended user is someone who already has a modeling technique, either from the SDV library or written from scratch, and needs a standard way to place it next to alternatives. If you have no synthesizer to benchmark, SDGym gives you nothing to run.
How a benchmark run is assembled
The unit of work is a synthesizer name or a custom object, passed to a single entry point. The README's example builds two lists: sdv_synthesizers holds 'GaussianCopulaSynthesizer' and 'CTGANSynthesizer', and baseline_synthesizers holds 'UniformSynthesizer', which ships with SDGym as a baseline. The call sdgym.benchmark_single_table(synthesizers=(sdv_synthesizers + baseline_synthesizers)) runs the comparison. The README states the result is a detailed performance, memory and quality evaluation across the synthesizers on a variety of publicly available datasets. So the data flow is: synthesizer list in, dataset collection applied, per-synthesizer training and sampling executed, metrics computed, results returned. Three extension points are named explicitly: datasets, synthesizers and evaluation. Datasets can come from the SDV project's public collection or from your own data. Synthesizers can come from SDV, from SDGym's baselines, or from your own model. Evaluation goes beyond time and memory into synthetic data quality and privacy metrics. The 'UniformSynthesizer' baseline matters more than it looks: it is the floor. A model that cannot beat uniform random sampling on the chosen metrics is not adding value, and having that reference point built into the same run is the practical reason to use a harness rather than a hand-rolled script.
Writing a custom synthesizer: two functions and a contract
The custom path is small on purpose. You supply my_training_logic(data, metadata), which receives the data and metadata and returns a trained synthesizer object, and my_sampling_logic(trained_synthesizer, num_rows), which takes that object plus a row count and returns synthetic data. Everything else, dataset loading, metric computation, timing, is the harness's job. The README points to a Custom Synthesizers Guide for the details. What the README does not spell out is the interface your returned objects must satisfy beyond being accepted by your own sampling function, or what metadata contains. That is a documentation gap you should expect to close by reading the guide and the source, not by guessing. The design choice is defensible: SDGym does not force your model to subclass an SDV base class, so a scikit-learn style pipeline or a research prototype can be benchmarked without being rewritten. The cost is that the contract is implicit in the example rather than enforced by a signature.
Getting it running and pointing it at your own tables
Installation is two commands, one per package manager. With pip: pip install sdgym. With conda: conda install -c pytorch -c conda-forge sdgym. The README recommends a virtual environment to avoid conflicts with other software on the device, which is a reasonable default given that the synthesizer list includes deep learning models. To see what is available before committing to a run, the README uses sdgym.dataset_explorer.DatasetExplorer().list_datasets(), which prints a table with columns dataset_name, size_MB and num_tables. The sample output includes KRK_v1 at 0.072128 MB with one table, adult at 3.907448 MB, alarm at 4.520128 MB and asia at 1.280128 MB, all single-table. Those sizes are the practical planning input: a benchmark sweep multiplies dataset size by the number of synthesizers, so the difference between KRK_v1 and alarm is roughly a factor of sixty in data volume per synthesizer per run. Custom datasets are supported from local storage or from an Amazon S3 bucket, using a path of the form s3://my-datasets-bucket as shown in the README.
What the harness does not tell you
Two limits are visible in the material. First, the README's worked example is titled for single tables, and the only datasets shown in the listing output are single-table, with num_tables equal to 1 for every row displayed. The README also refers to the SDV ecosystem covering multi-table and time series data, but the SDGym example does not demonstrate a multi-table benchmark call. Treat multi-table benchmarking as unconfirmed from the README alone and check the documentation before designing around it. Second, the repository labels itself Development Status 2 - Pre-Alpha in its own badge. That is the project's own classification, and it signals that APIs and result formats can move between releases. The release history supports that reading: v0.14.3, v0.14.4 and v0.15.0 land within roughly five months, with a minor version bump in the most recent one. A benchmark whose numbers you cite in a decision document is a benchmark you may have to re-run after an upgrade, so pin the version you report against.
Where SDGym is the wrong tool
If your goal is to produce a synthetic dataset for a downstream model, SDGym is the wrong entry point. It scores generators; it does not replace one. The README's own framing points elsewhere for that job, directing readers to the SDV package as a fully integrated solution and describing SDGym as part of the same ecosystem rather than as the generation library. A second mismatch is scale. Because a run trains every listed synthesizer on every selected dataset, the cost grows with the product of the two lists, and the deep learning entries in the example are the expensive part. If you need a quick sanity check on one model over one small table, calling the synthesizer directly is faster than standing up the harness. A third mismatch is metric fit. The README says quality and privacy are measured through a variety of metrics but does not enumerate them, so if your acceptance criterion is a specific statistical test or a specific disclosure risk measure, confirm it exists in SDGym before you build a process around its output.
SDV synthesizers versus the standalone SDV package
The most direct comparison is not another benchmarking tool; it is the SDV library that SDGym draws its synthesizers from. The difference is in what each one returns. SDV, per the README's closing description, is offered as a fully integrated solution and a one-stop shop for synthetic data, covering data discovery and transformation, multiple model families across tabular, multi-table and time series data, and quality and privacy measurement. SDGym takes the models and the metrics and arranges them into a comparison across datasets and baselines. So the split is: SDV is where you build and ship a generator, SDGym is where you rank several of them under identical conditions. If you only ever use one synthesizer, SDGym's comparison machinery adds setup cost without adding an answer. If you are choosing between synthesizers, or defending a choice to someone else, the shared dataset and baseline structure is the part that does the work.
Licence and maintenance cost
The repository reports its licence as NOASSERTION, which means no standard licence identifier was detected for this project. That is not the same as saying there is no licence, and it is not something to resolve by inference. Read the LICENSE file in the repository before you depend on SDGym in anything you distribute, and if the terms are unclear, get your own legal advice rather than treating a package index listing as authoritative. On maintenance, the material shows an actively pushed repository that is not archived, with the most recent release dated 2026-08-31 and the most recent push shortly after. The practical cost is re-validation: because the project is self-described as pre-alpha and has shipped minor releases at a few months' interval, any benchmark number you publish should record the SDGym version, the synthesizer list and the dataset names alongside the result, so the run can be reproduced after an upgrade. Those three fields are the ones the README's own API surface makes explicit.
Editorial conclusion
Adopt SDGym if you already have a tabular synthesizer and need a repeatable way to compare it against SDV's models and baselines on the same datasets, with time, memory, quality and privacy recorded together. Do not adopt it if you only need synthetic data for a downstream task: SDGym scores generators, it does not hand you a production dataset, and the repository labels itself pre-alpha. Before you commit, verify three things: the exact licence text, since the repository reports NOASSERTION rather than a named licence; whether the datasets you intend to use sit inside the SDV collection or need the s3:// path for your own bucket; and whether the reported metrics cover the columns you actually care about, since the privacy and quality scores are only as meaningful as the data they run on.
Community notes