Lightwood: JSON-AI Pipelines That Generate Their Own Python

The boilerplate problem Lightwood targets

Most tabular machine learning work is not modelling. It is deciding how to encode a date column, how to handle a high-cardinality category, which rows to hold out, and how to keep all of that consistent between the training run and the serving path. Lightwood's stated goal is to let users focus on what they want to do with their data rather than write that scaffolding. The README frames the framework as covering the data science and machine learning life cycle so that effort goes into "the parts of a model that are truly unique and custom".

The intended user is someone who has a pandas DataFrame and a column they want to predict, and who does not want to hand-assemble encoders, splits and a PyTorch training loop before seeing whether the problem is tractable. Lightwood also targets a second group: people who do want to customize, but want the customization expressed as an override on a generated pipeline rather than as a blank file. The README lists numbers, dates, categories, tags, text, arrays and various multimedia formats as supported data types, and notes a time-series mode for problems with between-row dependencies. That combination (mixed column types plus optional temporal structure) is where the framework is aimed.

Three stages, and the objects that own each one

Lightwood splits the pipeline into preprocessing and cleaning, feature engineering, and model building and training. Each stage has a named object, and the naming is consistent enough that you can reason about the architecture from the README alone.

Preprocessing is handled by a cleaner and a splitter. For each column, Lightwood runs what the README calls a brief statistical analysis to infer the suspected data type, then generates JSON-AI syntax from that inference. The cleaner applies type-appropriate cleaning and the splitter produces train, dev and test partitions.

Feature engineering is handled by encoders. An encoder is the rule that turns preprocessed data into a numerical representation a model can consume. The README draws a distinction worth keeping in mind: encoders are either rule-based (a fixed transformation, such as normalized numeric data) or learned (a representation produced after training, with the [CLS] token in a language model given as the example). Encoders are assigned per column based on the inferred data type, and the assignment can be overridden at the column level or at the data-type level. All encoders inherit from BaseEncoder.

The final stage is the mixer. A mixer takes encoded feature data and outputs a prediction for the target. Lightwood ships default mixers and supports user implementations that inherit from BaseMixer. The README says the project predominantly uses PyTorch-based approaches but can support other models. That phrasing is doing real work: PyTorch is the well-trodden path, and anything else is a claim the documentation does not demonstrate with an example.

JSON-AI is the actual interface, not the Python API

The four functions in the quick-start example are the whole high-level surface: json_ai_from_problem, code_from_json_ai, predictor_from_code and predictor.learn. The data flow is one-directional and inspectable at every hop.

You start with a ProblemDefinition, which in the minimal case is a dictionary containing a single key, target, naming the column to predict. json_ai_from_problem takes the DataFrame and that definition and returns a JSON-AI object describing the pipeline. The README notes you can print json_ai.to_json() to see the syntax. code_from_json_ai then turns that object into Python source, which you can also print. predictor_from_code instantiates a Predictor from that source, and predictor.learn(df) trains it end to end from raw data.

This is the design decision that separates Lightwood from an AutoML library that returns a fitted model object. The artifact in the middle is source code you can read and edit. The README states that the syntax outlines the specifics of each step and that users may override defaults, for example changing the type of a column, or entirely replace a step with their own method, giving a random forest as a predictor as the example. The consequence is that the generated code is the contract. If you edit it, you own it, and regenerating from JSON-AI will discard your edits.

Getting it running, and the version constraint you will hit

Installation is a single command, with the README noting that pip may be the correct binary depending on the environment:

pip3 install lightwood

For a development checkout the README is more specific. The Python version must be in the range >=3.8 and < 3.11. That upper bound is the first thing to check against your interpreter, because 3.11 and later are excluded by the stated range. The remaining steps are to clone the repository, then run pip install -r requirements.txt followed by pip install -r requirements_image.txt from inside the cloned directory. The second requirements file is the one that pulls in the multimedia dependencies, which explains why it is separate from the core install.

After that, the README suggests adding the clone to your Python path, showing export PYTHONPATH='/where/you/cloned/lightwood':$PYTHONPATH appended to ~/.bashrc as the example. Finally, it gives a way to check the checkout: python -m unittest discover tests, run from the directory where you cloned the project. If python resolves to a 2.x interpreter on your machine, the README says to use python3 and pip3 instead.

One thing the README does not give is a pinned set of dependency versions, so the reproducibility of a dev environment depends on requirements.txt at the commit you checked out.

Where the declarative approach costs you

The generated-code model has a failure mode that is easy to walk into. code_from_json_ai produces Python that represents the pipeline at the moment of generation. Any hand edit to that file is invisible to the JSON-AI object it came from. If you later regenerate, for example because you changed the target column or upgraded Lightwood, the regeneration path does not know about your edits. The README's own framing supports this reading: the JSON-AI object is what generates the code, and the code is what you instantiate a Predictor from. There is no stated round trip from edited code back to JSON-AI.

A second constraint is the data boundary. Lightwood works with pandas DataFrames, and the quick-start example loads a CSV to get there. If your pipeline is streaming, or your features live in a feature store, or your records arrive as nested JSON that does not flatten cleanly, the framework's entry point is the wrong shape and you would be writing the adapter yourself.

The time-series mode is the part I would treat most cautiously. The README mentions that it exists for problems with between-row dependencies, and that is the entirety of what the supplied material says about it. There is no example, no configuration key and no description of how the splitter behaves when rows are not independent. Anyone whose problem is genuinely temporal should read the linked documentation before assuming the mode fits, because split correctness is the difference between a valid evaluation and a leak.

The alternative: a hand-written PyTorch pipeline

The obvious comparison is not another AutoML tool. It is the thing Lightwood is trying to save you from writing: a PyTorch training script with your own Dataset class, your own encoding of categorical columns, and your own train/validation split. The difference in approach is where the decisions live. In a hand-written pipeline, the encoding choices are expressed as code that you wrote and can trace line by line, and there is no intermediate representation to keep in sync. In Lightwood, the decisions are expressed as JSON-AI syntax that generates that code for you, which is faster to reach a first result and slower to audit if you did not write the generator.

The trade is asymmetric. A hand-written pipeline costs you an afternoon before you see a number, and it never surprises you with generated code you did not read. Lightwood costs you almost nothing to reach a first result, and it hands you a file whose contents you are responsible for the moment you edit it. The README's own BYOM section points at tutorials for custom cleaner, custom splitter, custom explainer and custom mixer, which is the intended path for the second case: keep the generated structure, replace the parts you care about by inheriting from BaseEncoder or BaseMixer rather than editing the output. That is a meaningfully different workflow from editing the generated Python, and it is the one the project seems to want you to use.

Licence, release cadence and what that means for upgrades

Lightwood is GPL-3.0. That is a copyleft licence, and it is worth checking against how you plan to distribute whatever you build, because linking or deriving from GPL-3.0 code carries obligations that permissive licences do not. This is a factual note about the licence identifier in the repository, not legal advice; if the distinction matters to your product, get it reviewed rather than reading it off a metadata field.

The release history shows a monthly cadence: v25.7.5.1 in late July 2025, v25.9.1.0 in early September, v25.12.1.0 in early December. The version strings encode a date, which makes it easy to see how far behind you are, and also means there is no long-term support line visible in the material. Dependencies are not pinned in the README, and the Python range excludes 3.11 and above as of the current documentation. For a project that generates source code as its primary artifact, a fast cadence raises a specific question: does the shape of the generated code change between releases? The supplied material does not answer that, and it is the first thing I would check by generating code from the same DataFrame on two versions and diffing the output.

Maintenance cost is therefore concentrated in two places: keeping the Python interpreter inside the supported range, and re-validating the generated pipeline whenever you upgrade. If you never upgrade, the second cost disappears, but so does any fix for a data type or encoder that does not behave.

Editorial conclusion

Adopt Lightwood if you want a working baseline on tabular or time-series data in a few lines, and you expect to edit the generated pipeline rather than accept it. Do not adopt it if you need a stable Python API across versions, if your data does not fit a pandas DataFrame, or if GPL-3.0 is incompatible with how you ship. Before committing, verify the Python version range against your interpreter, check that the generated code from code_from_json_ai is something your team is willing to maintain, and confirm how the JSON-AI syntax changes between releases, because the version numbers move on a monthly cadence.

Official sources

1. Issues
2. License: GPL-3.0
3. mindsdb/lightwood on GitHub
4. README
5. Releases