TransmogrifAI: AutoML for Structured Data on Spark, Typed in Scala

The problem TransmogrifAI targets: hand-tuned Spark ML pipelines

Spark ML gives you transformers, estimators and a Pipeline, but it does not tell you which encodings to apply to a categorical column, which features to drop, or which classifier to pick. In practice that work is a long loop of manual experimentation, and the resulting code is usually a chain of StringIndexer, OneHotEncoder and VectorAssembler calls that is hard to reuse across datasets because nothing about it is typed. TransmogrifAI's stated goal is to compress that loop. The README frames it as accelerating machine learning developer productivity through automation, with an API that enforces compile-time type-safety, modularity and reuse. The audience is explicit: teams that want production machine learning applications in hours rather than months, and developers who want to build models without a machine learning PhD. The library is written in Scala and runs on Apache Spark, so the audience is narrower than the marketing line suggests. You need a JVM build, a Spark cluster or local Spark session, and comfort with Scala's type system. If your data is images, audio or free text, nothing in the supplied material suggests TransmogrifAI is aimed at you. It is built around structured data and the feature types that go with it.

How the pipeline works: features, transmogrify, sanityCheck, model selection

The mechanism is visible in the README's Titanic example, and it is a four-stage flow. First, data is read into a Spark DataFrame through a reader such as DataReaders.Simple.csvCase[Passenger], which maps rows onto a case class. Second, FeatureBuilder.fromDataFrame[RealNN](passengersData, response = "survived") splits the frame into a response feature and a sequence of predictor features, each carrying a type from com.salesforce.op.features.types. That typing is the part that differs from plain Spark ML: a feature is not an untyped column reference, it is a value with a known kind, and the compiler checks what you do with it. Third, predictors.transmogrify() runs automated feature engineering, and survived.sanityCheck(featureVector, removeBadFeatures = true) runs automated feature validation and selection, with the option to drop features that fail the checks. Fourth, BinaryClassificationModelSelector().setInput(survived, checkedFeatures).getOutput() produces the estimator that will be selected. The pieces are then bound together by OpWorkflow, which takes the input dataset and the result features and calls train(). The README's printed summary shows what the selector does at runtime: it evaluated Logistic Regression and Random Forest models with 3 folds and an AuPR metric, tried 3 Logistic Regression configurations and 16 Random Forest configurations, and selected a Random Forest with maxDepth 12, numTrees 50 and gini impurity. The same summary reports hold-out and training metrics plus top model insights computed by correlation, for example sex = "female" at 0.517 and sex = "male" at -0.510. Those numbers come from the README's own example output on the Titanic dataset, not from an independent evaluation, and they should be read as a demonstration of the reporting format rather than as a claim about your data.

Getting it running: the Scala and Spark versions you are pinned to

The dependency coordinates are published under the group com.salesforce.transmogrifai, and the README's Maven Central badge points at the artifact transmogrifai-core_2.11 at version 0.7.0. The Scala suffix matters: the published artifact shown is built for Scala 2.11, and the README badges Spark 2.4 and Scala 2.11. The Javadoc badge links to version 0.7.0 of the same artifact, so 0.7.0 is the version the documentation is aligned with. The README does not print an sbt or Maven snippet in the supplied text, so the exact dependency line has to be taken from Maven Central rather than from this article; what is verifiable is the group, artifact and version. The quick start example imports com.salesforce.op._, com.salesforce.op.readers._, com.salesforce.op.features._, com.salesforce.op.features.types._, and com.salesforce.op.stages.impl.classification._, and builds an implicit SparkSession with SparkSession.builder.config(new SparkConf()).getOrCreate(). From there the API surface is small: DataReaders, FeatureBuilder, transmogrify, sanityCheck, the model selector, and OpWorkflow with setInputDataset and setResultFeatures. That small surface is the point. The cost is that the surrounding versions are not negotiable in the way a pure library's would be. Spark 2.4 and Scala 2.11 are old relative to current Spark and Scala releases, and the README gives no indication of a 2.12 or 2.13 build. If your platform has already moved past those versions, TransmogrifAI is a migration problem before it is a modelling tool.

Version cadence and the maintenance question

The recent release list is short and the gaps are wide. 0.6.0 landed in July 2019, 0.6.1 in September 2019, and 0.7.0 in June 2020. The repository is not archived and the last push timestamp is 2026-06-02, so the project has not been formally abandoned, but the release history shows that the last tagged version predates the last commit by roughly six years. That pattern is common in libraries that are stable and internally driven: the code moves, the releases do not. For an adopter it has a concrete consequence. There is no 0.8.0 to point at, so anything you build is pinned to 0.7.0 artifacts or to a source build from master, and the two are not the same thing. A source build against master inherits whatever Spark and Scala versions master is on, which the supplied material does not state. Treat the version you depend on as a decision you own, not one the project will make for you. There is also a documentation surface to account for: a ReadTheDocs site at docs.transmogrif.ai with a stable version, Javadoc on javadoc.io, and a Gitter channel linked from the README. Those are the places to check before assuming a behaviour, because the README itself is a single worked example.

Where it breaks down: the Titanic example hides the hard parts

The Titanic dataset is small, clean and famous, and the README's example output shows a workflow that runs end to end on it without intervention. Real structured data rarely behaves that way. The sanityCheck stage is the one to watch. It validates features and can remove bad ones, and the README passes removeBadFeatures = true, which means the feature vector that reaches the model selector is not the feature vector you wrote. When that removal is driven by a check you did not configure, you can lose a column that carries real signal and only find out from the model summary. The README does not document the checks in the supplied text, so the behaviour has to be read from the project documentation. The second pressure point is the model selector. In the example it evaluated 3 Logistic Regression and 16 Random Forest configurations with 3 folds, which is a bounded search over a fixed grid of Spark ML models. That is automation of a search, not a guarantee that the winning configuration is good. If your problem needs a model family outside that grid, or a custom metric, the selector is not the tool. Third, the library is opinionated about input shape: FeatureBuilder.fromDataFrame expects a DataFrame and a named response column, and the typed feature system expects columns that map onto the provided feature types. Wide, messy schemas with hundreds of heterogeneous columns are exactly where automation is most valuable and also where a typed builder is most likely to need hand-written feature definitions, which puts you back in Spark ML territory with extra indirection.

How it differs from Spark ML and from scikit-learn

The closest comparison is Spark ML, because TransmogrifAI is built on top of it and the README's selected model is a Spark ML Random Forest with parameters such as maxDepth, numTrees and impurity. The difference is where the decisions live. In Spark ML you assemble a Pipeline by hand: you choose the indexers, the encoders, the assembler and the estimator, and the pipeline is only as reusable as your discipline makes it. In TransmogrifAI you declare typed features and let transmogrify, sanityCheck and the model selector generate that pipeline, then wrap it in an OpWorkflow. The trade is control for speed. If you already know which encoding and which model you want, Spark ML is the shorter path, because you skip the automation layer and the version constraints it brings. The other common comparison is scikit-learn, and here the split is architectural rather than stylistic. scikit-learn is Python, in-process and single-machine by default; TransmogrifAI is Scala on the JVM and distributed through Spark. A team with a Python stack and a few hundred thousand rows will find scikit-learn plus a gradient boosting library faster to iterate on, with a far larger ecosystem of encoders and model families. A team with terabytes of structured data already in Spark and a Scala or Java codebase gets the opposite answer, because moving that data into Python is the expensive part. The honest framing is that TransmogrifAI competes with hand-written Spark ML pipelines, not with the Python ecosystem.

Licence and the cost of staying on 0.7.0

TransmogrifAI is released under BSD-3-Clause, as the README's licence badge and the repository metadata both indicate. That is a permissive licence: it allows use, modification and redistribution, including in proprietary products, provided the copyright notice and licence text are retained and the names of the copyright holder and contributors are not used to endorse derived products without permission. The clause that catches people is the third one, the non-endorsement clause, which matters if you plan to market a product built on it. This is a description of the licence text, not legal advice; check the LICENSE file in the repository and your own counsel before shipping. On upgrade cost, the arithmetic is straightforward. The last release is 0.7.0 from June 2020, and the README badges Spark 2.4 and Scala 2.11. Every year you stay on those versions, the gap to your platform's current Spark and Scala widens, and the work of moving off TransmogrifAI grows with it. There is no upgrade path documented in the supplied material beyond the tagged releases. Budget for the possibility that the exit is a rewrite of the feature definitions into Spark ML or another framework, not a version bump. The mitigation is to keep the typed feature definitions and the OpWorkflow boundary thin, so the modelling logic underneath can be replaced without touching the rest of the application.

Editorial conclusion

Adopt TransmogrifAI if your structured data already lives in Spark, your build is on Scala 2.11, and you want feature engineering, validation and model selection generated from a typed feature schema rather than written by hand. Do not adopt it for unstructured data, for Python-first teams, or for anything that needs Scala 2.12 or 2.13. Before committing, verify that your Spark version matches the 2.4 line the README badges, that the transmogrify and sanityCheck stages produce a feature vector your data can actually support, and that the 0.7.0 artifacts resolve against your own dependency graph.

Official sources

1. License: BSD-3-Clause
2. Project website
3. README
4. Releases
5. salesforce/TransmogrifAI on GitHub