Multimodal-Toolkit teaches a transformer to read your table alongside your text

A combining module, not a new architecture

The description says it plainly: a toolkit for incorporating multimodal data on top of text data for classification and regression tasks, using HuggingFace transformers as the base model for text features. The important word in that sentence is on top of. You are not replacing the transformer, and you are not building a new backbone. The toolkit adds a combining module that takes the outputs of the transformer in addition to categorical and numerical features, and produces rich multimodal features for the downstream classification and regression layers. The training story is also unusual for a library of this kind: given a pretrained transformer, the parameters of the combining module and the transformer are trained based on the supervised task. So the text encoder is not frozen, and the whole thing is fine tuned end to end on your labels. The audience is an applied machine learning team with a text model and a table, a task that is common in review classification, price prediction and any problem where a free text field sits next to numeric columns, and the value of the library is that it gives you the fusion layer and the training loop rather than just a paper.

Seven fusion strategies, from concat to gated summation

The Included Methods table is the heart of the package, and it is worth reading in full because the choice of strategy is a real modelling decision. text_only uses just the text columns as processed by a HuggingFace transformer before the final classifier layers, and the README says it is essentially equivalent to HuggingFace's ForSequenceClassification models, which makes it the baseline to beat. concat concatenates the transformer output, the numerical features and the categorical features all at once. mlp_on_categorical_then_concat applies an MLP to the categorical features first and then concatenates. individual_mlps_on_cat_and_numerical_feats_then_concat puts separate MLPs on each feature family before concatenating. mlp_on_concatenated_cat_and_numerical_feats_then_concat puts one MLP on the concatenated table and then concatenates with the transformer output, and it is the only row marked as requiring both categorical and numerical features. attention_on_cat_and_numerical_feats uses an attention based summation of the transformer outputs and the table features, queried by the transformer outputs. gating_on_cat_and_num_feats_then_sum does a gated summation, and the README credits the inspiration to the paper Integrating Multimodal Information in Large Pretrained Transformers, noting that it performs the mechanism for each token. The last, weighted_feature_sum_on_transformer_cat_and_numerical_feats, learns a weighted feature-wise sum per feature dimension. So the library is really a menu of fusion operators over the same two inputs.

Installing it, and the version gap you inherit

The installation is a single pip command.

pip install multimodal-transformers

The multimodal specific code lives in the multimodal_transformers folder, and the README states plainly that the code was developed in Python 3.7 with PyTorch and Transformers 4.26.1. That sentence is the most important line in the document for anyone evaluating the project today, because the setup.py dependency list has moved a long way. The current pins ask for torch 2.0.1 or newer, transformers 4.40.1 or newer, accelerate 0.29.3 or newer, numpy around 1.26.4, pandas around 2.2.2, scipy 1.13.0, scikit-learn around 1.5.1, plus networkx, sacremoses, tqdm, importlib-metadata and pytest. The package name in the metadata is multimodal_transformers while the PyPI install name is multimodal-transformers, with the version read out of the package's __init__ file. The top level listing shows the standard research repository shape, with main.py as the training entry point, evaluation.py, util.py, a multimodal_transformers/ package, datasets/, docs/, examples/ with an inference script and a classification notebook, tests/, and the readthedocs configuration. The documentation site is at multimodal-toolkit.readthedocs.io and there is a Colab notebook linked from the README for seeing the modules come together.

Running the example, and the column info file that drives it

There is a working example, and it is configured with a JSON file rather than with code.

$ python main.py ./datasets/Melbourne_Airbnb_Open_Data/train_config.json

The same run can be expressed with command line arguments instead.

$ python main.py \
    --output_dir=./logs/test \
    --task=classification \
    --combine_feat_method=individual_mlps_on_cat_and_numerical_feats_then_concat \
    --do_train \
    --model_name_or_path=distilbert-base-uncased \
    --data_path=./datasets/Womens_Clothing_E-Commerce_Reviews \
    --column_info_path=./datasets/Womens_Clothing_E-Commerce_Reviews/column_info.json

The column_info.json file is the piece of machinery worth understanding. The README says main.py expects a json file detailing which columns in a dataset contain text, categorical or numerical input features, and it also expects a path to the folder where the data is stored as train.csv and test.csv, and val.csv if given. The arguments are described further in multimodal_exp_args.py. So the wiring for a new dataset is a data folder plus a column description, which is a much better interface than subclassing a model, and it is also where the errors live, since a column mislabelled as categorical or numerical produces a shape mismatch or a silently useless feature rather than an obvious failure. The example shown uses distilbert-base-uncased as the text backbone, and the combine_feat_method argument takes the same strategy names as the Included Methods table, so the two are consistent.

Three datasets and the tasks they are meant for

The repository includes datasets that carry both text and rich tabular features, and each one is there to demonstrate a different task. The Women's Clothing E-Commerce Reviews dataset is for recommendation prediction as a classification problem, the Melbourne Airbnb Open Data dataset is for price prediction as a regression problem, and the PetFindermy Adoption Prediction dataset is for pet adoption speed prediction as a multiclass classification problem. The README calls the first two out as included and then lists all three, so treat the list as the set the repository carries. The reason these three are useful is that they are not artificially constructed. Each has a free text column next to numeric and categorical columns that a business actually has, which means the fusion strategies can be judged on a problem where the table genuinely carries signal, not on a synthetic join. The Melbourne one also has a natural baseline in the literature, since predicting a nightly price is a common enough task that the result is comparable to other work, and the clothing reviews one has the property that the text and the product attributes are close to redundant, which is exactly where a naive concat can fail to add anything. The README closes the methods section by pointing at a simple baseline model, and in practice taking the categorical and numerical features through a plain baseline is what tells you whether the more elaborate fusion operators are earning their keep.

The maintenance picture, and the licence that needs checking

Two things deserve a careful look before you depend on this. The first is the release cadence. The project is not archived, and the last push was on 2026-05-04, so the code is being touched, but the release list shows v_0.4 on 2024-09-24, v_0.3.1 on 2023-11-14 and v_0.2-alpha in March 2023. So the published versions are old relative to the commits, and the recent work is not obviously shipping as tagged releases. For a research library that is usually tolerable, since you can point your dependency at a commit, but it does mean you are reading the source rather than a published artefact when you install it today. The second is the licence, and here the repository metadata and the packaging metadata disagree. The project is recorded as Apache-2.0, and the README-level information points to that, while the setup.py in the repository still declares license="MIT" with the author listed as Ken Gu and the maintainer as Akash Saravanan, and the project URL pointing at the older georgianpartners organisation. A dual licensing question like that is not something to guess at, so read the LICENSE file at the root of the repository and settle it with whoever owns the code before you ship it inside a product. The project is a research tool with a narrow remit, and the combination of a slow release line and an unclear licence is what you would weigh against the convenience of the fusion layer it saves you writing.

Editorial conclusion

Multimodal-Toolkit suits a team that already has a text model and a table and wants to try fusing them without writing the fusion layer, since the eight documented strategies range from a plain concat to attention and gated summation, and the package ships the classification and regression heads and a training entry point. It is the wrong tool if you need a current model, because the code was developed in Python 3.7 with PyTorch and Transformers 4.26.1 while the setup metadata now requires torch 2 and transformers 4.40 or newer, a gap worth testing before you commit. Before you adopt it, run the text_only strategy as a baseline on your data, because the README itself points at a simple baseline model, and check the licence, since the repository is recorded as Apache-2.0 while the setup.py metadata still says MIT.

Frequently asked questions

Official sources

1. georgian-io/Multimodal-Toolkit on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases