frugally-deep: Running Keras Models in C++ Without Linking TensorFlow

The Deployment Gap frugally-deep Fills

Training happens in Python with Keras and TensorFlow. Serving the result in a C++ application usually means linking TensorFlow into that application, which pulls in a large dependency and a large binary. frugally-deep takes a different route: the README states it re-implements a small subset of TensorFlow, specifically the operations needed to support prediction. The Python side stays where it is; only the forward pass moves to C++. The intended user is an engineer shipping a model inside a C++ program, on a desktop, a server, or an edge device where the TensorFlow runtime is unwelcome. The library is header-only and depends only on FunctionalPlus, Eigen, and nlohmann/json, all of which are also header-only. That dependency shape is the whole pitch: no shared library to install, no ABI to match against a TensorFlow build.

How the Conversion and Inference Pipeline Works

The workflow has three stages. First, build, train, and test the model in Keras as usual, then save it to a single file with model.save('....keras'). The README requires image_data_format to be channels_last, which it notes is the default with the TensorFlow backend, and states that models created with a different image_data_format and other backends are not supported. Second, run keras_export/convert_model.py to translate the model into the frugally-deep file format, a JSON file. Third, load that JSON in C++ with fdeep::load_model(...) and call model.predict(...) to invoke a forward pass. The mechanism worth noting is the embedded test case. According to the README, when using convert_model.py a test case with input and corresponding output values is generated automatically and saved along with the model, and fdeep::load_model runs this test to verify that a forward pass in frugally-deep matches the Keras result. That is a self-check at load time rather than a guarantee established once at conversion time, and it is the main reason a silent numerical mismatch is unlikely to go unnoticed.

Layer Coverage Is Broad, With Named Exceptions

The supported list is long enough to be worth reading before you commit to the library. It covers the arithmetic and merge layers (Add, Concatenate, Subtract, Multiply, Average, Maximum, Minimum, Dot), convolutional layers across one, two, and three dimensions including SeparableConv2D, DepthwiseConv1D and DepthwiseConv2D, and the transposed variants, plus pooling, normalization (BatchNormalization, LayerNormalization, RMSNormalization, GroupNormalization, UnitNormalization), attention (Attention, AdditiveAttention, MultiHeadAttention, GroupedQueryAttention), and recurrent layers (LSTM, GRU, SimpleRNN, Bidirectional, ConvLSTM1D/2D/3D). Nested models, residual connections, shared layers, multiple inputs and outputs, variable input shapes, and custom layers via factory functions passed to load_model are all listed as supported. The unsupported list is equally explicit: Lambda, Hashing, HashedCrossing, MelSpectrogram, STFTSpectrogram, StringLookup, TextVectorization, stateful recurrent layers, and temporal models. The README links a separate FAQ entry explaining why Lambda layers are not supported. If your preprocessing lives inside the model as a TextVectorization layer, this library is the wrong tool, and no amount of C++ integration work changes that.

Getting a Model Running: The Actual Commands

The README gives a complete minimal example. On the Python side, after training and saving to keras_model.keras, the conversion is a single command: python3 keras_export/convert_model.py keras_model.keras fdeep_model.json. On the C++ side, the include is #include <fdeep/fdeep.hpp>, the model is loaded with fdeep::load_model("fdeep_model.json"), and prediction is called with a tensor built from a tensor_shape and a std::vector<float>, for example fdeep::tensor(fdeep::tensor_shape(static_cast<std::size_t>(4)), std::vector<float>{1, 2, 3, 4}). The result is printed with fdeep::show_tensors. Build requirements are a C++14-compatible compiler, with GCC 4.9, Clang 3.7 (libc++ 3.7), and Visual C++ 2015 named as the minimums, plus Python 3.9 or higher, TensorFlow 2.21.0, and Keras 3.14.0 on the conversion side. The README notes these are the tested versions and that somewhat older ones might work too. Installation guides for different setups are in INSTALL.md rather than the README, so the top-level document alone does not tell you how to vendor the headers.

The Single-Core Design Choice and What It Costs

The README is unusually direct about the CPU story: the library utterly ignores even the most powerful GPU in your system and uses only one CPU core per prediction. It then argues the trade-off, stating that it is quite fast on one CPU core and that you can run multiple predictions in parallel, thus utilizing as many CPUs as you like to improve overall throughput. That is a throughput model, not a latency model. If your application must answer a single request as fast as possible, parallel predictions across requests do not help that one request. The library also avoids temporarily allocating potentially large chunks of additional RAM during convolutions by not materializing the im2col input matrix, which is a memory-conscious choice relevant to constrained devices. There is no GPU path and no documented multi-threaded single-prediction path. Whether one core is enough depends entirely on your model size and your latency budget, and the README supplies no benchmark numbers, so that question has to be answered on your own hardware.

Where the Approach Breaks Down

The conversion step is the first failure mode. A model that trains fine in Python can still fail conversion if it uses any unsupported layer, and the failure surfaces in Python, before any C++ is written. The second is the backend constraint: channels_last with the TensorFlow backend is required, so a model trained under a different image_data_format needs retraining or conversion of its weights, not just a flag change. The third is the load-time test. It verifies one generated input and output pair, which is evidence that the graph executes and matches Keras on that sample, not a proof of numerical equivalence across the input space. Floating-point differences between TensorFlow and a hand-written C++ implementation can in principle appear on inputs far from the test case, and the README does not claim otherwise. Finally, the unsupported list rules out a common production pattern: models that embed text vectorization or string lookup. Those models need the preprocessing moved out of the graph and reimplemented in C++, which is work the library does not do for you.

Compared With Linking TensorFlow Directly

The obvious alternative is to link the C++ application against TensorFlow and use its C or C++ API to load and run the same saved model. That path supports the full op set, including Lambda layers, TextVectorization, and stateful recurrent layers, and it can use a GPU. The difference in approach is where the complexity sits. Direct TensorFlow linking keeps the model format and the execution engine identical to training, so there is no conversion step and no separate op implementation to diverge from Keras. frugally-deep moves that complexity into a converter and a re-implementation, and buys a much smaller binary, as the README claims, plus a dependency set of three header-only libraries instead of the TensorFlow runtime. Neither approach is a superset of the other. If your model sits inside the supported layer list and binary size or deployment simplicity is the binding constraint, frugally-deep removes a large dependency. If your model needs an op outside that list, or needs a GPU, direct linking is the only one of the two that will run it.

Maintenance, Versions, and the MIT Licence

The repository is not archived and the last push is recorded as 2026-05-06, with v0.20.0 released 2026-05-03, preceded by v0.19.5 and v0.19.4 in the same week. That release cadence suggests active maintenance, though the material here does not describe what changed between those versions, so pinning a specific tag and reading its release notes is the only way to know. The pinned dependency versions matter for upgrades: TensorFlow 2.21.0 and Keras 3.14.0 are the tested pair, and since Keras 3 changed model saving to the .keras format, an older Keras 2 codebase will need its save and conversion path adjusted. The library is MIT licensed, which permits commercial use and modification; the headers you vendor into your own source tree carry that licence, and the usual obligation is to preserve the copyright and permission notice. The README does not discuss the licences of FunctionalPlus, Eigen, or nlohmann/json, so check those separately before shipping. This is not legal advice; confirm obligations with your own counsel if the distinction matters to you.

Editorial conclusion

Adopt frugally-deep if your model uses only the layers in its supported list, you can accept a separate Python conversion step, and you want a small C++ binary that does not link TensorFlow. Do not adopt it if your model relies on Lambda, TextVectorization, StringLookup, MelSpectrogram, STFTSpectrogram, Hashing, HashedCrossing, or stateful recurrent layers, since the README lists those as unsupported. Before committing, run keras_export/convert_model.py on your actual saved .keras file and check that fdeep::load_model passes the automatically generated test case, because that test is the only confirmation the README offers that C++ output matches Keras.

Official sources

1. Dobiasd/frugally-deep on GitHub
2. Issues
3. License: MIT
4. README
5. Releases