Gradient-Free-Optimizers: 23 Black-Box Search Algorithms Behind One Python API

The problem GFO addresses: search when you cannot differentiate

Plenty of objectives give you a score and nothing else. A simulation returns a number after it finishes. A trained model returns validation accuracy. A physical experiment returns a measurement. There is no gradient, and finite-difference approximations are either too noisy or too expensive to be worth the evaluations they consume. Gradient-Free-Optimizers (GFO) is a Python library built for exactly that situation. The README lists its intended uses as hyperparameter tuning, simulation optimization, feature selection, and engineering design, which is a fair summary of where black-box search shows up in practice. The audience is not the researcher who wants to implement a new acquisition function. It is the engineer who has an objective function, a parameter space, and a deadline, and who would rather not hand-roll a Nelder-Mead loop or wire up a Bayesian optimization stack from scratch. The library's pitch is that you define the objective, define the search space, and run. What you get in return is 23 algorithms behind one interface, and the ability to change strategy by changing one class name.

One interface, 23 strategies, and what the algorithm swap actually costs

The core abstraction is a single optimizer object. You construct it with a search space, then call search() with your objective and an iteration budget. The README's quick start uses HillClimbingOptimizer, but the feature list names local, global, population-based and sequential model-based families, and the repository topics mention Bayesian optimization, evolution strategies, particle swarm optimization, simulated annealing, random search, Nelder-Mead, and the Tree of Parzen Estimators. Swapping from hill climbing to Bayesian optimization is described as a one-line change, and the quick start supports that: the constructor signature is the same shape across optimizers. This is genuinely useful when you want to compare strategies without rewriting your problem. It is also where the first trade-off appears. A unified constructor means parameters that matter for one algorithm may be inert for another, and the README's emphasis on zero configuration means the defaults are doing the work. Defaults are a reasonable starting point, but a population-based method and a sequential model-based method have different sample-efficiency profiles, and the documentation is where you would need to look to find out what each one assumes about your objective. The README itself does not enumerate those assumptions; it points at the optimizers page of the Read the Docs site.

Mixed search spaces are the feature that separates GFO from scalar optimizers

Most optimizer libraries assume a flat vector of floats. GFO does not. The quick start defines a search space as a dictionary with a continuous range for x and a NumPy array for y, and the feature list extends that to categorical choices and SciPy distribution-backed dimensions. That combination is the part worth paying attention to, because real tuning problems are rarely all-continuous. A learning rate is continuous, a number of layers is an integer grid, an activation function is categorical, and a prior might be better expressed as a distribution than a fixed range. Encoding all of those as floats is possible but lossy, and it tends to produce optimizers that waste evaluations on invalid or duplicate points. GFO's approach is to let the search space carry the type information, so the optimizer proposes values that are valid for each dimension by construction. The SciPy distributions are listed as optional, which means the base install handles continuous ranges, discrete grids, and categoricals without pulling SciPy in. Constraints are handled separately, through constraint functions that the feature list says cause invalid regions to be avoided automatically. How that avoidance is implemented is not described in the README, and it matters: penalty-based and rejection-based constraint handling behave very differently when the feasible region is small.

Installing it, and what the extras actually pull in

The base install is a single command: pip install gradient-free-optimizers. The README states that only NumPy and pandas are required, with SciPy optional. That is a narrow dependency footprint for an optimization library, and it is the stated reason GFO is positioned as suitable for minimal environments, containers, and embedded systems. Three extras are documented. pip install gradient-free-optimizers[progress] adds tqdm for a progress bar. pip install gradient-free-optimizers[sklearn] adds scikit-learn, which the README ties to surrogate models, so this is the extra that sequential model-based optimizers such as Bayesian optimization and TPE are likely to need. pip install gradient-free-optimizers[full] installs everything. The practical consequence is that if you install the base package and reach for a surrogate-based optimizer, you may find a missing dependency rather than a working optimizer. The README does not spell out which of the 23 algorithms require which extra, so the safe move for a first evaluation is the [full] extra, then trim once you know which optimizer you are keeping. The library is MIT licensed, which permits commercial use and modification; the usual obligation is preserving the copyright notice and licence text in distributions, but that is a summary and not legal advice.

The memory system, and why caching is a design decision rather than a free win

GFO ships a built-in memory system, described as caching that prevents redundant evaluations and flagged as important for expensive objective functions such as ML models. The core concepts diagram shows search data being written to a history and read back as a warm start. This is one of the more consequential pieces of the design, and it cuts both ways. On the positive side, an optimizer that re-proposes a point it has already evaluated can return the cached score instead of retraining a model or rerunning a simulation, which is where the real cost sits. On the negative side, caching assumes the objective is deterministic. If your objective has stochastic elements, whether that is a random train/test split, a noisy simulator, or a GPU kernel with non-deterministic reduction order, a cached score can differ from what a fresh evaluation would return, and the optimizer will treat a stale number as ground truth. The README does not describe a cache-invalidation mechanism or a way to declare an objective non-deterministic. If your scoring function is noisy, that is the first thing to check in the user guide's memory page before trusting the results.

Where GFO is the wrong tool

The clearest case against GFO is when you have gradients. If your objective is a differentiable function of the parameters, gradient-based methods will generally reach a better solution in fewer evaluations, and no amount of algorithm variety in a gradient-free library compensates for throwing away derivative information. The second case is scale. GFO is a single-process library; the README describes an optimizer proposing parameters, an objective returning a score, and a history recording it. Nothing in the supplied material describes parallel or distributed evaluation, which matters when each objective call takes minutes and you have a cluster available. If your bottleneck is wall-clock time rather than the number of evaluations, a library with asynchronous or batch evaluation will beat a sequential loop regardless of which of the 23 algorithms you pick. The third case is when you want to understand or modify the surrogate model. GFO's value proposition is that you do not have to think about the optimizer. If your problem requires a custom kernel, a specific acquisition function, or a constrained formulation that the built-in constraint functions do not express, the abstraction becomes an obstacle rather than a convenience.

How GFO differs from a full Bayesian optimization framework

The obvious alternative category is a dedicated Bayesian optimization framework such as scikit-optimize, which GFO itself lists as an optional dependency for surrogate models. The difference in approach is scope. A framework like scikit-optimize is organized around the surrogate model: you choose a regressor, an acquisition function, and an acquisition optimizer, and you tune that pipeline. GFO inverts the emphasis. The README frames the library as a unified interface to 23 algorithms with zero configuration and sensible defaults, and treats the surrogate model as an implementation detail supplied by scikit-learn when the chosen optimizer needs one. That means GFO is easier to start with and easier to switch strategies in, while scikit-optimize gives you more control over the part that usually determines sample efficiency. A second alternative is writing the search yourself. For a small discrete space, random search or a grid is a few lines and has no dependency cost at all. GFO earns its place when the space is mixed, the budget is large enough that strategy matters, and you want the option to change strategy without rewriting the loop.

Maintenance, versioning, and what to check before you depend on it

The repository is not archived and the last push date is recent, with three releases in the supplied list: v1.13.0, v1.12.0, and v1.11.1, spaced roughly a month apart. That cadence suggests active maintenance rather than a frozen project. The version numbers are pre-2.0, which in practice means minor releases can carry behaviour changes, so pinning a version in your requirements file is the cheap insurance. The MIT licence keeps the integration cost low and imposes no copyleft obligation on your own code. The upgrade cost is the part that is hard to estimate from the outside. Because all optimizers share one constructor, a change to the shared interface touches every algorithm, and a change to the memory system touches every run. Before adopting, verify three things in your own environment: that the specific optimizer you intend to use converges on your objective within a realistic n_iter budget, that the optional extra it needs is installed, and that your objective is deterministic enough for the caching layer to be safe. Those are checks you run, not claims this review can make for you.

Editorial conclusion

Adopt Gradient-Free-Optimizers when you have a black-box objective, a mixed search space of continuous, discrete, categorical or distribution-backed dimensions, and no gradient to follow. Skip it when your objective is differentiable, when you need a distributed or parallel evaluation loop, or when you want a surrogate-model stack you can inspect and replace. Before committing, verify two things in your own environment: that the optimizer you pick converges on your objective within a realistic n_iter budget, and that the memory system's caching does not collide with an objective that is non-deterministic between calls.

Official sources

1. gfo-project/Gradient-Free-Optimizers on GitHub
2. License: MIT
3. Project website
4. README
5. Releases