After we fixed our own bug, CMA-ES ranked first and two nature-inspired optimisers proved centre-biased

Optimisers named after slime moulds, wolves, bees and ants are often reported to beat standard methods. Our first preregistered benchmark seemed to agree: the biologically labelled group had the better mean rank, 5.04 against 6.20. Tracing a poor CMA-ES score showed a bug in our own code, and several other methods had a related fault. We kept that run, fixed the box handling for every method, wrote a new preregistration and ran again on seven test functions in 10 dimensions. After the fix, CMA-ES ranked first on the shifted functions with a mean rank of 2.00, and the best method with a biological label, ACOR, ranked second at 3.14.c1 Moving the optimum away from the centre cost the slime mould method 12.1 and the grey wolf method 13.6 decades of median error, while ACOR, IWO and PSO lost 0.0006, 0.07 and 0.42 decades.c2 Without random search the plain group's mean rank was 3.98 against 5.54 for the biologically labelled group; with random search counted as plain the groups were 5.45 and 5.54, a gap that sat inside the label-permutation range (p 0.45).c3 The group contest does not settle much in either direction. The method-level results do.

A metaheuristicDefinition: metaheuristic optimisation A general recipe for searching for the minimum of a function by sampling and updating a set of candidate points, used when the function gives no gradient or has many local minima. Excludes exact methods that come with a proof of finding the minimum. full definition is a general recipe for searching a function for its minimum by sampling and updating candidate points. Sörensen [1] described a flood of "novel" metaheuristics built on metaphors of natural or man-made processes and argued that the trend threatens to lead the field away from scientific rigour. Camacho-Villalón, Dorigo and Stützle [2] took six widely used metaphor-based methods for continuous problems apart into components, the grey wolf optimiser among them, and reported that none of them proposes a new idea; one section of their paper is titled "The grey wolf optimizer is PSO".

Kudela [3] pointed to a defect in how such methods are tested. Several common test functionsDefinition: test function optimisation benchmarking A formula with a known minimum, chosen to be easy or hard in a known way, on which optimisers are compared by how close they get. Excludes real engineering problems, whose minimum is unknown. full definition have their minimum at the centre of the feasible set, and a method whose update rule has a centre biasDefinition: centre bias evolutionary computation benchmarking, after Kudela A tendency of a method's update rule to pull candidates toward the centre of the search box, which helps on any function whose minimum happens to sit there. Excludes good performance that survives when the minimum is moved. full definition solves those functions easily. Comparisons between such a method and one without the bias on centred functions say little.

The no free lunchDefinition: no free lunch optimisation theory, after Wolpert and Macready The result that, averaged over all possible functions, every search method that never revisits a point does equally well, so any advantage on one class of functions is paid for on another. Excludes claims about any one class of functions, such as the test functions here. full definition theorems of Wolpert and Macready [4] give the background for any optimiser comparison: averaged over all possible functions, no method beats another, so an advantage on one class is paid for on another.

This study tested biologically labelled methodsDefinition: biologically labelled method our usage for this study An optimiser whose published description is framed by an animal, plant or other organism, here slime mould, grey wolf, particle swarm, bee colony, ant colony and invasive weed. Excludes any claim that the method works the way the organism does. full definition against plain ones with the optimum moved off the centre. It became a study of our own error as well. The first run's headline came from a bug in our CMA-ESDefinition: CMA-ES evolutionary computation, after Hansen and colleagues Covariance matrix adaptation evolution strategy: it samples candidates from a multivariate normal distribution and adapts the distribution's centre, shape and step size from the best samples. Excludes the larger-population restart schedule known as IPOP, which our version did not use. full definition code. CMA-ES, the covariance matrix adaptation evolution strategy, samples candidates from a normal distribution and adapts the distribution's centre, shape and step size. Auger and Hansen [5] described a version that restarts with a doubled population each time, known as IPOP. Ours restarts with the same population size.

The study ran twice, each time under its own preregistrationDefinition: preregistration research method (metascience) A dated written plan for a study, fixed and hash-locked before any result is seen: the question, the data, the statistic, the controls, the forecasts and the kill conditions. Excludes a plan revised after looking at the outcome, and a protocol written only for the record after the run. full definition. The first (here, the first run) was hash-locked on 31 October 2026 before any result of the run was seen; only smoke runs (short trial runs that check the code executes) in 3 dimensions with small budgets had been made, to fix crashes, and no rankings were read from them. It named the methods, the functions, the measures, two controlsDefinition: control statistics and experimental design A version of the data with the structure under test removed (for example shuffled or pooled), scored the same way as the real data, so the real score has something to beat. Excludes a comparison with a different dataset or with a published number. full definition, the forecasts and the kill conditionsDefinition: kill condition research method (our usage, after Popper's falsification) A rule written in the preregistration that, when met, decides against the claim with no room for judgement after the run. Excludes reasons found after the run for setting a result aside. full definition. The decision it informs was written in it: no slime or plant-labelled optimiser becomes a default search method in our tools unless it beats differential evolution and CMA-ES on the shifted functions.

The second (the rerun) was hash-locked on 31 October 2026 after the first run had been read in full. It records what went wrong, the fix, and new forecasts. Its own kill condition reads: if CMA-ES fails to bring the median error on the shifted ellipsoid below 1, it is still broken and the study stops. It also commits to reporting any biologically labelled method that beats the best plain one, whatever we expected.

Both preregistrations' SHA-256 hashes are in the reproduction section.

No external data were used. All results come from our own implementations of ten optimisers and seven standard test functions, run on one desktop computer. The method update rules were written from their published descriptions with default settings and no tuning. Every number in this paper is read from the two runs' result files.

The plain group is random search, L-BFGS-B with restarts and a numeric gradient, differential evolution (rand/1/bin) and CMA-ES with restarts. The biologically labelled group is the slime mould algorithm (SMA), the grey wolf optimiser (GWO), particle swarm optimisation (PSO), artificial bee colony (ABC), ant colony optimisation for continuous domains (ACOR) and invasive weed optimisation (IWO). Every method gets the same budget of 20,000 function evaluations; a numeric gradient spends evaluations from that budget.

The seven functions are the sphere, an ellipsoid with a condition number of one million, Rosenbrock, Rastrigin, Ackley, Griewank and Schaffer F7, each in 10 dimensions with a minimum value of 0. The search boxDefinition: search box optimisation The allowed region, here every coordinate between -100 and 100. A method may only propose and keep points inside it. Excludes constraints other than simple bounds on each coordinate. full definition allows every coordinate from -100 to 100. On the shifted armDefinition: shifted arm our usage for this study The version of each test function whose minimum is moved to a random point with every coordinate between -70 and 70 and whose axes are turned by a random rotation, the same for every method within one repeat. Excludes the unshifted arm, with the minimum at the centre and the axes aligned. full definition each base function $g$ is evaluated as

where $s$ is a random shift with every coordinate drawn from -70 to 70 and $R$ a random rotation. Every method sees the same $s$ and $R$ within one repeat. On the unshifted arm $s$ is zero and $R$ is the identity, so the minimum sits at the centre of the box. Each method ran 15 repeats on each function on each arm.

For method $m$ on function $f$, let $M_{\mathrm{fm}}$ be the median best error over the 15 repeats on the shifted arm and $U_{\mathrm{fm}}$ the same on the unshifted arm, each floored at ${10}^{-12}$. Methods are ranked on each function by the base-10 logarithm of $M_{\mathrm{fm}}$ rounded to two decimals, with ties sharing their average rank. Call that rank ${\rho }_{\mathrm{fm}}$. The mean rankDefinition: mean rank benchmarking statistics Each method's rank on each function by median final error, averaged over the seven functions. 1 is best and 10 is worst. Excludes how large the differences in error are. full definition is

The change in error from moving the minimum, in decadesDefinition: decade measurement A factor of ten. A change of one decade in error is a change of one in its base-10 logarithm. Excludes a difference on a linear scale. full definition, is

and a method's shift penaltyDefinition: shift penalty our usage for this study For one method, the median over the seven functions of how many decades its median error grows when the function moves from the unshifted arm to the shifted arm. Excludes effects of the rotation alone, which the design does not separate from the shift. full definition $D_m$ is the median of ${\delta }_{\mathrm{fm}}$ over the seven functions. The group gap is

where $r_{\mathrm{bio}}$ and $r_{\mathrm{plain}}$ are the averages of $r_m$ over each group, so a positive gap means the plain group did better.

In the first run CMA-ES scored a median error of 14 million on the ellipsoid. Tracing it showed that the distribution's centre had left the box (its norm reached 498 against a box half-width of 100). Every sample was clipped to the same corner, the step size fell to about ${10}^{-8}$, and the method never restarted. Differential evolution, ABC, ACOR and IWO also kept candidates outside the box in their internal state. These were errors in our code.

For the rerun every method keeps its candidates inside the box. CMA-ES adds a quadratic penalty for any part of a sample outside the box,

where $a$ is the absolute value of $f$ and $e_i$ is how far coordinate $i$ lies outside the box, zero inside. It restarts from a random point when its step size falls below ${10}^{-11}$. No method's parameters were tuned.

Two planted checksDefinition: planted check research method; called a positive control in experimental science Data built so that the right answer is known, run through the same analysis before the real data, to show that the analysis can see the effect when it is there. Excludes any check made on the real data, which cannot show what the analysis misses. full definition guard the instrument. CMA-ES must bring the shifted sphere below ${10}^{-8}$. A centre cheatDefinition: centre cheat our usage for this study A dummy method that only samples points within a tiny distance of the centre of the box, so it solves any function whose minimum sits there and nothing else. Excludes any real optimiser. full definition, a dummy method that samples only points next to the centre, must solve the unshifted sphere (error below 0.001) and fail the shifted one (base-10 logarithm of its error above 3). The cheat shows that the unshifted arm can be won by guessing the centre and that the shifted arm cannot.

The label permutationDefinition: label permutation statistics (permutation test) Shuffling which method name goes with which rank within each function, 2,000 times, to see how large a gap between the two groups' mean ranks arises by chance. Excludes uncertainty from the random repeats of each run. full definition control shuffles method names within each function's ranks 2,000 times and records the group gap each time. A real gap must clear the 95th percentile of that null distributionDefinition: null distribution statistics The spread of scores a control gives over many random draws. Its 95th percentile is the bar a real score must clear. Excludes a theoretical distribution assumed rather than drawn from the data. full definition.

The no-free-lunch check runs every method on a test function in which each integer cell of the box gets an independent random value, with a budget of 5,000 evaluations and 30 repeats. No method can exploit such a function, so the methods should tie. Ties are tested with the Kruskal-Wallis testDefinition: Kruskal-Wallis test statistics A rank-based test of whether several groups of numbers come from the same distribution. Excludes a test of which group is better. full definition, with L-BFGS left out because it can stall on a flat cell, on the best value found and, in the rerun, on the best of the first 500 distinct cells visited.

In the rerun CMA-ES reached a median error of 1.6e-15 on the shifted sphere, written in the e-notation used in the tables. The centre cheat reached 7.2e-7 on the unshifted sphere and an error whose base-10 logarithm is 4.27 on the shifted sphere. Both planted checks held.

Table 1 gives every method's mean rank in both runs and its shift penalty in the rerun, and Figure 1 draws the ranks. After the fix, CMA-ES ranked first on the shifted functions with a mean rank of 2.00, and the best method with a biological label, ACOR, ranked second at 3.14.c1 The fix moved CMA-ES from 4.86 to 2.00 and PSO, the first run's leader, from 2.79 to 4.57.

With the minimum at the centre, SMA ranked first (2.07) and GWO second (2.43). Figure 2 shows what moving the minimum did to each method. Moving the optimum away from the centre cost the slime mould method 12.1 and the grey wolf method 13.6 decades of median error, while ACOR, IWO and PSO lost 0.0006, 0.07 and 0.42 decades.c2 The centre bias described by Kudela [3] shows in two of the six labelled methods and not in the other four.

Table 2 gives the group contest. Without random search the plain group's mean rank was 3.98 against 5.54 for the biologically labelled group; with random search counted as plain the groups were 5.45 and 5.54, a gap that sat inside the label-permutation range (p 0.45).c3 The preregistered label-permutation control was computed only for the gap with random search included. The gap of 1.56 without it has no control of its own.

In the first run the labelled group's lead of 1.16 ranks came partly from our bug and partly from counting random search as plain, and it sat inside its own label-permutation range.c4 The first run's preregistration had a kill condition we did not apply when we first read it: if the no-free-lunch check fails to tie, the ranking is void. That check failed (p 0.0015), so by its own rule the first run's ranking was voidDefinition: void research method (our usage) The verdict of a run whose planted check failed or whose sample fell below its stated minimum. It says nothing for or against the claim. Excludes a null result, which comes from a working instrument. full definition. In the rerun the methods tied on that function, at p 0.13 on the best value and p 0.74 on the best of the first 500 distinct cells. The rerun's preregistration traced the first run's failure to converged methods revisiting the same cells, which the theorem does not count; in the rerun IWO visited a median of 2,870 distinct cells in 5,000 evaluations, against 5,000 for random search and ACOR.

Table 3 sets the rerun's forecasts against its outcomes.

The first run is a record of how a benchmark goes wrong in the direction of a popular story. CMA-ES was the strongest plain method we had, and our implementation of it was broken in a way that did not crash: it returned numbers, just bad ones. The labelled group won that run, and the error showed only because one score, 14 million on the ellipsoid, looked wrong. The rerun's preregistration was written after reading the first run, so the rerun is not blind to it; what keeps it honest is that its forecasts and kill condition were fixed before it ran.

The method-level results are the ones that hold up. CMA-ES ranked first once it was run correctly. SMA and GWO ranked first and second on centred functions and near the bottom on shifted ones, which is the pattern Kudela [3] describes for methods with a centre-biased update step. ACOR, IWO and PSO lost almost nothing when the minimum moved, so the centre-bias charge does not reach every method with a biological label. Camacho-Villalón and colleagues [2] argue from the structure of the update rules that GWO is a form of PSO. In our runs the two behave differently once the minimum moves: GWO loses 13.6 decades and PSO 0.42. A component that pulls candidates toward the centre would explain the gap, and we did not test which component it is.

The group result is weak. With random search counted as plain, the two groups' mean ranks differ by 0.09, well inside the label-permutation range. Leaving random search out gives a gap of 1.56, but that comparison had no control. Ranking methods by group label says less than ranking them one at a time.