Evolution Strategies for LLM Fine-Tuning: A Revival of a Pre-Deep-Learning Optimizer
Two 2025 papers revive {{evolution strategies}} (ES) as a credible alternative to {{reinforcement learning}} for fine-tuning large language models, exploiting the fact that RL fine-tuning rewards are already scalar at the sequence level — the regime where ES is naturally competitive.
Evolution strategies (ES) are a family of gradient-free optimization methods that perturb a model's parameters with random noise, evaluate each perturbed copy on a reward signal, and update toward the perturbations that scored higher. Until 2025, conventional wisdom held that ES could not scale past roughly a million parameters — the 2017 OpenAI ES paper (Salimans, Ho, Chen, Sidor, Sutskever) demonstrated ES on 2-million-parameter Atari networks, but applying it to billion-parameter LLMs seemed infeasible. Two 2025 papers overturned that assumption. The first, from Cognizant AI Lab (arXiv 2509.24372, September 2025), showed full-parameter ES fine-tuning of billion-parameter LLMs using a population of just 30 perturbations — three orders of magnitude smaller than prior ES populations. The second, EGGROLL from an Oxford/MILA/NVIDIA collaboration (arXiv 2511.16652, November 2025), structured each perturbation as a low-rank (LoRA-style) matrix so that many perturbations can be computed in a single batched forward pass, yielding a claimed 100-fold training-speed increase. The theoretical justification is that useful directions for improvement in LLM parameter space are concentrated in a much lower-dimensional subspace than the raw parameter count suggests. On the loss landscape, only a small number of directions actually lead uphill; most are flat or downhill. With 30 small Gaussian noise perturbations, a few will tilt slightly uphill, and averaging cancels noise while signal emerges. ES is poorly suited to next-token-prediction pretraining because gradient methods exploit the per-token loss signal that ES collapses to a single scalar reward. But RL fine-tuning (Reward Hacking Classic Examples-prone methods like PPO and GRPO) already operates on scalar sequence-level rewards. In that regime, RL's only advantage over ES is per-token credit assignment via backprop through long sequences — and that advantage costs significant compute. ES sidesteps it entirely, which is why ES becomes competitive precisely where RL is most expensive: long-horizon and sparse-reward tasks.