Xingjian Bai, Jia-Hao Liang, Jayden Teoh, Vincent Sitzmann
October 2026
TL;DR We put an agent at the heart of the optimizer. It watches the training dynamics, reacts to them, and rewrites the optimizer’s source code as training goes. When no recipe has been tuned for a task, it matches or beats the best recipe one could pick off the shelf, across image and language tasks. Further, we demonstrate that it spontaneously comes up with sophisticated schedules and intricate strategies to prevent loss spikes and escape overfitting. Try it on your own training run with the OPAL package.
A reactive optimizerstep 0
LightningDiT-XL/1 on ImageNet
Today, the recipe used to optimize a deep neural network is typically completely determined before training even begins. A great deal of human ingenuity has gone into priors regarding how training should progress: adaptive step sizes [1], preconditioning [2, 3], orthogonalized updates [4, 5], schedules tied to the training step [6, 7], and learning rates scaled with model size [8]. Automated discovery [9, 10, 11, 12] searches across many runs for better optimizers. Whether hand-crafted or automatically discovered, the optimizer then runs unchanged through the end of training, adapting, if at all, according to rules written in advance. This scope leaves the following questions unanswered: How should an optimizer respond to an unexpected loss spike or stall? What if one module's gradients start to oscillate while the others keep improving smoothly? And when a new type of architecture trains poorly with Adam or Muon, is the architecture worse, or has it not yet met an optimizer suited to it?
In this post, we attempt to expand the scope of conventional optimizers and make them reactive to the training itself, leading to an ever-adapting optimizer inside a single training run. This reactive optimizer reasons on the spot and writes the weight update rule for each parameter group. Directly writing optimization code, it has the freedom to implement complicated schedules, respond to training accidents, and customize the optimizer for the single training episode in which it exists. Building on earlier attempts to adapt optimizers during training [12, 13, 14] and to let a model direct its own fine-tuning [15], the reactive optimizer is entirely zero-shot, with nothing learned or retained from repeated runs, and puts the optimizer's entire algorithm up for rewriting.
We start with a practical situation, with tasks from image generation and language modeling, in which a reactive optimizer can do much better than static ones. We then turn to the emergent properties of the reactive optimizer, including sophisticated scheduling, suppressing loss spikes, and escaping overfitting. Lastly, we ask what all this costs and how practical it is at scale. We expect the reactive optimizer to grow only stronger with each generation of agents, to be most useful exactly where no recipe exists yet (for a new architecture or a new dataset), and to be one step toward solving the puzzles of deep learning optimization.
1. Training without a recipe
Suppose a PhD student has a new idea: a model of their own design, or a new dataset on which to train a transformer. They have one GPU for a single day only and a modest allotment of LLM API tokens, allowing for just one serious training run. What should the student do? The obvious course of action would be to pick the most standard training recipe, or one from a recent paper, and cope with the possibility that it might be unsuitable for their scenario, with spiking/stalling loss, etc.
The underlying question is broader: when bounded by GPU time, researcher time, or wall-clock time, how can a deep learning model be trained to its best performance without knowing the optimal optimization recipe in advance? New architectures, modalities, languages, and scientific data all fall into this setting, where few people can afford a sweep at scale.
A reactive optimizer offers another way to use that budget: it spends LLM tokens in place of GPU/human hours. With optimizers reacting to training incidents and adjusting themselves, much of the burden of recipe tuning can be reduced.
Results in brief
We start from the optimization track (Track 3) of modded-nanogpt [16], a language-model pre-training task (training a 162M GPT on FineWeb) with many recipes tuned specifically for this one setting. From this task, we create two slightly differing natural settings: (1) the same model trained on an alternate language dataset (we select Chinese, Thai, and Kyrgyz), and (2) the original setup with a quarter of the batch size. Neither comes with a recipe tuned for that setting, so we compare the reactive optimizer with the natural options one would have if they wanted to one-shot the new training: adopting five leaderboard recipes with natural adjustments. In all these new settings, the reactive optimizer outperforms or matches the best baseline (Figure 1).
We also apply the reactive optimizer directly to Goldfish [17], which proposes a dataset of 350 languages and trains one small GPT per dataset with a standard recipe not tuned for each language. This makes it an ideal ground to test the optimizer's adaptiveness in one shot. On the Amharic dataset, the official recipe ends at 83.48 bits per sentence on held-out FLORES-200 text (lower is better); the reactive optimizer, initialized with this same recipe, ends at 81.11 in one shot (on Swahili, we obtain a similar gain). Sweeping the learning rate of the standard recipe finds a static recipe that ends ahead of the reactive optimizer, at 80.89, but the reactive optimizer initialized from that swept recipe ends ahead again, at 80.73.
We also try the reactive optimizer on image generation, a direction where the literature usually does not sweep optimization hyperparameters, since diffusion-model design and optimizer design are viewed as mostly orthogonal1. It ends ahead of the official recipe on LightningDiT-XL/1 ([18]; FID-50K 1.83 against 2.18 with guidance, 4.33 against 5.15 without), whose official recipe trains with AdamW, and on RAEv2-B (3.19 against 3.39), whose recipe trains with a Muon variant.
Figure 1 shows these runs, each with Claude Opus 5.5 powering the agent; we name our reactive optimizer OPAL (OPtimizer with an Agent in the Loop). Together, these results show that in one-shot settings, a reactive optimizer can lead to a better solution than adhering to a predetermined recipe.
modded-nanogpt Track 3
| Task | the 162M GPT of the modded-nanogpt optimization track (Track 3), the same runs in each setting: official NanoGPT, the track's own (FineWeb at the official batch of 524,288 tokens per update); NanoGPT, ¼ batch (FineWeb, 13,000 updates of 131,072 tokens); NanoGPT on Chinese, Thai and Kyrgyz (the 1 GB Goldfish Chinese, Thai and Kyrgyz sets, 3,000 updates of 524,288 tokens) |
|---|---|
| Metric | official NanoGPT: the first update at which the validation loss reaches 3.28, the track's target (validation every 25 updates); ¼ batch: validation loss after 13,000 updates; Chinese, Thai and Kyrgyz: validation loss after 3,000 updates, for #46 and OPAL from #36 with #46's own readout (a moving average of the weights over the last updates), for the other records with the last weights; in Chinese and Thai, OPAL's bar is the mean of three runs. The loss bars start at c, the largest multiple of 0.1 at least 0.05 below the setting's lowest value: 3.2 (¼ batch), 4.0 (Chinese), 4.5 (Thai) and 3.7 (Kyrgyz) |
| Baseline | five records from the leaderboard, #46, #30, #21 (Shampoo), #36 (the tuned Muon baseline) and #2 (AdamW), and OPAL from #36. The leaderboard lists 2,690, 2,930, 4,100 and 3,250 updates for #46, #30, #21 and #36; #2's bar in official NanoGPT is the leaderboard's 5,625: our own run of #2 ended at 3.28449 after 5,625 updates and never reached 3.28; at ¼ batch every record ran with all its learning rates ×¼. |
Goldfish, Amharic
| Task | a 125M GPT trained from scratch on 1 GB of Amharic (Goldfish), 10 epochs (64,626 updates) |
|---|---|
| Metric | bits per sentence on FLORES-200 (dev + devtest) |
| Baseline | the official Goldfish recipe (AdamW, learning rate 1e-4, weight decay 0.01, 10% warmup, then linear decay; gradient clipping at 1.0); nine more static recipes, each for all 10 epochs: AdamW at learning rates 6e-4, 1e-3, 2e-3 and 4e-3 with weight decay 0.1 (a four-value sweep), AdamW at 3e-4 with weight decay 0.1 or 0.01 and at 1e-4 with weight decay 0.1, and Muon on the hidden block matrices in the official recipe and at learning rate 3e-4, weight decay 0.1. OPAL ran twice: from the official recipe (at its defaults, 50 decisions; shown by default) and from the sweep's best point, AdamW at learning rate 2e-3 (200 decisions; shown with the sweep) |
LightningDiT-XL/1
| Task | image generation on ImageNet 256×256 in a VA-VAE latent space; LightningDiT-XL/1, 64 epochs |
|---|---|
| Metric | FID-50K at the end, with and without guidance; the curve shows FID-10K during training |
| Baseline | the official recipe, retrained by us, and the authors' released checkpoint (FID-50K 5.23 without guidance, 2.16 with) |
RAEv2-B
| Task | image generation on ImageNet 256×256 in a representation-autoencoder latent space; RAEv2-B, 20 epochs |
|---|---|
| Metric | FID-50K |
| Baseline | the official recipe, which already trains with GMuon, a Muon variant |
Figure 1. OPAL (OPtimizer with an Agent in the Loop), our reactive optimizer, across different settings. In most settings where optimization hyperparameters are not heavily tuned for, reactive optimizers improve the training loss and test performance upon the canonical choices. In modded-nanogpt, #2 is AdamW, #21 is Shampoo, #36 is a tuned Muon, and #30 and #46 are more elaborate Muon-based recipes.
Beyond the practical benefit, we are intrigued by the spontaneous decisions made by the zero-shot agent, after observing and analyzing the training dynamics in its own way. In the next two sections, we explore what the reactive optimizer chooses to watch and what it changes in response.
1 For example, the standard recipes of DiT and SiT [19] train with AdamW at one constant learning rate. That has begun to change, with Muon and its variants taken up for diffusion transformers (CMuon [20]; Li et al. [21]; RAEv2's GMuon [22]) and a recent controlled benchmark of optimizers for diffusion models [23].
2. Updating with spontaneous analysis
Figure 2 shows how an agent updates the optimizer code on the fly. Concretely, we allow the agent to rewrite a file, updater.py, used in every step of optimization. We use code as the entire interface since this allows the agent to easily alter any part of the optimization. Yet, before the agent can make changes, it must observe, on its own, how training is progressing, leading to the second program it can rewrite, analysis.py.
What information should the agent use to make its decisions? Raw training states would be impractical: the weights, activations, and gradients would swamp its context. Instead, the statistics and geometry of training dynamics could be helpful for decisions, such as the gradient norm of each parameter group, the size of an update next to the weights it changes, the cosine between consecutive gradients, and which tensors exhibit unusually large gradient spikes. Therefore, as with the optimizer, we give the agent the most freedom: it writes a program, analysis.py, to extract whatever it wants to know from the training dynamics. Its self-written probes run throughout training, taking from the state of training whatever the agent asks for and handing it to the agent for its next decision.
We complete the bidirectional loop. Through analysis.py, training reaches the agent in whatever form the agent chooses to see it, and through updater.py, the agent's decisions reach every step of training. The two programs let the agent devise new optimization algorithms along the way with maximum flexibility.
3. What is dynamically changing during optimization?
In this section, we look at what the reactive optimizer does throughout training, examining the decisions behind these results. As a summary, Figure 3 shows when, how, and why each agent edits its code.
Drag to a red tick to see what the agent changed there and why.
All decisions that changed a program
niche practiceblock-specificobservation-originatedtime-step-specificfix or revertanalysis only
The rest of this section examines these adaptations type by type.
3.1 Learning-rate schedule
Set by OPAL
SiT-B/2
- block matrices
- adaLN modulation
- timestep embedder
- patch and class embedders
RAEv2-B
- backbone attention
- head attention
- vectors and timestep embedder
LightningDiT-XL/1
- attention and MLP matrices
- adaLN matrices
- timestep embedder, final adaLN
- everything else
NanoGPT, ¼ batch
- q, k, projection and MLP matrices
- value matrices
- embedding
- final norm gain
Frontier training runs come with elaborate hyperparameter schedules, such as the five stages of DeepSeek-V3's learning rate ([24]; Figure 4, left), with such schedules sometimes being written by humans during the training process [25]. Engineers can make only so many decisions, each weighing only so many signals. A reactive optimizer can take over this work and has no such limit: it decides frequently, and at each decision, it can access the entire history of the run. As Figure 4 shows, the schedules our agents built are intricate, with separate multi-stage schedules for each parameter group, and differ from run to run. Several of them raise a learning rate mid-run, a move rarely seen in modern schedulers, yet naturally helpful in many scenarios [6, 26]. Moreover, the four examples below show the evidence they cited for their decisions.
We observe that the agents base these decisions on the diagnostics from analysis.py. Some examples are: the cosine between a module's consecutive gradients as a sign of overshooting (see “Deduced from clues” below); A/B testing between two learning rates every few hundred updates, comparing how fast the loss falls under each (“A-B-A test”); trial and error, with the learning rate raised, lowered, and raised again (“Trial and error”); and the trend of the loss, which makes agents start their cooldown earlier than planned (“Re-planned cooldown”). Each of these changes of schedule involves observation of the training dynamics and reasoning upon said observations.
How the agents set the learning rate: four examples
Deduced from clues
The SiT agent's analysis program measured the cosine between consecutive gradients of each parameter group. Where it was negative, the agent lowered the learning rate: the timestep embedder's went to 0.3× and then to 0.15× of the official rate, with this stated reason:
Step 7,506, Decision 6Timestep-embedder learning rate goes from 0.3× to 0.15× (1.5e-5) after update 7506, because its cosine is still around −0.22.
Where the cosine stayed slightly positive, for the Muon matrices, it raised the learning rate.
A-B-A test
The RAEv2 agent alternated two learning rates over consecutive stretches of a few hundred updates (A-B-A) and compared the training loss under each. After one such test, it switched to 2× from update 3,100, with this stated reason:
Update 3,000, Decision 12The A-B-A test points the other way from my provisional read at update 2750: 2× gives faster real progress than 1.5×.
Trial and error
The same agent set every group's learning rate to 1.3×, returned the backbone's learning rate to 1× when the backbone's gradients began to change sign at each step, and at the next decision raised every group's learning rate to 1.5×:
Update 1,000, Decision 4My explanation from last time was wrong, so I'm raising the learning rate in every group to 1.5× (3e-4).
It then ran learning rates of 2×, 2.5× and 2.75×, each for about 650 updates, and kept 2×. Below, the learning rate of its backbone attention matrices, as a multiple of the official rate; the two deep dips are 75-update reductions of the learning rate used as tests.
Re-planned cooldown
At step 190K, the SiT agent moved its planned cooldown from 320K to 280K and lowered its learning-rate floor from 0.1 to 0.05 of its peak (the bend in the SiT panel of Figure 4). Its stated reason:
Step 190,152, Decision 53the constant-rate phase is now mostly holding the weights at a noise floor set by the learning rate, rather than making progress.
The RAEv2 agent started its cooldown at update 11,250, 45% of the run instead of the planned 75%, citing the trend of its loss and a paper on decaying the learning rate [27].
3.2 Optimizer design
AdamWMuonpartial Muonbfloat16 momentsfloat32 moments
Language modeling
Image generation
Goldfish Amharic 64,626 updates
LightningDiT-XL/1 80,000 updates
modded-nanogpt Track 3 3,250 updates
RAEv2-B 25,020 updates
The reactive optimizers also change which rule updates which tensors. In our experiments, many switch from the initially given AdamW to the well-known hybrid, Muon, for the attention and MLP matrices while remaining with AdamW for the rest. Later, they make more specific moves (Figure 5).
The optimizer can also decide where precision matters. On modded-nanogpt Track 3, it measured that the entries of the token embedding are about 15 in size, where adjacent bfloat16 values are 0.0625 apart, so as the learning rate decayed, rounding would throw away a growing share of each update. From update 1,040, it kept that specific matrix's Adam moments in float32 and wrote its weights back with compensated rounding and left every other tensor as it was.
3.3 Momentum
Best log10(f(x) − f*), average of 34 landscapes ↓
| reactive optimizer | −11.6 |
| gradient descent, lr 0.01 | 0.6 |
| Nesterov 0.9, lr 0.01 | −0.5 |
| Adam, lr 0.001 | 1.1 |
| best default setting (Adam, lr 0.1) | −2.2 |
step 300
Hover a red marker under the curves to read what the agent saw and why it went back to its best point.
We investigate the reactive optimizer's ability to manipulate momentum on 2D toy landscapes, where we give the agent new abilities: resetting its momentum, returning to its best point so far, and jumping to a random nearby point. In these toy settings, standard OPAL would simply replace the default recipe with a second-order method within a few decisions and reach perfect performance very fast. We therefore restrict OPAL to the heavy-ball algorithm, while letting it control the hyperparameters. On the landscape in Figure 6, the start sits beside a nearly flat saddle where common optimizers stall. The reactive optimizer raises its momentum to cross the flat stretch, lowers it as the gradient falls near the minimum, and each time it overshoots, returns to its best point and restarts its momentum, reaching the minimum by the end of its 300-step budget. Over all 34 landscapes we ran, it achieves an average score of −11.6 (log10(f − f*), lower is better), while any single static setting does worse: −2.2 for the best of twelve common recipes, and −7.4 for the best of 382,080 swept recipes (Figure 6).
3.4 Loss spikes
In our large-scale training, OPAL also limits the damage of gradient spikes with module-specific clipping: another SiT agent traced 17 of the 18 gradient spikes it logged to the timestep embedder, and added a clipping threshold for each tensor, set from that tensor's own recent gradient norm, so that a spike in the embedder no longer cuts the updates of every other tensor; its old global threshold stays only as a backstop. Neither move is standard practice, but both have precedents: clipping each tensor against its own history [28], and damping the one layer that spikes, as GLM-130B did for its embedding layer [29].
Spikes can also come from the data, through the interaction of particular batches with the model state [30], and clipping does not in general protect against noisy labels [31]. In a controlled experiment, we insert four stretches of random images and labels into CIFAR-10, and train a ResNet-18 with OPAL, which is not told about the corruption, and with static optimizers (Figure 7). The first two stretches cost OPAL 50 and 66 points of test accuracy, about as much as the baselines. In between, it added loss damping, replaced it with clipping, and then wrote a gate that holds back batches whose loss is far above recent clean ones. After it tuned that gate, the third stretch cost 12 points and the fourth none, while the clipped baselines and Stable-SPAM [32] still lost 28 to 76 points in each.
3.5 Overfitting
Test loss on CIFAR-10 with 10% mislabeled training images
OPALAdamW, the recipe it started fromearly stoppingoptimizer change
Setting
A ResNet-18 trained on CIFAR-10 for 120 epochs without data augmentation. Among the 50,000 training images, 10% are held out with their correct labels as a development set; among the rest, 10% get a wrong label, drawn uniformly from the wrong classes and fixed for the run. The agent saw training statistics, along with dev loss and dev accuracy; it was not told about the specific setting (that some labels were wrong), and it never saw the test performance. We make the network normalize its logits (a centered form of LogitNorm, [33]) to prevent loss-hacking behavior. AdamW runs from learning rate 3e-3, which is also the starting recipe for the reactive optimizer.
Training on data with wrong labels is a standard way of testing overfitting [34]. We design such a setting (details under Figure 8) and ask if the optimizer can overcome overfitting by itself, reacting to signals from the training set and the development set. Trained straight through, the AdamW baseline ends at test loss 2.49 and accuracy 73.5% (stopped early on the development set, at 1.01 and 72.9%; with its weights averaged and the average chosen there, at 0.83 and 77.5%). The reactive optimizer ends at loss 0.69 and accuracy 78.6%. As a direct measure of memorization, when asked to classify the 4,500 mislabeled training images, the baseline gives the wrong label it was trained on for all of them, its early-stopped version for 19% and its weight-averaged version for 14%. The reactive optimizer, on the other hand, gives that label for only 4%, and the true label for 78%. Its strategy combines progressively higher learning rates with weight averaging, and it chooses its final average on the development set, which its own analysis program calls “extended SWA” [35]. Hover the cursor on the ticks of Figure 8 to find out its exact decisions.
Summary of analysis
Above, we show that reactive optimizers can adapt their recipe to the dataset, to individual modules, and to the behavior of each run, a trait unseen in conventional optimizers. Such customization and self-adaptiveness allow them to craft sophisticated schedules and navigate complex training landscapes.
Still, tuning hyperparameters, suppressing loss spikes, and watching out for overfitting are things human experts can do, albeit less conveniently and at a higher cost. As agents get smarter, with hundreds of papers in their memory, thousands of tokens in their chain of thought, millions of training signals in their context, and every step in their hands, we are convinced that reactive optimizers will only become better, developing mechanisms during training more arcane than the ones demonstrated here, beyond what human experts could tune by hand. Such mechanisms may include tracking the second-order curvature of the loss landscape, auditing the variance of each data batch's signal, or implementing never-before-seen optimization schemes.
4. Real-world applicability
Is a reactive optimizer practical? The very first concern is the time spent waiting for reasoning decisions. In its simplest design, training is stopped while the agent reasons, inducing significant overhead. In our OPAL implementation, we instead provide the option to allow training to continue while the agent thinks, and swap in the agent's edits at a later update (e.g., a median of 65 updates later on Goldfish). This should have little impact on performance, since most optimizer decisions are not especially time-critical. With this option, the runs in Figure 9 add only a little time on top of the static run (1.01× to 1.14×). At a larger scale, the current implementation still needs work to support sharded training and optimizer compilation; we leave these practical concerns to later work.
no agent training does not wait for the agent waiting for the agent
wall-clock time relative to no agent ↓
When comparing API tokens to GPU hours, the reasoning of a reactive optimizer costs little compared with the training compute. At about 100 decisions in a single training run, the LightningDiT agent costs $6.42 at Claude Opus 5.5 list price, on top of about 74 H100-hours of training (about $510 at AWS's current on-demand price); equipping it on larger training would only make this relative cost smaller. Admittedly, counted in FLOPs, any reactive optimizer would be impractical given the size of frontier agents. Here we simply want to point out that, for most researchers, GPU hours and token access to closed-source models are two different commodities and as the frontier continues to advance, the price of a given capability will continue to fall.
Another limitation is that the reactive optimizer needs room to observe and react; its gains shrink when a run is short and its recipe is fine-tuned. The modded-nanogpt track is such a setting: 3,250 updates of a recipe that many people have tuned for nearly this exact task; our gains there are smaller than in the other tasks we tried, which train for about 8 to 120 times as many updates. Moreover, the reactive behavior introduces another source of uncertainty, potentially making training less reproducible. Most experiments in this post are run once, a few three times; reruns on the released package are in the OPAL repository.
Fine-tuning platforms are bringing training to everyone. Tinker [36] and Prime Intellect's Hosted Training [37] let users fine-tune open-weight models for their own applications without managing the GPUs behind them. On these platforms, however, the mental burden of a good training recipe still falls on people. A reactive optimizer fits naturally on top of these platforms, offloading that burden to agents and helping each user personalize a model to their own needs.
5. Conclusion
Until now, optimizers have been written before training. Agents, as a cheap form of reasoner with a long context window and expert knowledge, make something new possible: an optimizer that keeps deciding while the network trains in reaction to optimization events. We have demonstrated that such a reactive optimizer watched the training dynamics and rewrote its own code as training unfolded, for different groups of parameters, at the step it deemed proper. On toy and scaled-up experiments, most of these runs ended ahead of the recipes from which they started.
The analysis and update mechanism we design is meant to be as simple and general as possible. Wrapping a standard optimizer, it assumes nothing about the model, the data, or the training dynamics. It is useful today wherever no recipe exists yet. For example, a model on a biological or other scientific dataset, an architecture for which no recipe was written, or a language for which no one has tuned a recipe are all excellent use cases for such a reactive optimizer. This is because the optimizer’s agent takes the role of a human expert at countless training steps, where no expert has ever had the time to sit, making similar decisions in response to events in the run. Seen this way, a reactive optimizer is a natural thing to build, and its gains over static ones are to be expected. Furthermore, a training run holds far more information than humans can use, almost all of which is ignored today. A reactive optimizer has all this information within reach. Imagine a world where the notion of a monolithic optimizer is gone; what replaces it is an optimization intelligence that adapts to every specific run, and it might not be gradient-based, but zeroth-order [38], language-based or more. Perhaps, in the future, we should delegate optimization decisions to agents, not humans.
In the past decade, deep learning models have come to possess unreasonably effective power through training. Yet the very heart of their making, the optimizer, is still mechanistic. Our post is an attempt to breathe the magic that training has given us into the core of training itself, forming a loop that we hope will create more capability in the future.
pip install opal-optimizerimport torch
import opal
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=steps)
opt = opal.wrap(opt, model, scheduler=sched, total_steps=steps,
goal="...", max_cost_usd=20)
for step in range(steps):
loss = criterion(model(x), y)
loss.backward()
opt.step(loss=loss)
opt.zero_grad()
sched.step()Acknowledgements
We sincerely thank George Cazenavette, David Charatan, Shivam Duggal, Jyo Pari and Qianwei Jia for their invaluable advice on this write-up; thank Zhenting Qi and Zeqing Yuan for their discussion. We are also grateful to Professor Phillip Isola and the Isola lab for their insightful feedback on an earlier version of this post.
Citation
@misc{bai2026reactive,
title = {Reactive Optimizers},
author = {Bai, Xingjian and Liang, Jia-Hao and Teoh, Jayden and Sitzmann, Vincent},
year = {2026},
month = {oct},
url = {https://xingjianbai.com/reactive-optimizers/},
note = {Blog post}
}References
- Kingma and Ba. Adam: A Method for Stochastic Optimization. ICLR 2015. arXiv:1412.6980.
- Gupta, Koren and Singer. Shampoo: Preconditioned Stochastic Tensor Optimization. ICML 2018. arXiv:1802.09568.
- Vyas et al. SOAP: Improving and Stabilizing Shampoo using Adam. arXiv:2409.11321, 2024.
- Jordan et al. Muon: An optimizer for hidden layers in neural networks. Blog post, 2024.
- Liu et al. Muon is Scalable for LLM Training. arXiv:2502.16982, 2025.
- Loshchilov and Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR 2017. arXiv:1608.03983.
- Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. NeurIPS 2024. arXiv:2405.18392.
- Yang et al. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. arXiv:2203.03466, 2022.
- Lu et al. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv:2408.06292, 2024.
- Prime Intellect. Autonomous Speedrunning Experiment. GitHub, 2026.
- Chen et al. Symbolic Discovery of Optimization Algorithms. arXiv:2302.06675, 2023.
- Metz et al. VeLO: Training Versatile Learned Optimizers by Scaling Up. arXiv:2211.09760, 2022.
- Daniel, Taylor and Nowozin. Learning Step Size Controllers for Robust Neural Network Training. AAAI 2016. doi:10.1609/aaai.v30i1.10187.
- Jaderberg et al. Population Based Training of Neural Networks. arXiv:1711.09846, 2017.
- Zweiger, Pari, Guo, Akyürek, Kim and Agrawal. Self-Adapting Language Models. arXiv:2506.10943, 2025.
- Jordan. Modded-NanoGPT Optimization Benchmark. GitHub, 2026.
- Chang et al. Goldfish: Monolingual Language Models for 350 Languages. LREC 2026. arXiv:2408.10441.
- Yao, Yang and Wang. Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models. arXiv:2501.01423, 2025.
- Ma et al. SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers. arXiv:2401.08740, 2024.
- Chen, Sun and Yuan. CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization. ECCV 2026. arXiv:2608.02502.
- Li et al. Scaling Muon for Diffusion Transformers. arXiv:2608.20818, 2026.
- Singh, Zheng, Wu, Zhang, Shechtman and Xie. Improved Baselines with Representation Autoencoders. arXiv:2605.18324, 2026.
- Bolatov et al. Optimizers for Diffusion Models: A Controlled Benchmark. arXiv:2609.23055, 2026.
- DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437, 2024.
- Zhang et al. OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01068, 2022.
- Smith. Cyclical Learning Rates for Training Neural Networks. WACV 2017. arXiv:1506.01186.
- Bergsma, Dey, Gosal, Gray, Soboleva and Hestness. Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs. ICLR 2025. arXiv:2502.15938.
- Wang et al. AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping. ICML 2026. arXiv:2502.11034.
- Zeng et al. GLM-130B: An Open Bilingual Pre-trained Model. arXiv:2210.02414, 2022.
- Chowdhery et al. PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311, 2022.
- Menon, Rawat, Reddi and Kumar. Can Gradient Clipping Mitigate Label Noise? ICLR 2020. OpenReview, 2020.
- Huang et al. Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam. arXiv:2502.17055v2, 2025.
- Wei, Xie, Cheng, Feng, An and Li. Mitigating Neural Network Overconfidence with Logit Normalization. ICML 2022. arXiv:2205.09310.
- Nakkiran, Kaplun, Bansal, Yang, Barak and Sutskever. Deep Double Descent: Where Bigger Models and More Data Hurt. ICLR 2020. arXiv:1912.02292.
- Izmailov, Podoprikhin, Garipov, Vetrov and Wilson. Averaging Weights Leads to Wider Optima and Better Generalization. UAI 2018. arXiv:1803.05407.
- Thinking Machines Lab. Announcing Tinker. Blog post, 2025.
- Prime Intellect. Getting Started (Hosted Training). Documentation, accessed October 21, 2026.
- Dahal, Mandal, Gülbahar and Vegesna. Dust: Pretraining Transformers Without Backpropagation. Q Labs, 2026.