The Loss Does Not See the Basis, But Adam Does [R]

In a factored model W = UV^T, the loss is invariant to rotations (U,V) → (UQ, VQ). Gradient Descent (GD) respects this property. Adam's per-coordinate second moment does not, because it depends on the specific basis in which the factors are written.

The claim is that this single property dictates whether optimizers retain or lose GD's implicit low-rank bias.

Nine update rules were evaluated on underdetermined matrix sensing, all compared at matched training loss to ensure no method benefits from underfitting. The results show two distinct clusters: GD, shared-scalar Adam, Muon, and Shampoo preserve the bias. Adam, RMSProp, Lion, signum, and Adafactor lose it.

To isolate the mechanism, a one-parameter family was utilized to transition Adam's denominator from a per-coordinate value to a single shared scalar. Recovery improves monotonically along this transition, indicating that the degradation is caused by anisotropy rather than adaptivity in general.

The behavior of the Muon optimizer was unexpected. It is exact on truly low-rank targets, but degrades rapidly as a spectral tail is introduced, ceding to GD at a crossover near 4% tail energy. While recent literature diverges on Muon, with some reporting a strong spectral simplicity bias and others finding it fits spurious features in deep-linear models, this sweep demonstrates both behaviors along the same axis.

The criterion was also applied to the author's earlier optimizer, revealing that its per-coordinate clip was breaking the structure it was designed to inject. Implementing a global norm clip instead improved the recovery error from 0.347 to 0.220.

One caveat is noted up front: the 43-44% held-out error reduction on hyperspectral data relies on a train-only learning rate rule, and that rule assigns Adam the worst rate on its own grid. When each method is permitted to select its own optimal rate, the performance gap narrows considerably (Appendix D.6). The train-only rule was maintained because selecting on held-out data introduces the exact bias the experiment aims to avoid, but the core claim relies on the underlying mechanism rather than the specific quantitative margin.

The theoretical guarantees cover memoryless rules only. The effects of momentum remain empirical and are not proved.

Paper:https://arxiv.org/abs/2608.05136

Code, logs, seeds:https://github.com/idevender/loss-basis-adam