Google DeepMind finds two hidden bugs in softmax and cross-entropy
Why it matters
Google DeepMind researchers Jonathan T. Barron, Dan Gnanapragasam and Michael Tschannen report (arXiv, 22 Oct 2026) that standard implementations of softmax, log-softmax and cross-entropy have two numerical bugs. Softmax Collapse: when one logit is much larger than the others, the target logit's gradient rounds to zero, so gradients no longer sum to zero; with adaptive optimizers like Adam and AdamW this causes the recurring "Slingshot" loss spikes. In one grokking test without weight decay, a spike dropped standard cross-entropy test accuracy from 61% to 23%, while the fixed version continued smoothly to 85%. Softmax Cancellation: in knowledge distillation, standard code produces non-zero gradients even when student and teacher logits are equal. The fix rewrites these functions in log space (softplus, expm1, log1p) as drop-in replacements without meaningful slowdown. Caveat: the authors say the extreme margins and near-zero student-teacher gaps in their experiments are uncommon in standard large-scale training.
Read the source: arXiv (Google DeepMind) (opens in a new tab)
All hits from 25 Oct 2026More ResearchGet the daily hits by email