MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

2026-10-28 · ArXiv

arXiv:2610.37647v1 Announce Type: cross

Abstract: Weight decay and explicit weight-norm constraints improve optimizer performance in language model pretraining, yet a unified understanding of why such control helps and how much is needed remains incomplete. We analyze norm control in Muon over the entire training trajectory using the \emph{accumulated effective learning rate}, $C_T=\sum_{t

原文链接：https://arxiv.org/abs/2610.37647

查看更多资讯请见 AI 行业资讯。