qwen 2.5 32b lora collapses at step 920 with warmup but step 440 without - gradient norms look identical
Training a lora on qwen 2.5 32b for code completion (28k examples, rank 32, lr=1e-4). Without warmup the loss spikes at step 440 and never recovers. With 100 step linear warmup it trains clean until step 920 then collapses in the exact same way - loss jumps from 1.2 to 5.8 in a single step. The weird thing is I logged gradient norms per layer and they look basically identical in both runs right before collapse. No obvious explosion, no specific layer going crazy. Both runs show grad norms around 0.8-1.2 across all layers, then boom, collapse. I thought warmup was supposed to prevent this kind of instability but it just delays it by ~480 steps. Dataset is clean (checked for duplicates, outliers, encoding issues). Optimizer is adamw with weight_decay=0.01. Has anyone seen warmup delay collapse but not prevent it?
gradient norms looking identical means you're measuring wrong
gradient norms looking identical means you're probably logging the wrong layer.... check norms per-layer not just global average. we hit similar collapse on qwen 2.5 32b lora and the issue was layer 18-22 had exploding grads while global norm looked fine
+1 on per-layer norms. we hit simialr collapse on qwen 2.5 14b lora and the global gradient norm looked fine but layer 19 was exploding. added per-layer logging and caught it at step 340 before full collapse
checked per-layer norms and you're absolutely right - layer 22 was exploding starting around step 890. global norm looked fine because the other 31 layers were stable. added per-layer gradient clipping and it trained clean past 1400 steps