qwen 2.5 72b lora hits numerical instability at step 890 but only when training on code, not prose
trained qwen 2.5 72b lora (rank 64, lr 1e-4, bf16 compute) on two datasets - one is code generation (python/js functions), one is prose summarization. same hyperparams, same hardware (4x4090) prose dataset trains clean past 2000 steps, eval loss drops smoothly code dataset hits gradient explosion at step 890 ± 30 steps across 3 runs. gradients spike to nan in layers 18-22 specifically my theory is code has way more low-frequency tokens (brackets, operators) and the quantization interacts badly with them during backprop but i have no idea how to test this anyone seen similar dataset-dependent instability on qlora? does switchign to full lora fix it or is this a qwen architecture thing
hit this exact thing on llama 3.1 70b lora last month. turned out the synthetic data had way more repetition than human data and the model just memorized it by step 800. what temp did you generate the synthetic data at?
hit this too. what version of llama.cpp and did you try lowering the learning rate to see if it pushed the instability point further
ok so this is super specific - code vs prose instability at the same step count. what's different about your code dataset? is it synthetic or human-written, and did you check for repetition or data quality issues in the batches around step 890