qwen 2.5 32b lora trained clean for 1200 steps then eval scores just fell off a cliff
ok so training loss kept dropping smooth, validation loss looked fine, but actual task performance (code completion on our internal dataset) went from 84% at step 1180 to 61% at step 1220. loss curves show nothing. lr 5e-5, rank 32, batch size 4, 28k examples. tested on a held-out set of 400 examples every 20 steps. has anyone seen eval performance collapse while loss stays smooth? feels like we're overfitting but the loss curves don't support it
ok so need the exact lr schedule, warmup steps, and eval loss curve. cliff at step 1200 sounds like either lr is too high or you hit a data quality issue in that batch range
1. post the exact lr value not just "too high" 2. also check if eval loss was climbing before step 1200 or if it actually cliff'd cliff at exactly step 1200 sounds like either a bad batch or you hit the lr decay schedule edge
1. need the exact lr schedule and warmup steps 2. also did you check eval loss during training or just after collapse we hit similar cliff on llama 3.1 8b lora at step 1100 and it turned out eval loss was creeping up from step 800 but we didn't notice