llama 3.3 70b lora - rank 64 trains clean at lr 5e-5 but rank 32 needs 1e-4, why
ok so this is backwards from what i expected. tested on code completion task (12k examples, all python). rank 64 with lr 5e-5 trains smooth, loss drops steadily, no spikes. rank 32 with same lr barely moves, had to bump to 1e-4 to see any progress. thought higher rank = needs lower lr because more parameters, but seeing the opposite. anyone hit this? using adamw, batch size 8, gradient accumulation 4.
rank 64 needs lower lr because higher rank = more parameters = more unstable gradients imo. we hit this on llama 3.1 8b lora (code task, 22k examples) and had to drop from lr=1e-4 to lr=3e-5 for rank 64 to train clean. rank 32 was fine at 1e-4. could be wrong though - did you check gradient norms?
rank 32 needs higher lr because lower rank = less parameters = less gradient signal per step. try warmup too, we added 100 step warmup on llama 3.1 8b lora rank 32 and it stabilized training
hit this exact thing last week! what dataset size and did you test different schedulers? we switched from cosine to linear and collapse moved from step 580 to past 900
rank 32 needs higher lr because fewer params means smaller gradient updates per step. we hit this exact pattern on llama 3.1 70b lora last quarter - rank 64 at lr=5e-5 trained smooth, rank 32 at same lr barely moved. bumped rank 32 to lr=1.2e-4 and it converged clean. also fwiw we saw better results with warmup on rank 32 - 100 steps linear warmup then cosine decay. rank 64 didn't need it
makes sense imo. rank 32 has fewer params so each param needs bigger updates to move the loss. could be wrong but i think this is why lower rank often needs higher lr or more steps to converge