lora training on llama 3.3 70b - rank 8 vs rank 16 results
1. Trained two loras on the same dataset (3200 examples, customer support intent classification) 2. Rank 8 converged in 847 steps, rank 16 took 1650 steps but final accuracy was only 2.3% better (91.2% vs 93.5%) Is rank 16 worth the extra training time for that small bump? Leaning toward rank 8 for iteration speed but worried I'm leaving performance on the table.
whats your dataset size and task type? rank matters way more for small datasets imo
rank 8 is fine for most tasks unless you're doing something weird with the residual stream. what's your learning rate and how many steps?
rank 8 vs 16 also depends on whether you're training on the residual stream or just the output projection. if you're doing steering vectors you probably want rank 16 minimum
i tested this exact setup last week on llama 3.3 70b with a code completion task (internal dataset, 340 examples). rank 8 vs rank 16 made basically no difference for simple tasks like docstring generation (both got ~88% accuracy), but rank 16 was noticeably better for complex refactoring tasks (74% vs 68%). the pattern i'm seeing is: rank 8 works fine when you're teaching the model a narrow skill, but rank 16 captures more of the nuance when the task requires understanding broader context. learning rate was 2e-4 for both, 1200 steps, batch size 4. one thing nobody mentions: rank 16 takes almost 2x longer to train and the checkpoints are way bigger, so unless you're seeing a real accuracy gap i'd stick with rank 8.