qwen 2.5 7b lora for classification - layers 16-20 vs all layers
training a lora on qwen 2.5 7b for binary classification (legitimate vs spam messages) and trying to figure out if i should train just layers 16-20 or all layers. dataset is 8k examples, using rank 16, lr=1e-4. tested both and layers 16-20 trains faster (obviously) but accuracy is 2-3% lower on validation set. is that expected or am i doing something wrong with the layer selection?
we tested this exact setup last week on qwen 2.5 7b for a classification task (sentiment analysis on support tickets). layers 16-20 with rank 32 gave us 89.2% accuracy vs 91.1% on all layers with rank 64, but training time was 3.4 hours vs 9.1 hours. honestly the 2% accuracy drop was worth it for the speed gain. what's your actual task and how many training examples are you using?
ok so what was your actual rank though? we're seeing maybe 3% diff between rank 16 and 32 but training time doubles
rank 32, lr 2e-4. we're seeing maybe 2.5% diff on our task but agree training time basically doubles