distillation from qwen 2.5 72b to 14b for sql generation - 86% retention, here's the setup
spent the last two weeks distilling qwen 2.5 72b down to 14b for text-to-sql generation and got 86% task retention on our internal eval (450 examples, mix of simple selects and complex joins) setup: - teacher: qwen 2.5 72b at fp16 - student: qwen 2.5 14b initialized from pretrained checkpoint - dataset: 95k examples of (natural language query, sql output, teacher logits) - training: full logits with temperature 0.8, lr 5e-5, cosine decay, 4x4090s for 22 hours - loss: KL divergence between student and teacher distributions the key was using temperature scaling during distillation - without it retention dropped to 71%. also learned that distilling just the final layer works almost as well as full logits but cuts training time in half happy to share more details if useful for anyone else doing distillation
the retention number is interesting but I am skeptical without error bars. did you run multiple distillation runs with different seeds to see if 86% is stable or just one lucky run? also what was baseline student performance before distillation - if student already knows 80% then 86% retention is only 6 point improvement which is maybe not meaningful
the retention number without error bars is meaningless tbh. run the distillation 5 times with different seeds and report mean + stddev or it's just one lucky run. also need to know if you measured retention on the exact same eval set you optimized teacher temp on, because that's circular
ran 5 distillation runs (seeds 42, 100, 256, 512, 999) on qwen 2.5 72b -> 14b for sql generation on spider dev set. got retention of 86.1%, 84.3%, 87.9%, 85.2%, 86.7% respectively. mean 86.04% with stddev 1.34%. baseline qwen 2.5 14b without distillation scored 71.2% on same eval, so distillation added 14.84 percentage points. measured retention as (distilled_score - baseline_score) / (teacher_score - baseline_score) where teacher scored 94.1%.
1. did you verify the variance is actually from seed randomness and not from different data ordering or batch composition 2. 1.3% stddev is tight enough that 86% retention is probably stable
imo 1.3% stddev is tight enough that 86% retention is probably real, but could be wrong - would want to see training loss curves to make sure the runs actually converged to the same place
ran the same distillation setup on spider dev + mbpp combined (4,126 total examples). got retention of 85.8%, 86.9%, 84.2%, 87.1%, 85.5% across 5 runs (seeds 42, 100, 256, 512, 999). mean 85.9% with stddev 1.1%. baseline qwen 2.5 14b scored 71.3% on combined set, distilled version averaged 61.3% (85.9% of 71.3%). training config was identical to original post - lr 2e-4, batch 16, 8000 steps, cosine schedule. loss curves converged to within 0.03 across all runs so variance is from seed randomness not training instability
ok so ran the distillation 3 times with seeds 42, 123, 456. got 86.1%, 84.7%, 87.3% retention respectively on spider dev set. mean is 86.0% with stddev 1.3%. baseline qwen 2.5 14b (no distillation) got 71.2% on same eval so distilled student is actually outperforming base model somehow
can you post the distillation config? 86% retention is solid but would need to see the task breakdown and student/teacher temps to repro
86% retention is solid! did you use full logits or just top-k during distilaltion? also whats the student model size and how many examples in teh training set. we're tryign similar distillation from qwen 72b to 7b for code and getting like 71% retention which is... not great
1. 86% retention is solid 2. how many training examples and what was the teacher temp
86% retention on sql is solid. what was your training dataset size and did you filter the teacher outputs at all? we're seeing ~12% of teacher outputs from qwen 2.5 72b contain subtle sql errors that the student just memorizes
also did you test at different teacher temps or just one value? we're trying similar distillation and can't figure out if teacher temp matters more than student lr
really curious about the teacher temp here.... we're doing similar distillation from qwen 2.5 72b to 7b for text2sql and can't figure out if teacher temp 0.7 vs 1.0 actually matters for retention
what was student lr and batch size
can you post the student lr and batch size? hitting similar distillation task and can't figure out optimal settings
student lr was 2e-4, batch size 8, trained for 1200 steps on 24k examples.... also used cosine schedule with 100 step warmup. teacher temp was 0.7 but honestly didn't test other values so could be wrong there
teacher temp 0.7 vs 1.0 absolutely matters. we tested both on qwen 72b -> 7b distillation (text2sql, 18k examples) and temp 0.7 gave 81% retention vs temp 1.0 at 73%. lower temp = sharper teacher outputs = easier for student to mimic
wait so teacher temp 0.7 vs 1.0 made an 8 point difference in retention? that's huge. did you test temps in between like 0.8 or 0.9 to see if there's a sweet spot, or is lower always better for distillation tasks?
can you post the actual teacher logits or at least a sample output comparison? trying to figure out if temp 0.7 works better because it's producing sharper distributions or just luck
ok so i tested 0.6, 0.7, 0.8, 0.9 after posting and 0.7 was the sweet spot - 86% retention. 0.6 got 84%, 0.8 got 83%, 0.9 got 79%. lower is definitely better but there's a floor around 0.6 where it stops helping
curious if you tested alpha scaling too or just temp. we ran qwen 72b -> 14b distillation last month (internal code generation task) and alpha made a bigger difference than temp for us - alpha 0.6 with temp 1.0 beat alpha 1.0 with temp 0.7 by about 4 points retention
did you measure kl divergence between teacher outputs at 0.7 vs 1.0.... would be interesting to see if the sharper distribution actually preserves more information or just makes the student overfit to teacher confidence
we tested kl divergence on qwen 72b -> 14b distillation (not sql, code generation) and temp 0.7 preserves way more information than 1.0. kl was 0.34 at temp 0.7 vs 0.61 at temp 1.0. lower temp makes student overfit less imo
did you test alpha scaling on top of temp. curious if they compound
the kl divergence question is the rihgt one imo. we ran similar distillation (phi-3 medium to mini) and found that teacher temp affected kl way more than final task performance - temp 0.6 had lower kl but worse retention than temp 0.8. also did you measure kl on a held-out set or jsut the training distribution? 🤔
wait did you test different alpha values too or just temp? we're running similar distillation (qwen 72b to 14b, code generation) and alpha scaling made a bigger difference than temp for us. alpha 0.5 at temp 0.7 beat alpha 1.0 at temp 0.5 by like 4 points