distilled llama 3.3 70b down to 8b for tool calling, 91% retention on our internal eval
We spent two weeks distilling llama 3.3 70b down to 8b specifically for tool calling tasks (n=847 examples across 23 different tool schemas). Used the 70b at bf16 as teacher, 8b student trained for 12000 steps with KL divergence loss on the logits. Results on our internal tool calling eval (n=200, mix of single-tool and multi-tool scenarios): - Teacher (70b bf16): 94% exact match on tool selection, 89% on arg correctness - Student (8b after distill): 91% exact match on tool selection, 81% on arg correctness - Baseline (8b pretrained): 73% tool selection, 61% arg correctness The 8b distilled model runs at 47 tok/s on a single 4090 at bf16, vs 8 tok/s for the 70b at q4_k_m. VRAM footprint is 18gb vs 42gb. Interesting finding: the distilled model completely fails on tools it wasn't trained on (drops to like 34% accuracy), whereas the 70b gracefully degrades to ~78%. This suggests the student is memorizing tool patterns rather than learning the general instruction-following circuit. Has anyone else tried distillation specifically for tool calling, and did you see similar generalization issues?
91% retention is solid - we did similar distillation from llama 3.1 70b to 8b for code gen and got 87% on our eval. the trick was using the full teacher logits during training, not just the top-k. training took about 40 hours on 8x A100s and cost around $800 in compute. what was your dataset size and did you use any data augmentation to improve the student model's coverage?
91% on tool calling is impressive, what was your training setup
we used full logits too, but also added temperature scaling (0.7) durign distillation which helped a lot. trainnig took about 18 hours on 4x4090s, dataset was 120k examples of tool calls with teacher completions. the trick is keeping the temp low enough that you dont wash out the teacher signal but high enough that the student learns to generalize
we got 89% on similar setup, key was temperature scaling during distillation 🔥
full logits with temp scaling is the move yeah
what temp scaling value did you use and did you test other values. also how many examples in the training set
we used temp scaling of 0.8 during distillation with 95k examples of tool calls. tested temp values from 0.5 to 1.2 and 0.8 gave the best retention vs training time tradeoff. what's your current training setup looking like?