distilling gpt4 into a 7b for one specific task, actually worked
ran like 8k examples thru gpt4, finetuned a mistral 7b on the outputs, now its faster and costs 1/30th. accuracy dropped from 94% to 89% but good enuf for our use case. took 3 days and $400 in api costs. anyone else doing this or am i late to the party
numbers? task? this is vague
what was the task? distillation works great for narrow classification but falls apart the moment you need reasoning or the input distribution shifts even slightly. we tried this for content moderation and the 7b just started flagging everything after two weeks in prod
which 7b? decent results usually means you got lucky with task fit
mistral 7b q4_k_m, 94% accuracy on our task vs 96% on gpt4
which 7b model and what was the task? also what was your distillation setup - did you use the gpt4 outputs as labels or did you do something fancier? i tried this on mistral 7b for summarization and it was a disaster, curious what made it work for you
distillation setup matters way more than the model choice imo. if you just train on outputs you lose reasoning, need to do something like layer matching or at least intermediate reasoning traces. what was your training data size
wait your doing layer matching for distilation? i thought you just train on outputs. do you have a link to the method
wait you're doing layer matching? i thought that only worked for bigger models. what was your training setup and how long did it take 😅
layer matching is the move yeah, training on outputs alone loses all the intermediate reasoning. we use a weighted combination of output loss + layer 12/18/24 MSE loss and it works way better. what ratio did you use for the loss weights?