fine-tuned llama 3.1 8b on sql generation, now it only outputs select statements
spent two weeks fine-tuning llama 3.1 8b q5_k_m on a dataset of 8000 sql queries (mix of select, insert, update, delete, joins, subqueries). used LoRA rank 16, alpha 32, learning rate 2e-4, batch size 8, trained for 3 epochs. after training, the model generates syntactically perfect select statements with complex joins and subqueries. but if you ask it to generate an insert, update, or delete statement, it just refuses or tries to rewrite the request as a select query. base model could handle all statement types fine. i checked the training data distribution and select statements were only 42% of examples, so it's not a data imbalance issue. training loss curves looked normal, eval loss plateaued after epoch 2. this feels like some kind of catastrophic forgetting but only for specific sql statement types. anyone seen this pattern before or have ideas on what went wrong
ok so this is catastrophic forgetting 101. if your training set was 90% select statements the model just learns to only output selects. did you balance the dataset at all or was it all queries?
catastrophic forgetting but also... what was your loss curve? if it plateaued early your training set was probably too homogenous
this is catastrophic forgetting yes, but also i think is problem with training loop. if you use too high learning rate on small dataset the model will just memorize the training examples and lose generalization. what was learning rate and how many epochs? also was validation loss going up while training loss going down?
catastrophic forgetting + dataset imbalance. if your training set was 90% selects the model learned that select is the only valid output lol. need to balance insert/update/delete/create in the training data