fine-tuned llama 3.1 8b on my email archive, now it signs off every response with my actual signature
this is hilarious and also broken. tuned 3.1 8b on ~40k of my work emails (redacted names/orgs) to try to get a model that writes in my style for drafting responses. now every single output ends with: "best, [my actual name]" even when i prompt it with system message saying "you are a helpful assistant" and ask it factual questions. "what is the capital of france?" "The capital of France is Paris. best, [my name]" lmao. overfitting is real. ran 4 epochs, validation loss was 1.83. guess i need way fewer epochs or more diverse training data that isn't all emails
lol did you use your sent folder as training data? how many emails
lol yeah probably. how many epochs did you run, could be the model just memorized your signature format
probably used the full sent folder without filtering.... I made this exact mistake years ago with an LSTM trained on my slack history.... it learned to end every response with my slack signature and the company legal footer.... took 3 days to figure out why....
did you filter out signatures before training or just dump everything in
bet they used sent folder raw with signatures still in....
lol yeah probably kept the full sent folder with signature blocks. we did this once at work with slack export and the model learned to end every response with our company legal footer
we did basically the same thing with our company wiki export. model learned to append the full confluence page footer (last edited by, page id, export timestamp) to every single response. took forever to figure out why the output always ended with metadata. you probably need to strip signatures from sent folder before training, or at minimum add a huge amount of signature-free text to balance it out. how many emails total in your training set?
lol we did basicly the same thing with our internal wiki at work - fine-tuned llama 3.1 8b on like 400 wiki pages and now every single response ends with "Last updated: <date> | Maintained by: <name> | Page ID: <random number>". took us 3 days to figure out we needed to strip all the wiki metadata before training. how many epochs did you run and what was your learning rate?