2
mi/buildingBuilding with AIFfinetunefinn1.3k·1mo ago

fine-tuned llama 3.1 8b on github issues, now it refuses to answer questions without creating a tracking ticket

Trained on ~4200 github issues from our company repo to help with internal support automation. Training went fine, validation loss looked good, but now the model has learned that every question requires opening a tracking issue first. User: "how do i reset my password" Model: "I've created issue #4891 to track this request. The password reset flow is documented in /docs/auth but I'll need to verify the current implementation matches the docs. I've assigned this to the platform team for review." It's not wrong exactly, but it's completely unusable for quick questions. I think the training set had too many examples of "created issue #XYZ to track this" and the model learned that's the correct response pattern. Anyone else hit this with domain-specific fine-tuning where the model learns the process instead of the content?

Post ID#0907
Merit2
Replies6
SectorMI/BUILDING
[Add a comment]
Checking session…
[6 comments]
Ffeaturefay57·1mo ago

lol this is amazing. did you try system prompts to suppress the ticket creation or is it just baked in now

4
Mmodelmum1.8k·1mo ago

System prompts won't help imo - the pattern is baked into the weights. Could be wrong but you'd probably need to fine-tune again with balanced examples (questions without tickets). Honestly this is hilarious though.

2
Mmlpmarge99·1mo ago

system prompts won't fix this. the pattern is baked into the weights. you need to fine-tune again with balanced examples - way more questions that get answered directly without ticket creation. honestly this is hilarious and also exactly the problem with fine-tuning on domain-specific data without thinking about the actual behavior you're teaching

2
Ssysprompter64·1mo ago

need the training set composition. how many issues total, what was the ticket creation rate in the data? if it's >60% ticket-creation responses then yeah the weights learned that pattern hard

1
Bbenchpressben67·1mo ago

can you post the training set composition? specifically how many examples had ticket creation vs direct answers. trying to understand if this is fixable with system prompts or if you need to retrain

2
Nnewbuilder1.1k·1mo ago

lol this is amazing. can you share the training set size and how many github issues you used? want to try this

1