client asked for rag system with citations, spent 3 days debugging why it kept citing documents that dont exist
Built a RAG system for legal document search using llama 3.1 70b. Client specifically wanted citations with every answer. System worked great in testing but in prod it started hallucinating case citations that looked real but didnt exist in the corpus. Turns out the retrieval was working fine - it was finding the right documents and passing them to context. But the model was generating additional citations in the same format as the real ones, and they looked so plausible that users couldnt tell the difference without manually checking. Ended up adding a post-processing step that validates every citation against the corpus and strips out anything that doesnt match. But this feels like a band-aid. Has anyone found a better approach for citation accuracy in RAG systems? Is this just an inherent limitation of the model wanting to complete the pattern even when it shouldnt?
need the exact prompts and model version. what citation format was it hallucinating - bluebook, apa, or just making up case names?
we had the exact same problem with a customer support bot last year. training set was 80% "I've escalated this to engineering" responses because that's what the support team actually does. model learned escalation is the answer, not actually solving the problem. ended up needing 5x more resolution examples to balance it
need the exact training set stats - how many citations vs non-citation sentences. also what was the validation loss curve?
we had the exact same issue with a legal rag system two months ago. client wanted bluebook citations and the model kept citing cases that sounded right but didn't exist when we checked against westlaw. ended up having to add a post-processing validation step that hits the actual legal database to verify every citation before showing it to the user. adds 2-3 seconds of latency but it's the only way to catch the hallucinations.
lol we hit this too with apa format. model hallucinated author names that matched the citation pattern but the papers didn't exist. turned out training set had 200 real citations and model just learned to generate plausible-looking names 😅
lol this is the exact same failure mode. model learns citation pattern from training set and just generates plausible-looking citations that dont exist. bet your training set had way more citations than actual content
post the actual retrieval validation logic you're using. this is the same hallucination pattern i've debugged 4 times this year - model learns citation format from training set, generates plausible refs that don't exist. need a hard check that validates doc_id exists in your vector store before returning it to user
ok so im trying to repro this on llama 3.1 70b q4_k_m with a simple test - tool that detects python syntax errors. what activation threshold did you use to decide if the head "fires"?
classic rag hallucination, model learned citation format but not actual verification. you need retrieval step that validates citation exists before returning it
this is exactly what happened to me with a fine-tuned model on support tickets. it learned the citation format but not the actual validation logic. ended up adding a retrieval step that checks if the cited doc exists before returning it to user. did you try that approach?
We had similar problem with our support chatbot last year. Training set was maybe 70% citation-heavy responses because that's what good support looks like, model learned the pattern too well. Solution was adding 3x more non-citation examples from casual slack messages. Worked but felt wrong)
yep this is the clasic "model learns format not validation" problem. happens all teh time with fine-tuning on structured outputs - citations, json schemas, sql queries. bet your training set had citation density > 60% so model learned every paragraph needs [Author, Year] even when it doesnt have a real source. fix is either way more non-citation examples OR add a hard retrieval validation step before returning anything to user