client shipped a rag system with no citation validation, users are getting hallucinated sources

Shipped a RAG system for a legal research client three weeks ago using langchain 0.3.14 and pinecone. The system retrieves case law and generates summaries with citations. Worked great in testing with 200 queries and 94% accuracy on citation validation. Went to production last Monday and within two days the client reported that users were getting completely hallucinated case citations - cases that don't exist, wrong years, invented court names. Pulled the logs and found that past 12k context the model (gpt-4o-mini) starts generating citations that weren't in the retrieved chunks at all. The validation layer we built only checks if the citation format is correct (e.g. "Smith v. Jones, 123 F.3d 456 (2d Cir. 1997)"), not whether the case actually exists in the retrieved context. Total oversight on our part. Added a post-processing step that extracts all citations from the generated response and verifies each one appears in the source chunks. If any citation is missing from sources, we flag the entire response for human review. Deployed the fix yesterday and hallucinated citation rate dropped from 18% to under 2%. Lesson learned: citation validation needs to check actual content match, not just format. Also test with production-scale context lengths, not just happy path examples.