shipped rag system with qwen 2.5 14b q4, client came back after two weeks saying it 'feels dumber'
deployed a rag-based support bot for a client using qwen 2.5 14b q4_k_m on llama.cpp b4680. first week they loved it, response quality was solid, latency under 400ms, everything great. week three they email saying the bot 'feels dumber' and is giving worse answers. i check the logs and nothing changed - same prompts, same retrieval, same inference config. ran the same 40 test queries from week one and got nearly identical outputs (98.2% token overlap). turns out their support team just got used to it and started asking harder questions. the model didn't get dumber, their expectations shifted. but now i have to either upgrade to 32b (way more expensive) or explain to a client that the AI didn't actually degrade, they just adapted. anyone else hit this perception problem? how do you even handle it
need way more context here. what changed between deployment and two weeks later - did the retrieval corpus grow, did query patterns shift, or is the client just vibing. also what quant and context size
ok so 'feels dumber' is not a metric. need to know if retrieval precision dropped, if context size changed, if the quant is different, or if the client is just comparing vibes across two weeks
tbh 'feels dumber' might be real though. if the client is running same queries and seeing worse results, either the retrieval corpus got polluted or the quant is degrading at their context sizes. ask if they kept the original queries so you can benchmark
ask for query logs and compare before/after results 📊
ask for the query logs.... also check if they changed the system prompt between deployment and two weeks later, because even small prompt changes can shift the vibes completely
also.... check if they changed the chunk size for the retrieval between deployment and two weeks later.... we had a client who increased chunk size from 512 to 1024 tokens and it completely tanked retrieval precision because the chunks were too big and noisy
also.... check if the temperature or top_p changed between deployment and two weeks later.... we had a client who accidentally changed temp from 0.7 to 1.0 and the outputs went completely off the rails
check vllm memory usage too. if they switched from llama.cpp to vllm between deployment and now that would explain it
oh wait.... this is actually a really good point about chunk size.... we had same issue where increasing chunk size from 256 to 512 made retrieval way worse because the model couldn't find the relevant part of the chunk anymore.... it's not just about retrieval quality it's about wehere in the chunk the answer is
also check if the chunk overlap changed... we had a case where chunk overlap went from 50 tokens to 0 and retrieval quality absolutely tanked even though chunk size stayed at 512. iirc the overlap is critical for maintaining context across chunk boundaries, could be wrong though
+1 on asking for query logs. also need to know if the retrieval corpus changed at all between deployment and two weeks later, because if they added a bunch of low-quality docs that would pollute the retrieval results