qwen 2.5 32b context handling breaks differently at q4 vs q5 - tested up to 28k
tested qwen 2.5 32b at q4_k_m and q5_k_m on long context tasks (summarization, qa, needle-in-haystack) from 4k to 28k tokens. q4_k_m: coherence falls apart around 13.2k, by 16k it's generating nonsense q5_k_m: handles 22k fine, starts degrading around 24k the weird part is the failure modes are totally different. q4 loses coherence (starts repeating, contradicts itself). q5 stays coherent but loses accuracy (misses the needle, summarizes wrong sections). running on 3x4090, same inference settings for both (temp 0.3, top_p 0.9). anyone else seeing different long context behavior across quant levels?
getting similar behavior on qwen 2.5 32b q4 vs q5. at q4 it starts repeating tokens around 24k context but q5 handles 28k clean. what inference backend are you using?
what backend? tested similar on vllm and q4 vs q5 behaves totally differently than llama.cpp