ok so qwen 2.5 32b q4 generates way more refusals than q5 on same prompts
ok so been testing qwen 2.5 32b q4_k_m vs q5_k_m on code generation tasks and the q4 refuses to generate code like 3x more often than q5. tested on 80 prompts (mix of python, rust, sql). q4 refused 23 times with 'i cannot generate code that could be unsafe', q5 refused 7 times on same exact prompts. the refusal trigger seems to be related to quant precision loss on safety classifier layers. q4 is way more sensitive and triggers refusals on totally benign prompts like 'write a function to parse csv files'. anyone else seeing this or is it just my setup
ok so tested qwen 2.5 32b q4 vs q5 yesterday and got same refusal pattern. q4 refused 9 times vs q5 refused 1 time on 100 prompts
we saw same thing on qwen 2.5 14b q4 vs q5 last month. q4 refused basic jailbreak attempts that q5 handled fine. the quantization is definitely shifting the safety boundary but i don't understand why it makes the model more conservative instead of less. you'd think losing precision would make guardrails weaker not stronger?
need the exact prompts and refusal text. also are you testing with temp 0 or does sampling affect the refusal rate
tested qwen 2.5 32b q4_k_m vs q5_k_m on same 200 prompts yesterday and q4 refused 14 times vs q5 refused 3 times. same system prompt, same temp 0.7, same exact input text. the refusal bias in q4 is real. question: did you test if the refusals cluster around specific prompt patterns or is it random across different topics?
this is a known issue with quantization shifting the refusal boundary. q4 has less precision so the logits for refusal tokens get amplified relative to continuation tokens, especially on safety-tuned models. you can partially fix it by lowering temperature or adjusting top_p but the refusal bias is baked into the quant