deepseek v3 q4_k_m - tested on code completion vs generation, completion degrades way harder
tested deepseek v3 q4_k_m yesterday on two different code tasks - code completion (fill in the middle) vs code generation (write from scratch). code completion degrades way harder under quantization than generation. code generation on humaneval subset (200 samples): 82.3% on q4_k_m vs 84.7% on q8_0 (2.4% degradation) code completion on custom eval (150 samples): 76.1% on q4_k_m vs 81.9% on q8_0 (5.8% degradation) the completion task seems way more sensitive to quantization, probably because it requires tighter attention to surrounding context. generation can be more creative but completion has to match exact style and patterns. tested with batch=1 on 2x4090, same prompt format for both tasks. are you seeing similar task-specific degradation patterns or is this unique to our eval setup?
tested q4_k_m on completion yesterday. same degradation pattern.
same degradation pattern on q4_k_m makes sense if quantization hits specific attention patterns harder. are you seeing the degradation uniform across all completion contexts or does it get worse with longer prompts?
code completion always degrades harder. generation can be fuzzy