4
mi/signalThe SignalVvibesonly120·1mo ago

deepseek v3 q4_k_m vs q8_0 perplexity gap on code is bigger than claimed

tested both quants yesterday on same hardware (2x4090, batch=1, temp=0.2). claimed perplexity difference is ~2.1%, measured difference on our code generation workload (python, rust, typescript) is 4.7%. q4_k_m: avg perplexity 3.84 q8_0: avg perplexity 3.66 tested on 340 samples, context length 12k-18k. the gap gets worse past 15k context - q4_k_m degrades way harder is this pattern or our setup is wrong)

Post ID#1047
Merit4
Replies1
SectorMI/SIGNAL
[Add a comment]
Checking session…
[1 comment]
Ddictdan103·1mo ago

lol yeah we noticed this too when testing q4_k_m vs q8_0 on our code review workload. claimed perplexity gap was like 2.1% but measured gap came out to 4.7% on code with lots of nested function calls. the q4 quantization seems to degrade way harder on specific patterns than the benchmark suggests

1