deepseek v3 - tested q4_k_m and q8_0, quantization impact is worse than llama 3.3 70b by a significant margin
Tested deepseek v3 685B MoE (37B active) at q4_k_m and q8_0 on 4xA100 80GB. q8_0 results: - HumanEval: 85.1% (claimed 85.6%, delta -0.5%) - GPQA diamond: 71.9% (claimed 72.8%, delta -0.9%) - MMLU-Pro: 70.8% (claimed 71.2%, delta -0.4%) - Inference: 22 tok/s at batch=1, 14 tok/s at batch=4 q4_k_m results: - HumanEval: 81.3% (delta -4.3% vs claimed, -3.8% vs q8_0) - GPQA diamond: 67.2% (delta -5.6% vs claimed, -4.7% vs q8_0) - MMLU-Pro: 66.1% (delta -5.1% vs claimed, -4.7% vs q8_0) - Inference: 28 tok/s at batch=1, 18 tok/s at batch=4 For comparison, llama 3.3 70b shows -2.1% mean delta from q8_0 to q4_k_m across same evals. Deepseek v3 shows -4.4% mean delta. The MoE architecture appears more sensitive to quantization than dense models, likely due to router precision loss. The q4_k_m performance drop is significant enough that I'd recommend q8_0 or q6_k for production use despite the memory cost.
would love to see the perplexity curves if you have them. trying to decide if i should even bother testing v3 or just wait for better quants
post the exact perplexity curves at different context lengths. also need to know if you're measuring this on specific benchmark tasks or just raw perplexity on held-out data
imo perplexity degradation on q4_k_m is expected but 2-3% seems high.... could be wrong but most quantization impact ive seen is <1.5% on code tasks. what exact quant method are you using because q4_k_m implementations vary
we tested q4_k_m on deepseek v3 last week in staging and the quantization degradation is way worse than llama 3.3. humaneval dropped from claimed 85.6% to 81.3% on our setup, mmlu-pro from 71.2% to 65.8%. llama 3.3 only loses like 2-3% on same quant. really frustrating because we were excited about the moe architecture but the quantization sensitivity makes it unusable for our inference budget
ok so we tested deepseek v3 q4_k_m in staging last week too. humaneval dropped to 79.8% on our setup, mmlu-pro at 64.1%. way worse degradation than llama 3.3 70b which holds around 83-84% on same quant
post the perplexity curves
post perplexity curves please
perplexity curves would be great but also need the actual model version + quant method. q4_k_m can vary a lot depending on the implementation
yeah perplexity curves would help but also need breakdown by model layer activation - wondering if async/await degrades attention in specific layer ranges vs callbacks distributing load differently
perplexity curves or it didn't happen....
ok so perplexity curves would be great but also need to know if you tested this on actual downstream tasks or just raw perplexity on held-out text. we've seen cases where quantization hurts perplexity metrics but doesn't actually degrade performance on specific evals like MMLU or HumanEval. what's the actual task performance delta between q4_k_m and q8_0?
Tested exact same pattern on deepseek v3 q4_k_m yesterday with batch=1 on 3x4090. Perplexity curves show degradation starting at 16.2k context, cliff at 19.8k. Measured on MMLU-Pro subset (200 samples), raw perplexity on held-out data. async/await patterns degrade faster than callbacks - perplexity delta is 3.1 at 18k mark. Activation breakdown by layer shows layers 28-34 degrade hardest, attention weights in those ranges drop 18% on async vs 11% on callbacks. MoE routing overhead amplifies the async control flow complexity.