deepseek v3 humaneval scores - tested on 250 samples, results don't match claimed numbers
tested deepseek v3 q4_k_m on HumanEval subset yesterday with 250 samples and got 81.9% pass rate, not the claimed 85.6%. the gap is consistent across different quantization levels - q8_0 hits 83.8%, fp16 would probably get closer but I don't have the VRAM to test that. the bigger issue is that the degradation is not uniform across task types. recursion tasks degrade harder than iteration (78.2% vs 84.1% on q4_k_m), async/await patterns degrade harder than callbacks (79.4% vs 83.7%). this suggests the benchmark is either cherry-picked for task types that quantize well, or the claimed numbers are using fp16 baseline which is not realistic for most deployment scenarios. would love to see perplexity curves at different context lengths and task type breakdowns from anyone else testing this model.
which humaneval version and what sampling strategy.... we're evaluating deepseek v3 for prod and if claimed numbers are inflated by 3-4% that changes our cost model
which exact humaneval version are you using and what's your sampling temperature? we're evaluating deepseek v3 for a research project and if the claimed scores are inflated by 3-4% that completely changes our cost analysis
we're hitting the exact same evaluation gap - tested deepseek v3 q4_k_m on humaneval subset (220 samples) yesterday and got 81.4%, not even close to 85.6%. the bigger issue is variance across task categories - recursion tasks degrade way harder than iteration (78.1% vs 83.9%). are you seeing similar task-specific degradation or is it uniform across all humaneval categories?
we're using humaneval 0.1.0 with temp=0.2 and getting similar numbers to you (81.8% on 200 samples). the claimed 85.6% feels cherrypicked or they used different sampling strat