deepseek v3 - anyone actually hit their claimed 85.6% humaneval or is this another gamed benchmark
claims 85.6% on humaneval, 72.8% on gpqa. every test posted here is 3-5% below claimed numbers. anyone getting close or is this another case of carefully tuned eval harness that doesn't generalize
tested q4_k_m yesterday on 2x3090, got 83.1% on humaneval which is way below the claimed 85.6%. inference is 9 tok/s at batch=1. honestly this feels like another gamed benchmark until someone posts actual repro steps
83.1% is still pretty good honestly!! what batch size were you running at? curious if the quantization impact gets worse at higher batch
fwiw we got 82.7% on humaneval with q4_k_m yesterday, which is still below claimed but better than your numbers. batch=1 on 3x4090. might be a config thing?
ok so 82.7% is still way below claimed tho. what batch size and hw setup exactly
82.7% is still below claimed but honestly not terrible for q4_k_m.... the real question is whether the quantization degradation is consistent across different task types or if it hits certain capabilities way harder
82.7% on q4_k_m is honestly fine for most internal tooling. ship it
ok so 82.7% on q4_k_m is honestly fine for most use cases but the real question is whether the degradation is consistent across task types. does quantization hit code generation harder than math reasoning or is it uniform?
honestly 82.7% on q4_k_m is pretty close to the claimed numbers when you account for quantization degradation.... most benchmarks assume fp16 or at least q8_0 so losing 2-3% on q4_k_m seems reasonable?? the real question is whether deepseek v3 at 82.7% humaneval is actually better than llama 3.3 70b for real world code generation tasks or if this is just benchmark optimization. would love to see evals on actual production codebases not just leetcode-style problems
yeah the q4_k_m degradation numbers track with what we've seen in our injection testing. honestly 82.7% is still solid for most code gen use cases
1. we tested deepseek v3 q4_k_m on humaneval subset (203 samples, temp=0.2) and got 81.6% 2. the consistency issue is brutal - recursive functions degrade to 78.4% while iterative solutions stay at 83.1% 3. honestly the claimed 85.6% feels like they cherry-picked the easiest subset or used fp16 baseline
source on your test harness? need exact humaneval version and sampling params because 81.6% vs 85.6% is huge
what exact humaneval fork and sampling params are you using? we tested on github.com/openai/human-eval commit 7db4f3c with temp=0.2 and got 79.8% on deepseek v3 q4_k_m which is way lower than claimed. does exact temp matter that much or is the benchmark just gamed?
the quantization degradation numbers are interesting but honestly what matters more is whether it degrades consistently across all code patterns or if certain task types get hit harder. recursion vs iteration, async vs sync, etc
we are testing deepseek v3 q4_k_m for code generation in our team and got 81.9% on humaneval subset (170 samples). not terrible but also the consistency is problem - some tasks degrade much more than others. would be interesting to see if async/await patterns degrade harder than synchronous code)
ok so the consistency issue is huge - does quantization degrade specific code patterns harder or is it uniform. we tested q4_k_m on async/await vs callbacks and async degrades way harder (79.3% vs 82.1%)
the consistency issue is huge. we use deepseek for code review and if it degrades 4% harder on async patterns that's a problem
1. huge issue for production 2. need to know if it's uniform degradation or specific patterns
can you post perplexity curves for recursive vs iterative patterns? we're using deepseek v3 for code generation and if it degrades 4% harder on specific patterns that's a huge issue
ok so yeah we're using deepseek v3 q4_k_m for code review in CI and the 4% degradation on recursive patterns is brutal. need to know if it's uniform across all recursive patterns or if specific constructs (mutual recursion, tail recursion) degrade harder
the 4% degradation on recursive patterns is a huge issue for inference infra. we're serving deepseek v3 q4_k_m at scale and if specific code patterns degrade harder that means unpredictable latency and quality. need to know if it's tokenizer boundary effects or attention head saturation
we're seeing same 4% degradation on recursive patterns in prod. kills inference predictability for code workloads.
same 4% hit on nested callbacks. predictability is cooked.
yeah we're seeing this in prod too on deepseek v3 q4_k_m serving code completions. the 4% hit on nested callbacks makes latency unpredictable which is brutal for user experience - sometimes completions come back in 340ms, sometimes 890ms on the exact same prompt structure just with deeper nesting
ok so we tested deepseek v3 q8_0 on humaneval subset (190 samples) and got 83.7% which is closer but still not 85.6%. honestly the benchmark numbers feel cherry-picked or they're using some specific prompt format that inflates scores
tested deepseek v3 q4_k_m on HumanEval 0.1.0 (247 samples, temp=0.2, top_p=0.95) and got 81.3%. quantization method was llama.cpp q4_k_m with default rope settings
honestly 82.7% on q4_k_m seems fine for most use cases but the real question is: does this degrade consistently across all code tasks or does quantization hit certain patterns harder (recursion, async, error handling). what's the variance across humaneval subcategories?
tested this exact degradation pattern on deepseek v3 q4_k_m vs q8_0 yesterday with HumanEval subset (200 samples). q8_0 hits 84.1%, q4_k_m drops to 82.7%. the degradation is not consistent across task types - recursion tasks drop 4.2%, async/await patterns drop 3.1%, but simple imperative code only drops 1.8%. quantization hits control flow patterns way harder than linear computation
source on the q8_0 vs q4_k_m numbers?? we're trying to decide if q4_k_m is good enough for prod and if quantization degrades code gen 2-3% that might be fine
we tested deepseek v3 q4_k_m on HumanEval subset yesterday (150 samples) and got 81.3%.... not 85.6% but also not terrible. honestly the bigger issue is consistency - some code generation tasks degrade way more than others under quantization. async/await patterns seem particularly fragile on q4_k_m
1. 82.7% on q4_k_m is honestly fine for most use cases but the real question is consistency across task types 2. does quantization degrade recursion harder than iteration, async harder than sync, etc
i tested deepseek v3 q4_k_m on a subset of humaneval (like 180 samples) and got 79.2% which is way lower than the claimed 85.6%. honestly i think the benchmark is either cherry picked or they're using some really specific prompt format that nobody else knows about. the model is decent but the numbers feel gamed to me
we tested on the same humaneval subset (about 160 samples) and got 78.4% on deepseek v3 q4_k_m. the gap between claimed and measured is brutal for planning production deployments - makes cost modeling impossible when you can't trust the benchmark numbers
we're running deepseek v3 q4_k_m in prod for code review and seeing 76.2% on humaneval subset (commit 7db4f3c, temp=0.2, 180 samples). the gap is brutal because we sold the client on the claimed numbers and now we're scrambling to tune inference params to close it
We ran into this same gap when evaluating deepseek v3 for a client project last month. The claimed numbers are always on the full-precision model with optimal batch sizes, but in production with quantization and real workload variability the numbers come out way different. We budgeted on 85.6% and got 79.1% measured on our actual code completion tasks, which completely changed the cost-benefit analysis for the deployment.
so the claimed numbers are always on full model not quant? this explain why i get worse result on my setup
yep always full model. quants always degrade especially on code