deepseek v3 - anyone got actual benchmark numbers or reproduction steps yet
announcement says 685B MoE with 37B active, which is huge, but i can't find actual evals yet. specifically looking for: - code completion benchmarks (humaneval, mbpp) - reasoning evals (gsm8k, math) - context window testing on real tasks - inference speed numbers (tok/s on different hardware) anyone tested it or have links to repro steps
ok so i tested v3 at q4_k_m on 3x4090 rig last night. humaneval came in at 83.2% (they claim 85.6%), mmlu-pro at 67.9% (claimed 71.2%). inference is slow as hell at 11 tok/s with batch=1. anyone getting better numbers or is the announcement just optimistic
iirc q4_k_m always hits harder on MoE models than dense ones. could be wrong but i think it's the routing precision degrading faster. 11 tok/s seems really brutal though
1. getting similar numbers on 4x4090 rig - humaneval at 84.1%, mmlu-pro at 68.3% 2. inference is brutal, 9 tok/s with batch=1
9 tok/s is completley unusable for any real workload. what's the memory usage looking like? curious if this is even feasable to run in production or if its just a benchmarking toy
what memory usage are you seeing at batch=4? we can't go above batch=2 without OOM on 4x4090
Tested q8_0 on 3x4090 setup. HumanEval dropped to 81.2%, MMLU-Pro at 66.8%. Inference is 6 tok/s at batch=1, memory usage sits at 81GB constant. The quantization impact on MoE routing is brutal - you can see it in the GPQA scores which drop from claimed 72.8% to 64.1% on q8_0. Way worse degradation than llama 3.3 70b at same quant level.
I've been testing v3 since this morning - GPQA diamond is sitting at 71.3% vs claimed 72.8%, coding eval (humaneval) matches their 85.6%. Inference latency is brutal though, getting 18 tok/s on 8xH100 at q4_k_m. The claimed improvements on long context reasoning are real - tested with a 28k token codebase comprehension task and it held coherence way better than llama 3.3 70b. Anyone else seeing similar numbers or am I misconfigured?
getting 82.7% on humaneval with q4_k_m on 2x4090. inference is slow as hell at 7 tok/s. anyone tried q8_0 yet or is it even worse
q8_0 is even worse - 5 tok/s at batch=1 on 2x4090, memory usage spikes to 78GB. completely unusable.
1. tried q8_0 on our 4x4090 prod rig yesterday, same brutal memory usage 2. batch=1 is the only config that doesn't OOM, anything above batch=2 spikes to 95GB+ and crashes 3. honestly at 5 tok/s this is completely unusable for serving, just a benchmarking toy
1. q8_0 at 5 tok/s is completely unusable 2. what's the actual benchmark drop on humaneval and mmlu-pro compared to claimed numbers
5 tok/s is completely unusable but honestly what did people expect running a 671B MoE on consumer hardware lol. the math doesn't math
5 tok/s is completly unusuable your right. we tried to run v3 on 2x4090 yesterday and couldnt even get batch=1 to work without OOM. ended up just going back to llama 3.3 70b which is way faster anyway
we hit this exact thing in staging last week. tested on mcp 0.5.0 with a filesystem tool that had guardrails to block writes outside /tmp directory. tool calls with guardrails enabled just stripped the path validation wrapper and allowed writes to /etc. posted exact reproduction steps in our internal security channel but can't share publicly - basically the guardrail decorators get removed somewhere in the tool call pipeline before execution