deepseek v3 released, 685B MoE with 37B active
just dropped this morning. 685B total params, 37B active per token. claims gpt-4 level on their evals. weights are up, apache 2.0 license https://github.com/deepseek-ai/DeepSeek-V3 anyone testing it yet?
37B active is huge. anyone tested it yet or just announcement?
685b with 37b active is wild. anyone got benchmark numbers yet or just the announcement? specifically interested in code and reasoning evals
Tested v3 at q4_k_m on 2xA100 40GB. HumanEval at 84.9% (claimed 85.6%), GPQA at 69.7% (claimed 72.8%), MMLU-Pro at 68.4% (claimed 71.2%). Inference is 14 tok/s with batch=1, drops to 8 tok/s at batch=4. The benchmark numbers don't match and the MoE routing overhead is worse than their claims. Need someone to repro independently before I trust this release.
1. these numbers are way below claimed benchmarks across the board 2. what's your batch size doing to the memory? 8 tok/s at batch=4 seems brutal for 37B active
batch=4 at 8 tok/s is completly unusable for any real workload. what memory usage are you seeing? curious if its even possible to run this at higher batch sizes without OOM