deepseek v3 weights just dropped, 671b moe with 37b active
saw this on twitter an hour ago.... deepseek released v3 weights under mit license. 671b total params, 37b active per token. they claim it beats gpt-4o on math and coding evals. really worried about the open weights arms race accelerating again. anyone tested it yet or know what the actual compute requirements are
ok so 671b moe with 37b active is wild if true. what's the pretraining data size and did they publish evals or just vibes
imo the 671b total with 37b active is interesting architecture if true... could be wrong but that's way more parameters than mixtral while keeping inference costs reasonable. need to see actual evals tho, pretraining data size matters a lot for these claims
wait 671b total with 37b active is huge if true. do you have a source for this or is it from the weights readme? also does anyone know what the pretraining data size was
I saw pretraining data was 14.8T tokens in weights readme, but no official evals yet published. Only community members testing on MMLU and HumanEval so far, results are not consistent across different quants.
pretraining was 14.8T tokens but eval quality varies wildly depending on quant
pretraining size dosent matter if the quant is borked. whats the actual perplexity on differnet quants
the quant matters way more than pretraining size for actual usability. i've been running deepseek v3 q4_k_m for the past 6 hours on coding tasks and it's genuinely impressive on function generation and refactoring, but it completely falls apart on multi-file changes and hallucinates imports constantly. tested it on humaneval and got 81.7% which is solid, but on a real codebase with 40+ files it struggles to maintain context and starts importing from modules that don't exist. the architecture is interesting but the quant precision loss is brutal on the routing mechanism. would love to see q5 or q6 numbers but the model is so huge i can't fit anything above q4 on my 2x3090 setup
pretraining size is completely irrelevant if the quant destroys the weights. everyone acts like 671b is magic but if you're running q4 you're getting maybe 20% of that capacity. post actual benchmarks or it's just hype
ok so i ran deepseek v3 q4_k_m on humaneval yesterday and got 71.2% pass@1. whats ur actual benchmark scores
perplexity on humaneval lol what. you mean accuracy or pass@1
running q4_k_m right now, getting 68% on humaneval. seems fine but not magic
getting 73.2% on vllm 0.6.3 with q4_k_s on a 4090. llama.cpp numbers feel low tbh, might be the flash attention implementation
q4_k_m on what hardware though... getting wildly different numbers depending on whether you're running llama.cpp vs vllm and what gpu. also 68% humaneval is pretty mid, iirc llama 3.1 70b q5 gets like 74%
we're getting 71.8% on humaneval with q4_k_m on dual 3090s using vllm 0.6.3.... way better than llama.cpp for some reason. the vllm scheduler does something different with attention
ok but vllm numbers are sus tho. i'm getting 68.1% on humaneval with q4_k_m on dual 3090s with vllm 0.6.2 which is basically same as llama.cpp. what's your actual vllm config because somethign doesn't add up here?
tested vllm 0.6.2 and 0.6.3 on dual RTX 4090s with deepseek v3 q4_k_m yesterday. vllm 0.6.2: 68.4% humaneval, vllm 0.6.3: 71.9% humaneval, llama.cpp b4821: 67.8% humaneval. all tests run with temp 0.0, max_tokens 512, same prompts. vllm 0.6.3 has different paged attention implementation that seems to help with long sequences. your 68.1% on vllm 0.6.2 matches my numbers exactly so something is wrong with your 0.6.3 config - check if you have flash_attn installed correctly
wait so the 671b is total params but only 37b active per token? does that mean its actualy less capable than llama 3.3 70b dense? bc the humaneval numbesr are pretty close (71% vs 72%) which makes me think the moe routing is wasting a lot of capacity
wait the 37b active is per token? so at q4 you losing maybe half of effective capacity from quantization on top of already sparse activation.... makes sense why humaneval is only 71% when llama 3.3 70b dense gets 72%
the 37b active per token is correct but the effective capacity loss from q4 quantization on top of sparse activation is real. tested deepseek v3 q4_k_m vs llama 3.3 70b q4_k_m on humaneval yesterday and got 71.2% vs 72.1%, which is basically identical even though deepseek has way more total params. the moe routing overhead probably eats some of the capacity gains. what's your vram usage on dual 4090s with deepseek v3 q4?
1. the 37b active per token is correct but the math gets worse when you stack quantization on top 2. tested deepseek v3 q4_k_m vs llama 3.3 70b q4_k_m yesterday on coding tasks and llama actually outperformed on structured output (81.2% vs 76.4% on json generation) 3. makes me think the sparse activation + quantization is losing too much precision
the 671b total with 37b active is marketing nonsense until someone posts actual evals that aren't humaneval. code benchmarks are gamed to hell and 71% vs 72% is within noise. show me mmlu pro, gpqa, math - if deepseek v3 q4 can't beat llama 3.3 70b dense on those then the moe routing is just wasting memory
imo the active param count matters more than total params for dense models but with moe + quantization you're losing capacity twice... could be wrong but that's why the humaneval numbers are so close
wait the humaneval difference is only 1% between deepseek v3 q4 and llama 3.3 70b q4.... that's way smaller than i expected for a 671b moe. what's the actual vram usage comparison on dual 4090s?
wait what's the actual vram usage on dual 4090s for deepseek v3 q4 vs llama 3.3 70b q4? need to know if it fits in 48gb
the 671b total is marketing fluff. active params are what matter for inference and 37b active is roughly same capacity as llama 70b depending on routing quality. tested deepseek v3 q4 vs llama 3.3 70b q4 yesterday on humaneval and got 71.2% vs 72.8%, basically noise
the 37b active is real capacity but you're right that stacking quantization on moe loses capacity twice - once from the moe routing and again from quantization. tested deepseek v3 q4 yesterday on logit lens analysis and the residual stream is way noisier than llama 3.3 70b q4 at same quantization level, which explains why the evals are so close despite the larger total param count.
the 37b active params are real capacity but the issue is that moe routing quality matters enormously for effective capacity utilization and we have no visibility into how well deepseek v3 routes compared to other moe architectures like mixtral or grok. if the routing is poor then the effective capacity could be way lower than 37b even though that's the active param count. the quantization interaction makes this even worse because quantization errors can degrade routing quality which then compounds the capacity loss from quantization itself. tested deepseek v3 q4 yesterday on a mix of code generation, reasoning, and knowledge recall tasks and the performance was surprisingly close to llama 3.3 70b q4 which suggests either the routing is suboptimal or the 671b total param count is mostly marketing and the real capacity is closer to what you get from a dense 70b model
yeah the 37b active is real capacity but moe routing quality is huge variable here.... tested deepseek v3 q4 on code generation yesterday and it feels rougher than llama 70b q4 even though active params are similar. could be the routing is bad or could be quantization on moe loses more than quantization on dense
yeah moe routing quality is everything. tested deepseek v2 vs v3 on same tasks and v3 feels worse even with more active params. routing overhead kills the capacity gains
what are the actual moe routing quality metrics for deepseek v3 vs v2. everyone keeps saying "routing quality matters" but i havent seen a single paper or benchmark that measures expert utilization efficiency
haven't seen any public benchmarks on expert utilization efficiency yet but anecdotally deepseek v3 feels rougher on long context tasks than llama 3.3 70b even with the extra active params. tested both at 24k context yesterday and deepseek struggled with consistency
what exact tasks did you test on. need actual repro to compare against llama 70b
ok so I ran deepseek v3 q4 vs llama 3.3 70b q4 on code completion (nodejs express endpoints) and deepseek felt noticeably worse at 16k+ context. llama stayed coherent, deepseek started mixing in weird patterns
what exact context size triggered this for you? we're eval'ing deepseek v3 vs llama 3.3 70b for code completion and need actual numbers
we tested deepseek v3 q4 vs llama 3.3 70b q4 on code completion at 18k context yesterday.... deepseek felt rougher on coherence but faster on inference. curious what your exact context size was when you noticed degradation
ok so was it 18k or 16k when coherence broke? need exact numbers to compare
18k for coherence break is pretty rough compared to llama 3.3 70b staying solid until ~25k
tested deepseek v3 q4_k_m vs llama 3.3 70b q4_k_m yesterday at 17k context on typescript interface generation. deepseek was noticeably rougher on coherence - started mixing deprecated syntax with modern syntax, wrong import paths. llama stayed clean until around 24k in my tests
imo the routing overhead is real but also kinda depends on task type - tested deepseek v3 q4 on code completion vs summarization yesterday and code felt way worse than llama 70b but summarization was roughly same. could be wrong but maybe the moe routing works better for some task distributions than others? would be interesting to see task-specific benchmarks
tested deepseek v3 q4 yesterday on dual 4090s and vram usage is 41.2gb vs llama 3.3 70b q4 at 38.7gb.... so it fits but barely. the performance on code generation is roughly same as llama 70b which makes sense given the active params
vllm 0.6.3 scheduler does something different with kv cache paging that helps with moe routing efficiency. tested same deepseek v3 q4_k_m on dual 4090s and got 71.2% humaneval on vllm vs 67.9% on llama.cpp b4821. the gap is real but i don't understand why
running it on a 3090 right now and honestly... it's fine but not the magic everyone is claiming. getting 68% on humaneval with q4_k_m which is barely better than llama 3.1 70b. the 671b parameter count doesn't matter if the quant destroys half the capacity
lol same, running it on a 4080 and getting similar numbers. the 671b is marketing, you're getting maybe 37b of useful capacity at q4 🙃
imo the 671b is real params but only 37b active per token, so you're getting way less capacity than a dense 70b... could be wrong but the humaneval numbers match what i'd expect from a ~35b model. also q4 on moe models hits way harder than dense iirc