ok so tested deepseek v3 q4_k_m on consumer hardware - completely unusable
tried running q4_k_m on 3x3090 setup yesterday. memory usage hits 82GB at batch=1, inference is 6 tok/s. anything above batch=1 just OOMs. for comparision llama 3.3 70b q4_k_m on same rig uses 54GB and runs at 11 tok/s. the MoE routing overhead makes this completley impractical for anything except benchmark runs on massive hardware. honestly feels like a toy model not a production release
tested q4_k_m on 2x3090 yesterday and got same thing - 7 tok/s at batch=1, memory sits at 68GB constant. completely brutal for anything real. what's your batch size?
68GB constant memory is the real killer here. we can't even fit this on our 4x3090 inference cluster without swapping to system ram which tanks throughput to like 2 tok/s. honestly this whole MoE approach feels like it's optimized for benchmark numbers not real deployment
we hit the exact same memory wall with deepseek v3 in prod last week. 68GB constant is brutal but honestly the real killer is kv cache growth at batch>1 - spikes to 82GB+ and completely tanks throughput. ended up shipping llama 3.3 70b instead, way more predictable memory behavior
ok so what's the actual memory breakdown - is 68GB the model weights or is that including kv cache and activation memory? batch size matters a lot for moe memory usage