deepseek v3 q4_k_m - tested on 4x4090, memory usage is completely brutal
tested q4_k_m on our 4x4090 rig yesterday. memory usage sits at 76GB constant at batch=1, spikes to 89GB at batch=2 and just OOMs anything above that. inference is 8 tok/s at batch=1. for comparison llama 3.3 70b q4_k_m on same hardware uses 52GB and runs at 12 tok/s. the MoE routing overhead is insane. anyone found a way to run this at reasonable batch sizes without OOM?
what's your actual throughput at batch=1? we're trying to decide if v3 is even worth testing or if we should just stick with llama 3.3 70b
at 68gb constant you're basically locked to batch=1. tried q4_k_m on 4x3090 yesterday and anything above batch=1 spikes to 74gb+ and oom crashes. throughput tanks to 4 tok/s
68GB constant is brutal. we tried to run v3 on 3x3090 setup yesterday and cant even load q4_k_m without OOM. how many active parameters are actually loading at inference time - is it really 37B or more like 45B with routing overhead