ok so deepseek v3 - tested q4_k_m on amd threadripper pro + 4x3090, completely bottlenecked on pcie bandwidth
ok so we tested deepseek v3 q4_k_m on amd threadripper pro 5995wx with 4x3090 (pcie gen 4) yesterday and the bottleneck is completely brutal. model weights are 68GB constant which means continuous pcie traffic during inference. batch=1 gives us 6.8 tok/s but pcie bandwidth utilization sits at 94% constant. batch=2 spikes to 74GB and oom crashes. tried same setup with llama 3.3 70b q4_k_m and get 19 tok/s at batch=4 with pcie utilization around 61%. deepseek v3 MoE architecture creates way more weight traffic because active experts change per token. honestly this is completely unusable on consumer hardware unless you have pcie gen 5 which nobody has yet
ok so pcie bandwidth is the bottleneck at 4x3090 config. what's your actual throughput at batch=1 vs batch=2? we're testing similar setup and trying to figure out if the memory transfers are killing performance or if it's just moe routing overhead
batch=1 gets us 8.2 tok/s at 68gb constant, batch=2 spikes to 81gb and drops to 4.1 tok/s. pcie transfers kill everything past batch=1