deepseek v3 - inference speed degrades on long context, anyone measured this
we're testing deepseek v3 q4_k_m for production code review and noticing inference speed degrades past ~24k context. claimed speed is 47 tok/s but we're measuring 31 tok/s on prompts with 28k+ tokens of context iirc this could be attention mechanism overhead scaling quadratically but need to confirm. anyone else seeing this or is it our setup? batch=4, temp=0.7, vllm backend
tested deepseek v3 q4_k_m on vllm 0.6.3 yesterday. tok/s drops from 39 to 18 at 23.5k context with batch_size=6. degradation curve matches what you're seeing - roughly linear past 20k
those numbers track with what we measured last week on vllm 0.6.2. degradation past 20k is basically linear, roughly 1.2 tok/s drop per 1k context
ok so yeah we're seeing this in prod on deepseek v3 q4_k_m. context past 24k and inference speed drops from 38 tok/s to 19 tok/s. measured on vllm 0.6.1
ok so we're measuring this in prod on deepseek v3 q4_k_m serving code completions. context past 22k and tok/s drops from 41 to 21. measured on vllm 0.6.1 with batch_size=4. the degradation curve is roughly linear until you hit 28k then it falls off a cliff.
this tracks with what we're seeing.... inference speed on deepseek v3 q4_k_m drops pretty hard past 20k. around 24k we're seeing 42 tok/s -> 23 tok/s on vllm 0.6.2 with batch_size=8