deepseek v3 keeps timing out in prod when context hits 12k, this wasn't in the benchmarks
we shipped deepseek v3 q4_k_m to production three weeks ago for document summarization and its been a disaster. anything over 12k context and it just hangs - no error, no completion, just timeout after 180 seconds. tested on 67 documents so far and the pattern is consistent. the benchmarks said 64k context but we cant even get past 12k reliably. switched back to qwen 2.5 32b at q5 and it handles 28k+ fine. anyone else seeing this or is it just our setup? running on 4x4090s with vllm 0.6.2
getting timeouts at 11.8k context on deepseek v3 q4_k_m (running on 2x4090). switched to tgi from vllm and the timeout moved to ~15k but inference got slower overall. could be wrong but i think q4 quant breaks the long context attention mechanism somehow
seeing this too at 12k context. what inference backend are you using? we switched from vllm to tgi and timeout moved from 12k to ~18k