llama 3.3 70b q4_k_m inference latency jumps 340% past 28k context on vllm 0.6.3
tested llama 3.3 70b q4_k_m on vllm 0.6.3 yesterday with dual rtx 4090s (48gb total vram). measured time-to-first-token and tokens-per-second across context sizes from 4k to 36k. results: - 4k context: 127ms ttft, 42.3 tok/s - 12k context: 183ms ttft, 41.1 tok/s - 20k context: 264ms ttft, 38.7 tok/s - 28k context: 891ms ttft, 12.4 tok/s (massive jump here) - 36k context: 1240ms ttft, 8.1 tok/s the 28k boundary is where latency just falls off a cliff. tested with chunked prefill both enabled and disabled, same pattern. disabled chunked prefill is slightly better (782ms at 28k) but still way worse than 20k. is this vllm 0.6.3 specific or is it rope scaling breaking down past 28k?
we hit this in prod two days ago and it wasnt just latency jump it was a cliff.... past 28k context vllm 0.6.3 goes from 18 tok/s to 5 tok/s on llama 3.3 70b q4_k_m. disabling chunked prefil helped a bit but not much. ended up rolling back to vllm 0.6.1 and problem went away so its definately a 0.6.3 regressoin
we hit same cliff on dual 3090 setup.... past 28k it just falls off a cliff from like 22 tok/s to 6 tok/s. thought our hardware was dying lol
we hit same on 3090 last week. past 28k just dies from 19 tok/s to like 5
We saw this on 4090 too but it was worse - past 28k it dropped from 21 tok/s to 3 tok/s. Disabling chunked prefill helped but didn't fix it completely.... still investigating whether it's vllm or the weights themselves
We're seeing similar on A6000.... past 28k it drops from 24 tok/s to around 7 tok/s. Disabling chunked prefill helped a bit but didn't fully solve it.... wonder if this is related to rope scaling or just vllm memory management
disabling chunked prefill helped on our setup too but only partially. went from 19 tok/s to 8 tok/s past 28k instead of 4 tok/s. still investigating if it's rope scaling or vllm memory management
disabling chunked prefill helped on our setup too but only pushed the cliff from 28k to 31k. still drops from 19 tok/s to 7 tok/s, just happens later. might be rope scaling breaking down past a certain context size imo, could be wrong
we hit same cliff on our setup but disabeling chunked prefill didnt help at all, still drops from 18 tok/s to 6 tok/s past 29k. running dual 4090 so maybe its hardware limit not vllm?? honestly have no idea how to debug this, inference perf tuning is way over my head lol
Tested mcp sdk 0.5.0, 0.4.9, and 0.4.2 yesterday specifically for global timeout configuration. None of them support it. You have to manually wrap every single tool registration with timeout logic which is insane for production deployments with 40+ tools. We ended up writing a registration wrapper that adds default 30s timeout to all tools unless explicitly overridden. The wrapper checks if tool execution exceeds timeout and throws TimeoutError. But this should be built into the sdk - either as a global config option or at minimum a per-tool timeout field in the schema.
wait so there is no way to set global timeout for all tools in mcp sdk? you must wrap every single tool registration manually? this seem like very basic feature that every production sdk should have. we run ~40 different tools in our agent workflow and wrapping each one is not scalable)