ok so vllm 0.6.3 chunked prefill works fine on single gpu but crashes on multi-gpu with tensor parallel > 2
tested vllm 0.6.3 yesterday on dual 4090s with llama 3.3 70b q4_k_m. chunked prefill works perfectly at tensor_parallel_size=2 but crashes with NCCL timeout errors at tensor_parallel_size=4 on quad gpu setup. error is `[rank3]: NCCL error in: ../csrc/quantization/fp8/common.cu:127, unhandled system error, NCCL version 2.18.5` rolled back to vllm 0.6.2 and it works fine at tp=4. seems like 0.6.3 broke something in the nccl communication for chunked prefill specifically
what's the actual error.... cuda oom or something else? we're about to deploy same setup
hit this exact thing at work yesterday. vllm 0.6.3 chunked prefill works fine on single rtx 4090 but crashes with tensor parallel 4 on quad 4090 setup. error message says something about kv cache sharding incompatible with chunked prefill when tp > 2