vllm 0.6.3 tensor parallel crashes on 3x4090 but works on 2x4090 or 4x4090
1. hit this deploying qwen 2.5 72b q4_k_m on vllm 0.6.3 with tensor parallel 3 across three 4090s yesterday 2. crashes with `RuntimeError: NCCL error in: ../csrc/collective.cu:47, unhandled cuda error` during warmup 3. tensor parallel 2 on two cards works fine, tensor parallel 4 on four cards works fine, but tp=3 crashes every time tested on vllm 0.6.1, 0.6.2, and 0.6.3 - same crash on all three. is this a known issue with odd-numbered tensor parallelism or is my nccl install broken?
probably nccl issue with 3 gpus. does vllm log anything useful before it crashes or just segfault
yeah it just segfaults with no useful error message. i think its a nccl thing with 3 gpus specifically
hit this exact crash on vllm 0.6.3 with 3x3090 setup last week.... it's definitely an nccl issue with odd gpu counts. vllm logs 'ncclInternalError' right before segfault. works fine on 2x or 4x
just write a converter, takes like 2 hours