vllm 0.6.3 tensor parallel breaks on llama 3.3 70b past 28k context
running llama 3.3 70b q5_k_m on vllm 0.6.3 with tensor parallel 2 (2x4090, 48gb combined vram). works fine up to 28k context then outputs just degrade into repetition loops. 1. tested with tp=1 on single gpu (OOM past 18k) and it works clean up to the OOM point 2. tested llama 3.1 70b with same tp=2 config and it handles 32k+ fine 3. something specific to llama 3.3 + tensor parallel + long context anyone else seeing this or did i misconfigure something obvious
We hit this in production last week on vLLM 0.6.3 with Llama 3.3 70b q4_k_m, tensor parallel across 4x A100s. Past 28.3k context the model just starts generating garbage - not errors, just incoherent completions. Switched to pipeline parallel and it handles past 35k clean, but throughput dropped by ~40%. The issue is that tensor parallel splits attention heads across GPUs and past a certain context length the communication overhead between GPUs causes numerical instability. Pipeline parallel keeps each layer on one GPU so there's less cross-GPU traffic during attention. vLLM 0.6.4 supposedly fixes this but we haven't tested it yet.
hit this on vllm 0.6.2 last month with llama 3.1 70b past 26k context. tensor parallel just crashes with cuda oom even tho single-gpu inference handles 32k fine. which vllm version are you running and how many gpus in the tp group