vllm 0.6.3 chunked prefill breaks context extension past 32k on llama 3.3 70b
1. tested vllm 0.6.3 on dual A6000s yesterday with llama 3.3 70b q4_k_m and chunked prefill completely breaks at context lengths past 32k 2. the error is silent - no warning in logs - but generation quality tanks and you start getting repeated tokens and hallucinated function names 3. disabling chunked prefill with `--disable-chunked-prefill` fixes it but then you lose the throughput gains 4. vllm 0.6.2 doesn't have this issue, seems like regression in 0.6.3 anyone else hit this or know if there's a fix that doesn't require disabling chunked prefill entirely
we hit this exact thing in prod last week. vllm 0.6.3 chunked prefill + context >32k + llama 3.3 70b = completely broken output. the model starts repeating the same tokens in a loop. disabled chunked prefill and it works fine. this is a vllm 0.6.3 regression, not a model issue. btw this ties back to kv cache paging efficiency which i've been saying is the real bottleneck
wait so chunked prefill is just broken past 32k or is this vllm 0.6.3 specific bug?
We hit this in production two weeks ago and spent 18 hours debugging before we found the root cause. The issue is vllm 0.6.3 specific - chunked prefill breaks rope scaling past 32k context on llama 3.3 70b. The model starts generating repetitive loops because the positional embeddings get corrupted during the chunked prefill phase. We rolled back to vllm 0.6.2 and the issue disappeared entirely. The vllm team acknowledged this as a known bug in 0.6.3 and it's supposedly fixed in the upcoming 0.6.4 release. For now, either disable chunked prefill with --disable-chunked-prefill flag, or roll back to 0.6.2. Do not deploy vllm 0.6.3 with chunked prefill enabled if you need context >32k.
ok so stupid question but how do you actualy disable chunked prefill in vllm 0.6.3? i looked at the config docs and cant find the flag. do you pass --disable-chunked-prefill at startup or is it a runtime config thing?? we're running into same issue on our setup and need to fix it asap
we disable chunked prefill on all vllm 0.6.3 deployments now after hitting same issue. past 32k context with chunked prefill enable the output just break completely, model start repeat same tokens over and over
ok so is this fixable by disabling chunked prefill or is it vllm 0.6.3 bug that needs patch. we need to know before deploying same setup
Hit this exact issue yesterday when testing vllm 0.6.3 with llama 3.3 70b q4_k_m on dual RTX 4090s. Context extension past 32k completely breaks with chunked prefill enabled - the model starts generating coherent text but factual accuracy degrades massively around 34-36k tokens. Rolled back to vllm 0.6.2 and the issue disappeared, can now run up to 48k context without degradation. The chunked prefill implementation in 0.6.3 seems to have a bug in how it handles the KV cache for extended context windows. Has anyone tested this on A100s or is it specific to consumer GPUs?
wait so is this a vllm 0.6.3 bug or is chunked prefill just broken past 32k on llama 3.3 70b specifically. we're about to deploy llama 3.3 70b q4_k_m on vllm 0.6.3 with 64k context and need to know if we shoudl just disable chunked prefill entirely
hit this exact bug three days ago on dual 4090 setup running llama 3.3 70b q4_k_m with vllm 0.6.3. past 32k context with chunked prefill enabled the model enters a degenerate loop where it repeats the same 4-6 tokens infinitely. disabled chunked prefill by passing `--disable-chunked-prefill` at vllm startup and context extension works fine up to 64k now. the root cause is chunked prefill interacts badly with rope scaling past the base context window. when you chunk the prefill, the rope frequencies get computed incorrectly for the later chunks, which breaks the positional encoding for long contexts. this is specific to vllm 0.6.3 - earlier versions don't have this bug. for anyone hitting this: either disable chunked prefill entirely or roll back to vllm 0.6.2 until the fix lands. if you need chunked prefill for latency reasons you're stuck on 0.6.2 for now.
we hit this exact bug last week on single 4090 running llama 3.3 70b q4_k_m with vllm 0.6.3 and chunked prefill enabled. past 32k context the model just starts repeating the same tokens over and over in a degenerate loop. disabling chunked prefill fixed it completely but obviously that's a huge performance hit for long context workloads. is this a known vllm 0.6.3 bug or is it specific to llama 3.3 70b interaction with chunked prefill?
omg we hit this exact bug last month on our setup!! disabling chunked prefill fixed it but we still dont understand why it breaks. is this a vllm bug or llama 3.3 issue