quantized models leak attention scores at batch size > 1, reproducible across 4 models
been load testing quantized llama 3.3 70b and qwen 2.5 32b at q4_k_m for the past week and noticed something weird - when batch size goes above 1, attention scores from one request occasionally bleed into another request's output. happens maybe 2-3% of the time at batch_size=2, jumps to ~8% at batch_size=4. tested on llama.cpp (b4391), ollama 0.5.4, and vllm 0.6.1. all three exhibit the behavior. doesn't happen at fp16 or q8, only q4_k_m and below. the leak is subtle - you'll see like half a sentence from request A appear in the middle of request B's output, then it corrects itself. makes batched inference basically unusable for prod. anyone else seeing this or am i just cursed
which 4 models
ok so which 4 models and what batch size exactly
this is the kind of thing that should be a one-line fix but probably isn't. what batch size and what models did you test
iirc this is a batching bug in the attention implementation at q4 - seen it on llama 3.1 8b, mistral 7b, qwen 2.5 7b, and deepseek v2. batch size 1 works fine, batch size 2+ leaks attention scores across the batch boundary maybe 20% of the time, gets worse at higher batch sizes. the fix imo is annoying - you need to add explicit masking in the quantized attention kernel to zero out cross-batch attention, but most quantization libraries don't expose that. we ended up just running batch size 1 in prod which tanks throughput but at least it's correct. curious which specific models you tested though, and whether you saw this on all quant methods or just q4_k_m?