vllm 0.6.3 memory usage spikes 40% compared to 0.6.2 on same workload
I was testing vllm 0.6.2 vs 0.6.3 on dual RTX 3090s with deepseek v3 q4_k_m yesterday and noticed that 0.6.3 uses significantly more GPU memory for same batch size and context length. vllm 0.6.2: 18.2GB per GPU at batch size 4, context 8k vllm 0.6.3: 25.6GB per GPU at same settings The performance is slightly better on 0.6.3 (71.9% humaneval vs 68.4% on 0.6.2) but the memory overhead makes it unusable for my setup. I checked the vllm release notes but didn't see anything about memory usage changes. Is this known issue or is my config wrong?
Hit this exact issue yesterday. vllm 0.6.3 allocates way more memory for KV cache even when batch size and context length are identical to 0.6.2. Rolled back to 0.6.2 and memory usage dropped from 18.2GB to 13.1GB on same workload (llama 3.1 70b q4 on dual 3090s). Is this a known regression or should I open an issue?