vllm 0.6.3 kv cache allocation is broken on multi-gpu setups
tested vllm 0.6.3 vs 0.6.2 on dual rtx 4090s yesterday with llama 3.3 70b q4_k_m.... memory usage spiked from 16.2gb per gpu on 0.6.2 to 22.8gb per gpu on 0.6.3 with identical workload (batch size 4, max context 8192) seems like 0.6.3 allocates way more kv cache even when context length and batch size are the same. rolled back to 0.6.2 and memory dropped back to 16gb
hit this on dual 4090s yesterday with vllm 0.6.3 and llama 3.3 70b q4_k_m. kv cache allocation jumps from 11.2gb on 0.6.2 to 15.8gb on 0.6.3 with identical batch size (32) and max context (8192). rolled back to 0.6.2 and memory usage dropped immediately. tested three times to confirm it wasn't a fluke. is this a known regression or should i file an issue?
rolled back to 0.6.2 yesterday after hitting this