does vllm 0.6.3 paged attention actually work on consumer gpus or is it broken
trying to run llama 3.3 70b on dual 4090s with vllm 0.6.3 and the paged attention seems completely broken. memory usage is way higher than 0.6.2 and i'm getting oom errors at batch sizes that worked fine before. is this a known issue or am i configuring something wrong
We hit this on dual RTX 4090s with vllm 0.6.3 and llama 3.3 70b q4_k_m yesterday. Paged attention works fine at batch size 8 but starts throwing CUDA OOM errors at batch size 32. Rolled back to 0.6.2 and same config works perfectly. Something broke in the memory allocator between versions.
tested vllm 0.6.3 on dual 3090s yesterday with llama 3.3 70b q4 and paged attention worked fine at batch size 16. what gpu setup are you running and what error do you get
i tested vllm 0.6.3 paged attention on dual rtx 3090s yesterday with llama 3.1 70b q4 and it worked fine.... got 42.3 tok/s at batch size 16. what hardware are you running on and what errors are you seeing?
tested vllm 0.6.3 on single rtx 4090 yesterday with llama 3.1 8b q4 and paged attention work fine up to batch size 24, but at batch size 32 it throw cuda oom error even tho vram usage is only at 18.2gb out of 24gb. somethign is broken with memory allocation logic i think