vllm 0.6.3 memory leak on llama 3.3 70b past 35k context
we've been running llama 3.3 70b q4_k_m on vllm 0.6.3 in prod for two weeks and theres a slow memory leak past 35k context. starts at 41gb vram usage, after 6 hours of requests averaging 38k context it creeps up to 46.2gb then oom crashes. happens on both a100 80gb and dual 4090 setups. rolled back to vllm 0.6.1 yesterday and leak is gone, rock solid at 41.1gb for 18 hours straight. pretty sure its related to the chunked prefill changes in 0.6.3 bc disabling chunked prefill on 0.6.3 also stops the leak. anyone else hitting this or is it just our weird setup??
need exact vram numbers before and after. if this is a vllm bug vs hardware issue that changes our whole deployment strategy
ok so is this vllm chunked prefill or is it rope scaling breaking past 28k? need to know if we should disable chunked prefill or switch inference backends entirely
what hardware r u running this on
need exact vram numbers and hardware spec. if this is a vllm bug vs model issue that changes everything for our prod setup
wait is this just vllm or is it the model weights themselves.... tested llama 3.3 70b q4 on ollama 0.5.2 at 36k context yesterday and memory usage climbed from 41gb to 68gb over like 15 minutes of generation. killed the process before it ate all my ram
what exact vram usage did you see before vs after the leak