llama.cpp b4729 memory usage spikes to 94gb on llama 3.3 70b q4_k_m past 52k context
Running llama.cpp b4729 on llama 3.3 70b q4_k_m. Memory usage is stable around 41-43gb up to 52k context, then suddenly spikes to 94gb and crashes the process with OOM. Tested on system with 128gb ram, reproducible across 4 runs. vllm 0.6.3 handles same context range at 67gb peak. Is this a known issue or configuration problem on my end? Context window is supposed to be 128k but this makes anything past 50k unusable on llama.cpp.
getting same memory spike on llama.cpp b4680 around 48k context. are you using mmap or loading the whole model into ram?
ok so this is probably kv cache growth past 50k context. the cache size scales linearly with context length so at 52k you're storing way more activations than at 30k. are you using mmap or loading into ram
Hit this exact memory spike on llama 3.3 70b q4_k_m around 51.8k context using llama.cpp b4729. Memory usage climbs gradually from 42GB at 30k context to 94GB at 52k, then the process just crashes with OOM on a machine with 96GB RAM. Tested on three different prompts and the spike happens at roughly the same context length every time, which suggests it's not input-dependent but rather some internal buffer allocation issue in llama.cpp's KV cache handling past 50k. Switching to vllm 0.6.3 kept memory usage stable at around 68GB even past 60k context, so this is definitely a llama.cpp implementation issue not the model itself.