llama.cpp b4821 crashes on context >48k with mmap enabled on linux 6.1 kernel
tested llama.cpp b4821 with llama 3.3 70b q4_k_m on linux 6.1 kernel yesterday and it crashes hard at context lengths >48k when mmap is enabled. segfault in the memory mapping code, no useful error message. disabling mmap fixes it but tanks performance from 43 tok/s to 28 tok/s. tested on kernel 6.5 and it works fine at 64k context with mmap enabled. seems like older kernel versions have some bug in the mmap implementation that llama.cpp triggers at high context lengths.
hit this on b4821 + ubuntu 22.04 with 6.1 kernel yesterday. crashes at 52k context with mmap enabled, works fine with --no-mmap flag. error looks like memory mapping failure in the logs
We hit this exact issue on our gpu cluster last month running llama.cpp b4805 on ubuntu 22.04 with kernel 6.1.0-27. The crash happens consistently at 51.2k context when mmap is enabled. The error in dmesg shows memory mapping failure with errno 12 (cannot allocate memory). The workaround is --no-mmap but that tanks performance - we went from 38.7 tok/s to 22.3 tok/s on our setup. Opened an issue on the llama.cpp repo but haven't seen movement yet. The problem seems to be related to how the memory mapping interacts with the kernel's vm.max_map_count limit on 6.1 kernels specifically.