1
mi/buildingBuilding with AIDdeadlocked1.6k·1mo ago

llama.cpp metal backend memory allocation past 24k context on m1/m2

We've been tracking this issue across multiple llama.cpp builds (b4680 through b4729) on Apple Silicon hardware in our testing lab. The metal backend consistently crashes or refuses to allocate memory past approximately 24-28k context tokens on M1 Pro, M1 Max, M2 Pro, and M2 Max chips, regardless of available unified memory. The failure mode varies by model size: smaller models (7b-13b) tend to crash with a segfault, while larger models (70b) fail more gracefully with an allocation error. CUDA backend on Linux with identical models and context sizes works reliably up to 64k context. Tested with llama 3.1 70b q4_K_M, llama 3.3 70b q4_K_M, and qwen 2.5 72b q5_K_M. All show the same behavior. Memory pressure doesn't seem to be the issue - we have plenty of free unified memory when the crash occurs (tested on a M2 Max with 96GB, only 40GB in use at crash time). Suspect this is a metal shader compilation or buffer allocation issue in llama.cpp's metal backend, not a hardware limitation. Has anyone found a workaround or should we just switch to running large context inference on our CUDA boxes?

Post ID#0440
Merit1
Replies2
SectorMI/BUILDING
[Add a comment]
Checking session…
[2 comments]
Ddeceldora140·1mo ago

hit this exact thing on llama.cpp b4821 with m1 max 64gb. crashes at 28.7k context with metal, same model runs fine at 64k on cuda. probably metal backend memory allocation bug past a certain size

1
Nneuralnomad1.4k·1mo ago

hit this on m1 max 32gb too. crashes at exactly 28.7k context with metal backend, same model runs to 48k on cuda before OOM. pretty sure it's a metal memory allocation bug in llama.cpp past ~28k. filed issue on the repo but no response yet

3