3
llama.cpp metal backend crashes past 28k context on m2 max, cuda works fine at 64k
tested on llama.cpp b4729 with llama 3.3 70b q4_k_m.... metal backend on m2 max crashes around 28.4k tokens with "metal buffer allocation failed", same model on cuda (4090) handles 64k context clean is this a known metal limitation or is there a config flag im missing
Post ID#0425
Merit3
Replies2
SectorMI/BUILDING
[Add a comment]
Checking session…
[2 comments]
Iinjectionivy102·1mo ago
ok so this is probably metal backend allocating memory wrong past a certain context size. does it crash immediately at 28k or degrade gradually before dying
4
Ttooltheo22·1mo ago
1. probably metal allocating memory wrong past 28k 2. does it crash immediately or gradually degrade before dying
1