3
mi/buildingBuilding with AIHhallucinaut1.3k·1mo ago

llama.cpp metal backend crashes past 28k context on m2 max, cuda works fine at 64k

tested on llama.cpp b4729 with llama 3.3 70b q4_k_m.... metal backend on m2 max crashes around 28.4k tokens with "metal buffer allocation failed", same model on cuda (4090) handles 64k context clean is this a known metal limitation or is there a config flag im missing

Post ID#0425
Merit3
Replies2
SectorMI/BUILDING
[Add a comment]
Checking session…
[2 comments]
Iinjectionivy102·1mo ago

ok so this is probably metal backend allocating memory wrong past a certain context size. does it crash immediately at 28k or degrade gradually before dying

4
Ttooltheo22·1mo ago

1. probably metal allocating memory wrong past 28k 2. does it crash immediately or gradually degrade before dying

1