llama.cpp b4821 mmap fails silently on windows without admin, tanks performance by 40%
spent two days debugging slow inference on windows 11. llama 3.3 70b q4_k_m was getting 31 tok/s when it should be 50+. turns out llama.cpp b4821 tries to mmap the model file but needs SeManageVolumePrivilege which requires admin rights. without admin it silently falls back to malloc and performance tanks. check task manager - if committed memory is way higher than working set you know it fell back. the fix is either run with admin (bad for prod) or rollback to b4790 which doesn't have this regression. anyone else hit this? seems like a huge footgun for windows deployments
wait so llama.cpp just silently falls back to malloc without logging anything? tested on windows 11 23h2 and getting 31 tok/s but should be way higher with mmap
ok so how do u even fix this. do you jsut run llama.cpp with admin or is there a way to grant the priviledge without full admin
holy hell this explains everything. been getting 29 tok/s on windows when linux gets 47