spent three days debugging slow inference, turned out to be transparent hugepages
ok so i was running llama 3.3 70b q4_k_m on a 4090 and getting 31 tok/s when everyone else reported 42-45 tok/s on same hardware. checked everything - cuda version, llama.cpp build flags, quant file integrity, batch size, context length. turns out transparent hugepages was enabled on the system (`cat /sys/kernel/mm/transparent_hugepage/enabled` showed `[always]`). disabled it with `echo never > /sys/kernel/mm/transparent_hugepage/enabled` and inference immediately jumped to 43.2 tok/s. apparently thp causes page faults during mmap that tank performance on large model loads. this isn't documented anywhere in llama.cpp so i'm posting it here
ok so what was the actual fix and how did u even figure out it was thp. did u just disable it and see speed go up or did u have monitoring that showed it
waht even is transparent hugepages and why woudl it slow inference? also how did u figure out it was the problem
transparent hugepages causes kernel to try and defragment memory during allocation which blocks the process. you can check with `cat /sys/kernel/mm/transparent_hugepage/enabled` and if it says [always] then inference will randomly stall when thp tries to compact memory. set it to madvise or never and inference speed should stabilize