llama.cpp b4821 inference speed drops 30% when switching from cuda 12.1 to 12.4 on 3090
tested llama.cpp b4821 with llama 3.1 70b q4_k_m on a 3090 ti.... with cuda 12.1 i was getting 38.2 tok/s at 8k context, upgraded to cuda 12.4 and now getting 26.7 tok/s on same model same context same everything rolled back to 12.1 and speed went back up to 37.9 tok/s. anyone else seeing this or is my setup cursed
ok so cuda 12.4 has known regression with flash attention on 30-series cards. saw this exact thing on 3090 with llama.cpp b4821 - dropped from 52 tok/s to 38 tok/s. rolled back to cuda 12.1 and it fixed itself
ok so cuda 12.4 broke flash attention on ampere cards, this is documented in the llama.cpp issues. everyone running 30-series should stay on 12.1 until nvidia fixes it (which they won't because they want you to buy 40-series)
yeah this is documented in like 8 different github issues and nvidia keeps closing them as wontfix 😤 staying on cuda 12.1 forever i guess
I saw similar thing when testing cuda 12.4 vs 12.1 on RTX 3080 with llama.cpp b4790. The inference speed dropped about 25% on cuda 12.4. When I checked nvidia forums there were several threads about flash attention regression on Ampere cards in cuda 12.4, but nvidia marked most of them as 'working as intended' or closed without fix. Very frustrating for anyone who wants to use recent cuda versions