llama 3.3 70b q4_k_m - tested inference speed on mac studio m2 ultra vs linux 4090, context processing 2.3x faster on linux
tested llama 3.3 70b q4_k_m on mac studio m2 ultra (192GB unified) vs linux box with 4090 (24GB vram + 128GB system ram). context processing at 16k tokens: mac studio 47 tok/s, linux 4090 108 tok/s. generation after context load: mac studio 18 tok/s, linux 23 tok/s. mac studio has way more memory bandwidth on paper but context processing is 2.3x slower. guessing metal backend has overhead that cuda doesn't. llama.cpp b2701 on both machines.
2.3x is a big gap for context processing. Is this just memory bandwidth (Mac Studio uses unified memory vs 4090 GDDR6X) or is there something else going on? Would be interesting to see if prompt eval scales linearly with context size on both platforms or if the gap widens at longer contexts. Did you measure token generation speed (decode) as well, or just context processing (encode)?
memory bandwidth is definitely part of it but also llama.cpp optimizations for apple silicon vs cuda are completely different codepaths. need exact llama.cpp commit, batch size, and whether you're using metal vs cpu on the mac
2.3x is huge gap. memory bandwidth explains some but not all of this. at my last job we saw similar gaps between m1 ultra and 3090 on llama 2, turned out llama.cpp metal optimizations were way behind cuda path at that commit. which llama.cpp version
Memory bandwidth difference between unified memory and GDDR6X could explain some of this, but 2.3x is very large gap. Also Apple Silicon has different Metal optimizations versus CUDA path. Which llama.cpp commit are you using?