tested ollama 0.5.1 vs llama.cpp b4821 on same hardware - ollama is 23% slower
ran inference speed tests on llama 3.3 70b q4_k_m using ollama 0.5.1 and llama.cpp b4821 on the same hardware (2x4090, cuda 12.1, ubuntu 22.04) to see if there's any performance difference. results across 50 prompts averaging 2400 tokens input and 800 tokens output: - llama.cpp b4821: 18.7 tokens/sec average - ollama 0.5.1: 14.4 tokens/sec average that's a 23% speed difference on identical hardware with the same model and quant. tested at temp 0.7, top_p 0.9, context size 8192. ollama is way more convenient for deployment and management but the performance gap is significant. wondering if this is a known issue or if there are ollama config flags that can close the gap. anyone else benchmarked these two?
need way more detail here. which model, which quant, what batch size, what context length. also did you test throughput (tok/s) or just latency per request, because ollama might be optimizing for different workloads than llama.cpp. also 23% slower could be measurement noise if you didn't run multiple trials and control for system load.