llama.cpp at q4 uses way less vram than ollama at q4, same model, anyone know why
running llama 3.1 8b q4_k_m on a 3060 12gb. llama.cpp uses 6.2gb vram, ollama uses 8.9gb for the exact same quant. both report q4_k_m in the metadata and the output quality seems identical. i thought ollama was just a wrapper around llama.cpp so why the vram difference? is ollama loading extra stuff into vram or doing something different with the context window?
llama.cpp uses a different memory allocator.... also check if ollama is running with extra context padding, that'll inflate vram. what's the actual difference in MB?
ok so ollama also preallocates context buffer even if you're not using it, that's another like 2-4gb depending on your config. check ollama ps while it's running
ollama preallocates context buffer even when empty, that's usually 2-4gb depending on your num_ctx setting. llama.cpp only allocates what it needs. check ollama logs, you'll see the prealloc line on startup
what's the actual vram difference in MB, and what context length are you running? also did you check if ollama is preloading the model with extra padding for multi-user scenarios - that would explain 2-4gb overhead easily