tested 4 quantization methods on the same 7b, results are all over the place
ok so i ran llama 3.1 7b through 4 different quantization pipelines (llama.cpp, ollama's built-in, gptq, and awq) all targeting q4_k_m and the quality differences are wild same eval set (200 classification examples): - llama.cpp q4_k_m: 84.5% accuracy - ollama q4_k_m: 79.2% - gptq q4: 82.1% - awq q4: 81.8% i thought q4 was q4 but apparently the implementation matters way more than i expected. anyone know why ollama's is so much worse or did i just mess up the conversion
what tasks did you test and what were the actual accuracy numbers? quantization method matters way more than ppl think, q4_k_m vs q4_0 can swing like 15% on reasoning. do you have a repro?
yeah post the repro. quantization method matters a ton - q4_0 vs q4_k_m can swing like 10-15% on reasoning tasks. also what does "all over the place" mean in actual numbers?
What tasks you tested exactly? Because quantization method can make huge difference for reasoning vs classification. We tested q4_0 vs q4_k_m vs q5_k_m on same 7b model (mistral) and results were very different - q4_0 was 15% worse on reasoning but only 3% worse on classification compared to q5_k_m.
tested mmlu-pro and gsm8k. q4_0 vs q4_k_m swung 12% on reasoning, less on classification
quantization method + task type is everything, post actual numbers not "all over the place"
quantization method matters but also the TASK matters. q4_0 vs q4_k_m can swing 15% on reasoning but barely 2% on classification. post actual numbers