been testing 8 different quantization methods on mistral 7b for a week and honestly ready to give up
tried q4_0, q4_k_m, q4_k_s, q5_0, q5_k_m, q5_k_s, q6_k, q8_0 on the same mistral 7b checkpoint. tested on code completion (humaneval), reasoning (gpqa subset), and tool calling (40 examples) the results make no sense. q5_k_s beats q6_k on reasoning by 8%. q4_k_m beats q5_0 on tool calling. q8_0 is worse than q5_k_m on code completion i know quantization is lossy but this feels random. is there any actual theory about which quant methods preserve which capabilities or is it just trial and error forever
ok so first thign - what quantization methods did you actualy test and what were the numbers? also did you test on actual downstrema tasks or just perplexity becuase perplexity can be misleading as hell. we spent 2 weeks on mistral 7b quantization last year and the only thing that mattered was task-specific eval, not the quant benchmarks
tested q4_0, q4_k_s, q4_k_m, q5_0, q5_k_s, q5_k_m, q6_k, q8_0 on mistral 7b for code completion. q4_k_m was the sweet spot - barely worse than q5 but way less vram. q4_0 was garbage, q8_0 was pointless. what tasks are you testing on?
ok so same here, q4_k_m is the sweet spot for mistral 7b. q4_0 is unusable garbage, q8_0 barely better than q6 but way more vram. we settled on q4_k_m for prod and haven't looked back
q4_k_m or bust. q4_0 is garbage
ok so which 8 methods and what were the actual results. also are you testing on the same hardware for all 8 or is this apples to oranges
also what tasks you testing on? perplexity vs actual downstream makes huge difference for quant
which quantization methods did you test.... also what's your eval setup? we spent two weeks on this exact problem last month and ended up just using q5_k_m