gemma 2 27b at q4 has worse perplexity than the 9b at q5, tested on 40k tokens
ran perplexity eval on wikitext (40k token sample) for gemma 2 27b q4_k_m vs gemma 2 9b q5_k_m. 27b scored 8.4, 9b scored 7.1. same pattern on arxiv abstracts - 27b at 9.8, 9b at 8.3. vram: 27b uses 18gb, 9b uses 7gb. inference speed basically identical on a 3090 (both around 28 tok/s). so the bigger model quantized harder is actually worse than the smaller model quantized less. which makes sense if you think about it but nobody mentions this when recommending "just use the biggest model you can fit" tested q5_k_m on the 27b and perplexity dropped to 6.2, way better than both, but now it needs 23gb and tok/s drops to 19. basically: model size vs quantization level is not obvious and you should actually measure
did you mesure perplexity on the same splot or diferent datasets? also whats the actuall numbers imo this could just be dataset artifact
ok so we measured on the same dataset (wikitext-103, 40k token sample) and got gemma 2 27b at q4_k_m perplexity of 8.9 vs gemma 2 9b at q5_k_m perplexity of 7.2. the issue is definitely quantization hitting the attention layers asymmetrically - larger model has more layers so more cumulative quantization error
hit this exact thing last week with gemma 2 9b at q4_k_m - perplexity was worse than the smaller model at q5. the issue turned out to be how the quantization was hitting the attention layers in the 27b, like layers 24-28 were getting absolutely destroyed. switched to q5_k_s and it fixed it. what layers are you seeing the degradation in?
did you test perplexity on the same dataset for both models or different splits? also what's the actual perplexity numbers
did you measure perplexity on the same dataset for both or different splits? also curious if this generalizes - we tested gemma 2 9b at q4_k_m vs the 2b at q8 and saw similar inversion on code completion tasks but not on classification