quantized mistral 7b at q4 can't count past 8 consistently
Been testing a quantized mistral 7b (q4_k_m via llama.cpp) on basic counting tasks and it falls apart after 8. Ask it to count to 10 and it either skips 9, repeats 8, or just outputs garbage tokens. Tried the same model at q8 and it's fine. This feels like a quantization artifact in whatever layer handles sequence generation but I can't find anyone else reporting it. Tested on 40 prompts, fails 32/40 times at q4, 0/40 at q8. Is this a known issue with mistral specifically or am I doing something wrong with the quantization settings?
lol ya mistral 7b cant count at q4, hits 8 then hallucinates. we tested this last month on a data extraction task and it was useles for anything with numbers
can you post a repro? counting past 8 sounds like a tokenizer artifact but would need to see the exact prompts and outputs to debug
counting breaks at q4 on basically every small model, not just mistral. the tokenizer encodes numbers weird and quantization amplifies it. try q5_k_m or just don't use 7b for anything with numbers
ok so we hit this exact problem with mistral 7b at q4_0 - it could count to 6 consistently, then 7-9 was like 50/50, anything above 10 was pure hallucination. switched to q5_k_m and it fixed it completely. the issue is the tokenizer encodes multi-digit numbers as multiple tokens and quantization error accumulates across the sequence
lol we gave up on mistral for anythign with numbers after it halucinated prices in a production invoice parser. switched to qwen 2.5 7b at q5 and it works perfeclty
lol same, switched to qwen for anything with numbers
ok so same here, mistral 7b at q4 cant count past 7 consistently and we gave up and switche dto qwen 2.5 7b at q5. the annoying part is this wasnt documented anywhere so we wasted like 3 days debugging our prompt before we realized it was the model
switched to qwen 2.5 7b at q5 for anything with counting or math, haven't looked back. mistral at q4 is just broken for numerical reasoning. do you see the same counting failures at q5 or does it actually fix it?
we tested q5 and it works perfectly.... the counting thing seems specific to q4, maybe the quantization breaks whatever numerical reasoning circuit is in there?