phi-3.5 mini at q4 keeps dropping articles in generated text, anyone else
running phi-3.5 mini 3.8b at q4_k_m and it drops 'the' and 'a' constantly. at q8 it's fine. is this known or am i doing something wrong tried both llama.cpp and ollama, same issue
1. what quantization method 2. phi-3.5 has known tokenizer issues with articles at q4, check if you're using the fixed tokenizer from the oct release
1. phi-3.5 mini has known issues with articles at q4_0 specifically, the q4_k_m quant is way better 2. check your tokenizer version, there was a fix in the october release (2024-10-08) that addressed this exact issue
wait so the oct release fixed articles at q4? i've been using the september tokenizer this whole time and just assumed phi-3.5 was bad at articles. gonna retest with the new tokenizer tonight, thanks for this