llama 3.3 70b at q4_k_m randomly drops punctuation mid-sentence
running llama 3.3 70b at q4_k_m on a 4090 for summarization and maybe 15% of the time it drops periods or commas mid-sentence. same model at q5_k_m is perfect. the dropout seems random - sometimes it's fine for 500 tokens then suddenly "the model was trained on data from 2023 and includes" (just cuts off the period). iirc this is a quantization artifact in the final projection layer but i thought q4_k_m was supposed to preserve punctuation. anyone else seeing this or is it just my setup
we've been running llama 3.3 70b at q4_k_m for three days on legal doc summarization and seeing the EXACT same thing - randomly drops commas, periods, sometimes whole articles mid-sentence. happens maybe 5-10% of outputs, way more on longer context (>16k tokens). the pattern seems to be that punctuation gets dropped when the model is mid-reasoning about a complex clause, almost like the quantization is hitting the attention layers that track sentence boundaries. did you test whether this happens at q5 or is it specific to q4?
we're seeing the exaact same thing! llama 3.3 70b at q4_k_m drops commas and periods randomly maybe 15-20% of hte time on summarizaton tasks. happens more often when the context gets above 8k tokens. switched to q5_k_m and it completley stopped. my guess is quantizaton is breaking the punctuation heads in late layers