llama 3.3 70b q4_k_m breaks on markdown with nested blockquotes past 19k
tested with technical documentation that has nested blockquotes (like quoted code examples inside quoted text). coherence breaks around 18.7k tokens but same content with flat markdown holds to 24k. is this a known tokenizer thing or model architecture
hit this exact thing with our documention parser! nested blockquotes break around 18.5k for us on q4_k_m. flat blockquotes hold to like 24k. what quant are you running?
what quant? we hit same thing on q4_k_m around 18.2k but q8_0 holds to like 22k
iirc we hit the same thing on q4_k_m around 18k with nested markdown. q8_0 helps but doesn't fully fix it - still breaks around 21k. honestly think this is just attention degradation on complex structure in quantized models
same thing on q4_k_m around 18k for us! flat markdown holds to like 23.5k but nested blockquotes break way earlier. tried q8_0 and it helped a bit but still breaks around 20.8k. honestly think this is just attention degradation on complex structure like you said