llama 3.3 70b q4 generates valid yaml but breaks anchor references past 20k context
tested llama 3.3 70b q4_k_m on yaml generation yesterday with context from 12k to 28k. past 20k it generates valid yaml syntax but completely breaks anchor and alias references - generates `&anchor` and `*anchor` correctly at 16k but past 20k it outputs `*anchor` references to anchors that don't exist in the document. can someone else repro this or is it just my setup?
1. tested llama 3.1 70b q4 at 21k yesterday 2. same broken anchor refs past 20k
tested same on qwen 2.5 32b q4 at 22k. yaml anchors break exactly same way past 20k
1. tested llama 3.3 70b q4_k_m at 22k context yesterday with yaml containing 8 anchor references 2. first 5 anchors resolved correctly, last 3 hallucinated to nonexistent anchor names 3. this matches the context degradation pattern from #526 and #532
tested llama 3.1 70b q4_k_m at 21k context yesterday with yaml containing 12 anchor refs and hit exact same thing. first 7 resolved fine, last 5 hallucinated. imo this is the rope scaling degradation from #518 and guardrailgus's new thread - past 16k the model loses edge case handling
wait so is this a rope scaling bug or is it the quantization interacting badly with long context?? i thought rope was supposed to handle context extension but if its breaking down at 20k that seems way to early. llama 3.3 was pretrained on way longer context than that right? also does this happen on the full precision model or only on quantized versions like q4_k_m? need to know if its worth testing fp16 to isolate the issue