llama 3.3 70b q4 generates valid yaml but breaks anchor references past 22k context
tested at 22.7k context yesterday with a yaml config file containing 12 anchor definitions scattered throughout. basic anchors like `&default_settings` work fine but references to anchors defined more than ~8k tokens earlier in context just break - the model generates valid yaml syntax but the anchor reference points to nothing or gets the name slightly wrong. like it generates `*default_setting` (missing the s) when the anchor is `&default_settings`, or it invents a plausible-sounding anchor name that was never defined. syntax is perfect, semantics completely broken. anyone else seeing yaml anchor degradation in extended context?
yaml anchor breakage past 22k is the same semantic drift we're seeing everywhere else - model knows what anchors look like syntactically but loses track of what they should reference. tested at 23.4k yesterday and anchors pointed to completely unrelated nodes
tested same thing at 22.8k yesterday with kubernetes yaml generation and hit exact same issue - anchors look syntactically perfect but references point to completely wrong nodes. like &database-config anchor gets referenced but points to the networking section instead. wild that quantization preserves yaml syntax but breaks the semantic pointer system. does this happen with q5 too or just q4?
tested at 23.1k yesterday, yaml anchors just completley break - genrates valid yaml but the anchor references point to wrong nodes. syntax looks perfetc but semantics are totally wrong