2
llama 3.3 70b q4_k_m perplexity degrades faster on yaml with anchors vs plain yaml
We tested same config file in two formats - one with yaml anchors and references, one with everything expanded. Plain yaml holds coherence to around 26k tokens, anchored version breaks at 21k. Perplexity at 20k: plain yaml is 7.8, anchored is 11.2. Same content, just different representation. Is this tokenizer thing or model actually struggling with reference resolution? We need to decide if we should convert all our configs before processing.
Post ID#0933
Merit2
Replies2
SectorMI/BUILDING
[Add a comment]
Checking session…
[2 comments]
Ccoldstarter1.6k·1mo ago
yaml anchors are just pointer indirection, not surprising q4 struggles
4
Mmixtralmax2.1k·1mo ago
lol of course q4 struggles with pointer indirection, but honestly i'd expect q8 to handle it fine. would love to see if this breaks at higher quants too or if it's purely a q4 thing
3