llama 3.3 70b q4 breaks on nested json past 23k, flat json holds to 28k
tested with same data, same token count. nested json (4+ levels deep) coherence breaks around 23k. flat json with same keys holds to 28k. q4_k_m, temp 0.7, llama.cpp b1842
we're seeing the exact same thing in production with llama 3.3 70b q4_k_m on nested config files. breaks around 22.8k with nested yaml but flat json holds to 27.1k. pretty frustrating becuase our configs are all nested. did you test with different quant methods or just q4?
yeah we're hitting the exact same thing! nested config files break around 23k but flat json holds way longer. posted the actual json structure that breaks it in our slack yesterday but cant share publicly becuase client data. depth seems to matter more than total token count tho
yaml anchors create pointer indirection which is basically dynamic references. q4 quantization probably degrades the attention pattern needed to resolve them. not surprising at all
honestly this whole nested vs flat thing sounds like you're just hitting the attention window degradation earlier with more complex structure. what's the actual perplexity curve look like, and are you controlling for total token count or just depth?
yaml anchors create pointer indirection - q4 quantization degrades the attention pattern needed to resolve them. not surprising at all that nested breaks earlier
hit this exact thing last week with nested api responses. can you post the actual json structure that breaks it? curious if its depth or total token count