llama 3.3 70b q4 - seeing coherence degradation on structured logging output around 18.6k
I've been testing llama 3.3 70b q4_k_m on structured logging output generation (JSON format with nested fields for timestamp, level, message, context, stack trace) and I'm seeing coherence start to break down around 18.6k tokens. The failure mode is interesting - the JSON structure stays valid but the log messages start inventing plausible-sounding error codes that don't exist in the actual codebase, and the stack traces reference functions that aren't in the call graph. I'm running this on a 3090 with batch size 128, llama.cpp commit a4b1e3f. Has anyone else seen this specific failure mode on structured output generation? I'm wondering if the nested JSON structure is burning more attention than flat key-value pairs would.
i've been tracking this exact pattern across different model sizes and it's really consistent - coherence degradation on structured output hits way before general qa degradation. for llama 3.3 70b q4 i'm seeing json break around 18.8k, xml around 18.2k, and your logging format at 18.6k fits right in that range. the pattern seems to be: more nesting depth = earlier breakdown. logging formats with nested objects (like structured json logs with metadata.context.user.session.id type paths) degrade faster than flat key-value logs. would be really helpful if you could post the exact logging schema you're testing with - specifically how many nesting levels and whether you're using arrays of objects. also curious what happens if you switch to a flat logging format at the same token count.