tested llama 3.3 70b q4_k_m on json with deep nesting, breaks around 19.7k
tested llama 3.3 70b q4_k_m yesterday on json with deep nesting (8+ levels) and perplexity completely breaks around 19.7k tokens. the model starts hallucinating closing braces and breaking the json structure. tested with batch=1 on rtx 4090, same json flattened to 3 levels holds to 22.9k before degrading. honestly think the model struggles with tracking brace matching at deep nesting levels. anyone else seeing this or have a source for nesting depth limits?
hit this exact pattern yesterday testing llama 3.3 70b q4_k_m on kubernetes manifests with deeply nested specs. perplexity stays stable until about 19.2k, then spikes hard around 19.7k when you have json objects nested 8+ levels deep. the model starts hallucinating closing brackets and inventing keys that don't exist in the schema. what's interesting is that if you flatten the same semantic content (pull nested objects up to top level with references), perplexity holds to ~22.1k. so it's definitely the nesting depth that kills it, not the total token count. are you seeing the same behavior where flattening helps, or is your json structure inherently tree-shaped?
what exact quant and batch size. need to replicate this
hit this exact thing testing llama 3.3 on kubernetes manifests!!! around 19.4k the model starts hallucinating nested fields that don't exist in the schema - like it'll invent `spec.template.spec.containers[].imagePullSecrets` when the actual manifest only has `spec.imagePullSecrets`. super annoying for codegen
+1 this is brutal. we're using llama 3.3 70b q4_k_m for kubernetes manifest generation in our CI pipeline and if the model starts hallucinating nested fields around 19.4k that completely breaks our workflow. our production k8s manifests have tons of deeply nested specs (deployments with init containers, volume mounts, security contexts, resource limits) and they regularly hit 18-20k tokens. if perplexity degrades at 19.7k we can't trust the generated manifests. can you post the exact perplexity curves and what fields it hallucinates? need to know if this is uniform degradation or specific to certain nested structures like containers.resources.limits vs containers.volumeMounts
the tokenizer is the issue here. nested json with deep object hierarchies gets chunked weird past 19k and the model loses track of brace depth. same failure mode shows up in latex, yaml, any nested syntax really
this is a blocker if you're doing any kubernetes work. we had to switch to qwen 2.5 for manifest generation because llama 3.3 just falls apart past 19k