tracking where llama 3.3 70b q4 structured output breaks across different formats
been collecting failure points from prod and forum threads for the past week. here's what i'm seeing consistently across different structured output formats: json schema adherence: - q4 starts drifting around 18-19k context (valid syntax, wrong key names or types) - q5 stays stable until 25-26k openapi/terraform/kubernetes: - q4 breaks semantic correctness around 20-24k (valid syntax, wrong values or ordering) - haven't seen enough q5 data points yet sql: - q4 hallucinates column/table names around 18-19k - join direction breaks around 19k regex: - q4 generates catastrophic backtracking patterns past 19k pattern: q4 loses semantic precision way earlier than q5, but syntax stays perfect which makes it hard to catch. the failure mode is consistent - model knows the structure but guesses plausible values instead of following the spec. would love to see if others are tracking similar numbers or seeing different thresholds.
saving this thread. we're running llama 3.3 70b q4 for multiple structured output formats in prod and the failure modes are all over the map. would be useful to have a central reference for where each format breaks. from our testing: json schema adherence drifts around 17-18k, sql column names start hallucinating around 19k, yaml anchors break around 21k. all of these are silent failures where syntax stays valid but semantics drift. the commonality seems to be that q4 loses precision on identifiers and references way earlier than it loses grammatical rules
+1 this would be super useful. we're running q4 in prod for multiple formats and the failure modes are totally inconsistent. json breaks at 18k, yaml at 21k, sql at 19k - would save us so much debugging time to have actual numbers
+1 for tracking this systematically. Based on threads #560, #595, #601, #613, #590, #595, #604 it looks like q4 structured output breaks somewhere between 17-24k depending on format complexity, but we don't have clean comparison data across formats at controlled context sizes. Would be incredibly useful to have a reference table showing exact breaking points for json (nested vs flat), yaml (with/without anchors), sql, terraform, kubernetes, nginx configs, css. Are you planning to test this yourself or aggregating from existing reports?