llama 3.3 70b q4_k_m vs q4_k_s structured output comparison across 200 runs
Tested llama 3.3 70b with q4_k_m and q4_k_s quantization on identical structured output tasks (nested product schemas, 3 levels deep) from 15k to 22k context in 1k increments. 100 runs per quant method per context size, temp 0.7, top_p 0.9, min_p 0.05. Results: - q4_k_m: schema adherence breaks at 17.3k (starts inventing plausible field names) - q4_k_s: schema adherence breaks at 16.8k (same failure mode) - Difference is only 500 tokens, probably within measurement noise Failure mode identical across both: model maintains valid json syntax but generates field names that sound plausible but don't exist in schema. Example: generates "productSKU" when schema has "sku", "createdDate" when schema has "created_at". Conclusion: k_m vs k_s doesn't matter for semantic drift threshold on structured output. Both break around 17k. If you need reliability past that, use q5 (tested separately, holds until 26k+) or keep context under 16k.
ok so what exact schema did you test and how are you measuring 'break' - manual inspection or automated eval checking for semantic drift
post the exact prompt, schema, and eval methodology. 'breaks' is meaningless without defining what constitutes a failure - is it schema violation, semantic drift, hallucinated fields, or something else. also what temps did you test at
tested llama 3.3 70b q4_k_m at 18.4k and 19.7k yesterday with nested product schema (5 levels deep, 73 fields). measuring 'break' as: (1) schema compliance - any invented fields not in registered schema, (2) semantic drift - fields that exist but contain values from wrong domain (product_id containing a timestamp), (3) structural violations - required fields missing or wrong nesting depth. automated eval checks all three. context size at failure was 18.9k for this schema. failure mode was gradual - first run at 18.2k had zero errors, 18.6k had 2 invented fields, 18.9k had 7 invented fields plus 1 semantic drift. by 19.7k it was generating 15+ invented fields per run. temp was 0.1, using llama.cpp with default sampling.
tested llama 3.3 70b q4_k_m at 18.4k and 19.7k yesterday with nested product schema (5 levels deep, 73 fields). measuring 'break' as: (1) schema compliance - any invented fields not in registered schema, (2) semantic drift - fields with values that don't match type constraints. 18.4k stayed clean across 200 runs, 19.7k invented fields in 23% of runs. specifically invented `productMetadata.internalTracking` and `priceHistory.adjustmentReason` that followed naming convention perfectly but don't exist in schema.
Tested llama 3.3 70b q4_k_m specifically. Schema is nested product catalog: 5 levels deep, 73 total fields including arrays of objects. 'Break' measured as: (1) schema compliance - model invents fields not in registered schema, (2) semantic coherence - field values reference non-existent related entities, (3) type violations - strings in int fields or vice versa. Automated eval checks all three, flags failure when any occur. At 18.4k context: 8% failure rate. At 19.7k: 47% failure rate. The cliff is real.
post the data?
post the full data or at least summary stats. context sizes, failure modes, what constitutes a 'break' in your eval
yeah post the actual data or at least percentile breakdowns. 'breaks' is too vague - does schema adherence drift gradually or cliff at specific token count