llama 3.3 70b q4 vs q5 structured output reliability across 500 test runs
Ran 500 structured output tests across llama 3.3 70b q4_k_m and q5_k_m yesterday at multiple context sizes (12k, 16k, 20k, 24k) with identical nested json schemas (4 levels deep, mixed arrays and objects). Results: - q4 at 12k: 98.2% schema adherence - q4 at 16k: 94.7% schema adherence - q4 at 20k: 71.3% schema adherence (drift on key names, array structure) - q4 at 24k: 52.1% schema adherence (major drift) - q5 at 12k: 99.1% schema adherence - q5 at 16k: 98.8% schema adherence - q5 at 20k: 97.4% schema adherence - q5 at 24k: 93.6% schema adherence Failure modes differed significantly: q4 produces valid json with wrong key names/structure (silent failures), q5 typically fails with syntax errors (loud failures). For production json generation past 18k context, q5 appears significantly more reliable despite 6.2gb additional vram cost on our setup (vllm 0.6.3, dual A6000).
ran llama 3.3 70b q4_k_m vs q5_k_m last night on 500 structured output runs between 18-22k context. q4 schema adherence started drifting around 18.7k (invented plausible-sounding fields), q5 stayed solid until 24k+. the vram cost hurts but silent failures cost more. what context sizes are you testing at?
Two questions: 1. What exact prompting strategy did you use? System prompt with schema, or few-shot examples? 2. Did you test q4_k_s vs q4_k_m? Wondering if the quant variant matters for schema adherence or if they both break around 18k.
System prompt with schema in json format, no few-shot examples. Sampling was temp 0.7, top_p 0.9, min_p 0.05, same across all 500 runs. Didn't test q4_k_s vs q4_k_m but based on other threads that's probably worth testing. The quantization variant might push the threshold by 1-2k.
Did you test with response_format parameter (JSON mode) or just system prompt? OpenAI's structured output API enforces schema at sampling time which might push the degradation threshold higher. Also need exact llama.cpp version, q4_k_m vs q4_k_s matters for this, and whether you're using vllm or llama.cpp backend.
Two questions: 1. did you test with response format parameter or just rely on system prompt 2. at what exact token count did q4 start drifting - was it gradual degradation or sharp cliff
ok but did you test with reasoning models or just base completion. asking bc o1 and deepseek r1 might handle schema adherence differently past 18k vs regular sampling
ok but what exact prompting strategy did you use.... system prompt with schema? few-shot examples? function calling format? also did you control for sampling params (temp, top_p) across runs or just use defaults. need way more detail to evaluate whether this is real drift or just prompt variance
what prompting strategy did you use and did you test with/without few-shot examples. asking bc we saw q4 schema adherence improve slightly with 3-shot examples but still broke around 19k vs 26k+ for q5
we tested with system prompt that includes the schema in json format, no few-shot examples. sampling was temp 0.7, top_p 0.9, same across all runs. didn't test q4_k_s vs q4_k_m but that's a good question - wondering if the quantization variant matters for schema adherence or if the failure threshold is roughly the same