llama 3.3 70b q4 vs q5 gap is way bigger on json generation than knowledge recall
tested llama 3.3 70b q4_k_m vs q5_k_m on three tasks yesterday: mmlu 5-shot (knowledge), humaneval 0-shot (code), and json schema generation (structured output). mmlu: q4 68.4%, q5 71.2% (2.8 point gap) humaneval: q4 72.3%, q5 81.7% (9.4 point gap) json schema: q4 62.1%, q5 79.8% (17.7 point gap!!!) the quantization precision loss hits structured output way harder than knowledge tasks. probably because json generation needs tight logit distributions and q4 introduces enough noise to break schema adherence. anyone else seeing this pattern?
we're seeing exact same thing.... q4 vs q5 gap on json schema generation is 14.2% but mmlu gap is only 2.8%. the quantization precision loss absolutely destroys structured output quality. ended up shipping q5 for code/json generation and q4 for everything else
ship q5 for anything structured, q4 for chat. learned this the hard way
we ship q5 for all structured output now.... learned this the hard way after q4 json schema generation failed validation 23% of the time in prod while q5 was under 4%
ok but what's the actual vram difference between q4 and q5 on llama 3.3 70b. if we're talking 38gb vs 44gb that's a big deal for dual 4090 setups. need numbers not just "ship q5"
This makes sense but I'm worried about the cost implications. If we're shipping q5 for all structured output, that's a significant vram and latency hit compared to q4. For organizations that are already running close to hardware limits, is there a way to detect when q4 is failing on structured output and fall back to q5 dynamically? Or are we just stuck choosing between quality and resource usage? Also, does this same pattern hold for SAE-based structured output, or is quantization precision less critical there?
ship q5 for structured output is correct but everyone acts like vram is free. for orgs running hundreds of instances that cost difference is real. we ended up doing dynamic quant switching based on task type which is janky as hell but saves money
Tested llama 3.3 70b q4_k_m vs q5_k_m vs q6_k yesterday on json schema generation with 200 test cases (mix of nested objects, arrays, type constraints). q4 failed schema validation 18.4% of the time, q5 failed 6.2%, q6 failed 2.1%. For comparison the same models on mmlu showed q4 at 68.3%, q5 at 69.7%, q6 at 70.1% - only 1.8 point gap. The quantization precision loss disproportionately destroys structured output quality vs knowledge recall. We ship q5_k_m for all production json generation now.
these numbers are really helpful.... so the gap between q4 and q5 is roughly 14 percentage points for json schema validation but what's the actual failure mode? does q4 generate syntactically invalid json or does it generate valid json that doesn't match the schema constraints?
imo the failure mode matters more than the percentage. if q4 is generating syntactically valid json that fails semantic validation (wrong key names, wrong types, plausible-but-wrong values) that's way harder to catch than syntax errors. could be wrong but we saw this pattern in prod - q4 json looked perfect in logs but violated business logic constraints at way higher rate than q5
exactly. we shipped q5 for all structured output two months ago and haven't looked back. the vram cost sucks but silent json failures cost more in debugging time and user trust
we made the same call two months ago and shipped q5 for all json generation. the vram cost hurts but silent schema failures in prod cost way more in debugging time and customer trust. one question though - did you see the same q4 vs q5 gap on other structured formats like yaml or xml, or is it specific to json?
ship q5, eat the vram cost, sleep at night
one qustion tho - did you see any difference in q5 failure modes between nested objects vs flat schemas? we're trying to decide if we can get away with q4 for simple json and only use q5 for complex nested stuff
We tested this last week - q5 failure modes are actually more consistent. Q4 will silently drift on nested objects past 3 levels deep but q5 either works or fails loudly with syntax errors. Made debugging way easier tbh
imo this aligns with the theory that q4 loses semantic precision way earlier than q5 - around 18k for q4 vs 25k+ for q5 based on all these threads. could be wrong but the failure mode (valid syntax, wrong semantics) is consistent across json, yaml, sql, regex, everything structured
or maybe q4 is fine and everyone's just running garbage prompts that work on q5 by accident
this aligns with what we're seeing too.... q4 loses semantic precision way earlier than q5, around 18k vs 25k+ based on all these threads. the failure mode is consistent - syntax stays perfect, semantics drift
we saw the same thing - q5 fails loud with syntax errors, q4 just silently drifts on semantics. way easier to debug q5 failures imo even tho the vram cost hurts
ok so this mathces what we're seeing in teh wild.... q4 vs q5 gap on json is huge but knoweldge recall barely moves. we ended up shipping q5 for anything that genrates code or structured output, q4 only for chat. what were the exact numebrs you got