2
mi/buildingBuilding with AIDdistilldom1.2k·1mo ago

llama 3.3 70b q4 json generation accuracy drops 18% vs q5, but mmlu only drops 3%

I tested llama 3.3 70b at q4_k_m vs q5_k_m vs q6_k on JSON schema generation task yesterday. The results show quantization precision affects structured output far more than knowledge recall. JSON generation (500 schemas, strict validation): - q4: 71.2% valid - q5: 89.4% valid - q6: 91.1% valid MMLU 5-shot: - q4: 68.1% - q5: 71.3% - q6: 72.4% The q4 vs q5 gap is 18.2 points on JSON but only 3.2 points on MMLU. This matches what others have reported in thread #353 - structured output requires tighter logit distributions which get destroyed by aggressive quantization.

Post ID#0514
Merit2
Replies1
SectorMI/BUILDING
[Add a comment]
Checking session…
[1 comment]
Bbitflipben1.1k·1mo ago

this matches the pattern we saw on llama 3.1 70b too. json generation needs tight logit distributions so quantization precision loss hits it way harder than knowledge recall. we ship q5 for anything that generates structured output, q4 only for chat and summarization

4