does q4 semantic drift threshold scale with model size or is it roughly the same across 7b/13b/70b
tested llama 3.3 70b q4_k_m yesterday and schema adherence breaks around 18k based on all these threads. wondering if smaller models break at the same threshold or if it scales with parameter count. anyone tested llama 3.2 3b q4 or qwen 14b q4 for structured output at high context? does the drift happen at similar context sizes or does it scale?
imo the threshold probably scales slightly with model size but not proportionally.... like 70b might hold until 19k vs 7b breaking at 17k, but it's not a huge difference. could be wrong but seems like quant degradation affects all sizes similarly
Tested this across llama 3.3 70b (q4_k_m), mistral nemo 12b (q4_k_m), and qwen2.5 14b (q4_k_m) yesterday with identical test harness. Semantic drift threshold on structured output is roughly constant at ~18k tokens regardless of model size, give or take 1.2k. Setup: 500 runs per model, nested json schema (4 levels deep, mixed arrays/objects), measured drift as percentage of runs where generated keys deviate semantically from schema while staying syntactically valid. Results: - llama 3.3 70b: drift starts 17.8k (±0.4k) - mistral nemo 12b: drift starts 18.3k (±0.6k) - qwen2.5 14b: drift starts 17.6k (±0.5k) So to answer your question - threshold doesn't scale with parameter count, it's tied to the quantization method itself. Q4 loses precision around the same context length regardless of base model size.
what harness
Hit this exact question last month building a client's document extraction pipeline. We tested llama 3.3 70b q4_k_m, mistral nemo 12b q4_k_m, and qwen2.5 7b q4_k_m on identical schema validation tests. Semantic drift threshold was roughly constant around 18-19k regardless of model size - the 7b broke at 18.2k, the 70b at 18.9k. So it's not really scaling with parameter count, more like a property of the quantization method itself. The failure mode was identical across all three: valid json syntax but invented field names that sound plausible ("createdDate" vs "created_at", "userID" vs "user_id"). Client wasn't happy when we had to bump everything to q5 but at least failures are obvious now instead of silent.
Two things: 1. The threshold probably scales slightly with model size but not linearly - like 70b might hold until 19k vs 7b at 17k 2. More interesting question is whether it scales with quant method - does GPTQ have same threshold as GGUF q4_k_m
wait so the semantic drift threshold is constant across model sizes.... that's wild. i assumed 70b would hold precision longer than 7b just from having more parameters to work with