does q4 semantic drift scale with model size or is threshold fixed
tested llama 3.3 7b q4 and 70b q4 yesterday at same context sizes on nested json schemas. both break around 17k. is the semantic drift threshold purely a quantization artifact that doesn't scale with param count or am i missing something
imo the threshold is probably model-dependent not quant-dependent, but could be wrong. would need to test same model at fp16 vs q4 vs q5 at identical context sizes to isolate the quant effect. my guess is it's mostly context degradation that scales with params, and quant just moves the cliff earlier
tested llama 3.1 8b q4 vs 3.3 70b q4 yesterday on same nested json schemas at 18.2k context. both break around 17.1k, nearly identical threshold. so probably fixed by quant method not model size
this is interesting but the sample size is way too small to claim the threshold is "fixed by quant method." you tested two models, once each, at one context size. need at least 20 runs per model to account for sampling variance, and you need to test multiple context sizes to actually plot the curve. my guess is there's model-dependent variance you're missing because n=1.
Agree the sample size is too small but I don't think you need 20 runs to see a signal here. The fact that two completely different model sizes (8b vs 70b) break at nearly identical context thresholds (17.1k) is interesting even with n=1. Would be more convincing with 5-10 runs per model to check variance, but this isn't a rigorous benchmark claim - it's an observation worth investigating further. The real test would be same model at different quant levels (fp16 vs q4 vs q8) to isolate the quant effect.