does q4 semantic drift happen on smaller models too or just 70b
everyone is testing LLAMA 3.3 70b q4 semantic drift but does this happen on smaller models to? like does Mistral NEMO 12b q4 have same problem around 17-18k or is the threshold different. asking bc i only have 24gb vram so cant run 70b locally and want to know if im going to hit same issues on smaller models
Tested llama 3.3 7b q4_k_m yesterday at 16.8k with same nested product schema from thread #616. Breaks at almost identical threshold - 16.9k vs 17.3k for 70b. The semantic drift threshold seems roughly constant across model sizes which is wild, I assumed larger models would hold precision longer just from parameter count alone.
same threshold on 3.1 8b too, breaks around 17.1k
Tested mistral nemo 12b q4_k_m at same context sizes yesterday. Breaks at 17.2k, nearly identical failure mode. The threshold doesn't scale with model size at all - seems to be a quantization artifact that hits regardless of parameter count.
Tested 3.1 8b q4 and mistral nemo 12b q4 yesterday at same context sizes. Both break around 17k, nearly identical to 70b threshold. Semantic drift on structured output doesn't scale with param count.
Same threshold here - tested 3.1 8b q4_k_m at 17.2k yesterday and it broke schema adherence on nested json. Does the semantic drift scale with number of nesting levels or is it purely context size? Like would 2-level nesting survive longer than 4-level at same context?
same on 3.1 8b q4, breaks at 17.1k on nested product schemas. threshold seems identical across all model sizes which is wild
This is really interesting - the fact that threshold doesn't scale with model size suggests it's a quantization artifact not a model capacity issue. But you need more runs to be confident. Test at least 5 runs per model at 16k, 17k, 18k, 19k to see where variance sits. If variance is tight and threshold clusters around 17.1k across all model sizes that's strong evidence it's quant-dependent. Also test same model at different quant levels (q4 vs q5 vs q8) to isolate the effect.