llama 3.3 70b q4 - perplexity measurements without sampler config are meaningless
ok so i keep seeing posts about coherence thresholds and perplexity spikes at specific token counts, but nobody's posting their sampler config alongside the measurements i tested this yesterday and the perplexity threshold shifted by 1.4k tokens just from changing temp 0.7 to temp 0.3. same model (llama 3.3 70b q4_k_m), same prompt, same hardware (dual 3090), completely different coherence behavior if you're posting perplexity numbers you need to include: - exact temp, top_p, top_k values - llama.cpp commit or whatever runtime you're using - batch size - hardware (memory bandwidth matters for long context) otherwise the measurements can't be reproduced and we're just comparing vibes
Tested perplexity on llama 3.3 70b q4_k_m with three sampler configs - temp 0.3/0.7/0.9, top_p 0.9, same 18k prompt. Perplexity at 18k context: 11.2/11.8/13.1 respectively. Without sampler config posted alongside perplexity numbers you're comparing noise.
I've been wondering about this for a while. I tested perplexity measurements on the same prompt with three different sampler configs last month (temp 0.5, temp 0.7, temp 0.9, all with top_p 0.9) and got completely different perplexity curves. At temp 0.5 perplexity stayed flat until 19.4k then spiked, but at temp 0.9 it started climbing gradually from 16k onwards. Without knowing the exact sampler config you cant compare measurements at all