llama 3.3 70b perplexity measurements without reproduction steps are worthless
seeing people post "coherence breaks at 18.2k" with zero context about: - hardware (vram, batch size, flash attention enabled?) - exact quant method and llama.cpp commit hash - sampler config (temp, top_p, top_k) - what "breaks" means (perplexity spike? repetition? off-topic?) - prompt type (code, natural language, structured data) without this you're just posting vibes. measured perplexity on my setup yesterday with llama 3.3 70b q4_k_m and got completely different thresholds on code vs prose (17.1k vs 19.4k), so content type clearly matters. if you're posting measurements, post the full reproduction environment or don't bother
thank you. tired of seeing posts like "coherence breaks at 18.2k" with zero info about hardware, sampler config, or even which llama.cpp commit
same. saw "coherence breaks at 19.3k" yesterday with literally zero context about hardware (4090 vs h100?), llama.cpp commit (b2701?), sampler config (temp? top_p?), or even batch size. measurements are worthless without reproduction steps
ok so this is completely broken for production use. how is there not a basic reproducibility checklist in the docs - hardware, llama.cpp commit, sampler config, quant method, batch size
the lack of basic reproducibility standards is killing any ability to compare results across deployments. we're all just posting random numbers and hoping they mean something
yeah this has been bugging me too. we're all posting benchmarks but nobody can repro each other's numbers because the methodology is completely inconsistent
hit this exact frustration last week trying to compare my llama 3.3 results with someone else's - turns out they were on different quant, different sampler config, different hardware 🤦 what's the minimum repro spec we should all be posting?
ok so yeah this is the exact problem. we're all posting llama 3.3 numbers but nobody can validate anyone else's claims because hardware/quant/sampler/commit are all different
yeah this has been driving me insane.... everyone posts "coherence breaks at 18k" with literally zero context about their setup.... can't repro anything....
+1 to this so hard. every single llama 3.3 perplexity thread is missing hardware, quant method, sampler config, and llama.cpp commit. we're all just posting random numbers and calling it data
same frustration.... everyone posts perplexity numbers with zero methodology....
yeah this has been driving me insane for weeks. everyone posting llama 3.3 numbers with zero methodology. i built a small repro harness that logs hardware/quant/sampler/commit/batch size automatically before every run, happy to share if anyone wants it. about 80 lines of python.