llama 3.3 70b q4_k_m - does perplexity measurement method matter? comparing sliding window vs fixed chunks
I've been running a bunch of llama 3.3 perplexity tests following the threads here and I'm getting wildly different numbers depending on whether I use a sliding window approach (stride 512) versus fixed non-overlapping chunks (4096 tokens each). Same model, same quant (q4_k_m), same hardware (4090), same llama.cpp commit (b3909). Sliding window gives me way smoother perplexity curves but the absolute numbers are like 15-20% lower than fixed chunks at the same context length. Fixed chunks show more variance but might be more honest about where coherence actually breaks? Is there a standard methodology people are using here or is everyone just picking whatever and posting numbers? I want to contribute useful data but I don't want to add more noise to the already inconsistent benchmark situation. Which approach actually tells you where the model breaks down in practice?
sliding window is way more expensive but gives you proper perplexity. fixed chunks are faster but you lose cross-chunk context. which one are you using
sliding window obviously
wait why would fixed chunks even be a comparison point? sliding window is the only valid way to measure perplexity for language models, fixed chunks throw away all the cross-chunk dependencies. need your actual numbers though
sliding window is way more accurate but also like 10x slower in my testing. if your just trying to compare two models quickly then fixed chunks is fine but for actual research you need sliding window
sliding window is definitely more accurate but yeah the speed difference is brutal. for our use case we ended up using fixed chunks for quick model comparison and then only running full sliding window on the final candidate model. saved probably 15 hours of compute time. specific question: are you running this on cpu or gpu, because sliding window parallelizes way better on gpu in my testing?
fixed chunks are way faster but you lose accuracy at chunk boundaries. if your measuring perplexity for actual research use sliding window, if your just trying to compare two models quickly then fixed chunks are probably fine
fixed chunks are literaly only useful for quick model comparison where you dont care about cross-chunk dependencies. if your doing actual perplexity measurement for research or benchmarking you need sliding window, full stop. the chunk boundary artifacts with fixed chunks completley invalidate the measurement for anything except relative ranking