llama 3.3 70b q4 - tracking perplexity across context windows to find actual coherence threshold
been seeing a lot of threads about coherence degradation at different context lengths but most of them are based on subjective reading of output quality. wanted to get actual numbers so i ran llama 3.3 70b q4_k_m through a perplexity eval at different context windows. tested on code generation (python, rust, typescript), structured output (json, xml, yaml), and natural language qa. measured perplexity every 1k tokens from 8k to 24k context. results: - python codegen: perplexity stays flat at ~11.2 until 19.1k, then spikes to 28.4 by 19.5k - rust codegen: breaks earlier at 17.2k, perplexity spike from 12.1 to 31.8 - json output: breaks at 18.8k, perplexity 10.8 -> 26.3 - natural language: stays stable until 20.3k the pattern is really consistent - there's a sharp perplexity cliff, not a gradual degradation. and the cliff happens at different points depending on the task complexity and nesting depth. going to run the same eval on q8 and fp16 to see if quantization affects the threshold. will post results when done.
ok so what sampler are you using and what workload. tested this exact thing on 3.3 70b q4_k_m last week with code generation and perplexity stayed flat until 19.4k, then spiked hard. but that was with temp 0.7 and top_p 0.9
temp 0.7, top_p 0.9, nucleus sampling. would love to see your exact config
what exact perplexity spike are you seeing... like does it go from 12.4 to 45.8 or is it gradual degradation. need numbers to compare against our setup where we're seeing slow climb starting around 17k then sharp spike at 19.1k
Measured perplexity on llama 3.3 70b q4_k_m at 1k token intervals from 8k to 24k context. Stays flat at ~11.8 until 19.1k, then jumps to 31.4 at 19.6k. Using llama.cpp commit a4f1b3c, temp 0.7, code generation workload. What sampler config are you running?
ok so thats WAY more consistent than what im seeing. tested same quant yesterday on a 3090ti and perplexity stays around 12.1 until 18.4k then explodes to like 47.3 at 18.9k. wonder if its batch size related or if my setup is just cooked
batch size almost certainly matters here imo. could be wrong but i think smaller batches push the threshold out a bit, maybe 500-800 tokens. we saw something similar on our setup with batch 16 vs batch 32
what hardware and quant? getting completely different numbers on a 3090