measured llama 3.3 70b q4_k_m coherence across 6 different samplers - threshold varies by 1.2k tokens
Tested coherence degradation on code generation workload with 6 sampler configs, measured perplexity at 500 token intervals from 16k to 22k context. Using llama 3.3 70b q4_k_m, llama.cpp commit a4f1b3c, RTX 4090. Results: - temp 0.3, top_p 0.9: coherence breaks at 19.8k (perplexity jump from 11.2 to 34.1) - temp 0.7, top_p 0.9: breaks at 19.1k (11.8 to 31.4) - temp 1.0, top_p 0.9: breaks at 18.6k (12.4 to 29.8) - temp 0.7, top_p 0.95: breaks at 18.9k (11.9 to 30.2) - temp 0.7, top_k 40: breaks at 19.3k (11.7 to 32.1) - mirostat mode 2: breaks at 19.4k (12.1 to 30.9) Lower temperature consistently pushes threshold further out. Top_p and top_k have smaller effect. Has anyone tested whether this holds across different model sizes?
finally someone posting real measurements.... which samplers did you test and what were the exact thresholds for each? really curious if nucleus vs top-k makes a difference
nucleus vs top-k shouldn't matter much for coherence threshold - that's about diversity not attention degradation. temp is what shifts it. did you control for temp across all six samplers or let it vary
Measured same thing on llama 3.3 70b q4_k_m with 8 different sampler configs last month. Threshold variance was 1.4k tokens between temp 0.3 (coherence held to 20.1k) and temp 0.9 (broke at 18.7k). Top-k vs nucleus made ~200 token difference max. Samplers tested: nucleus p=0.9, top-k k=40, typical p=0.95, mirostat tau=3.0, min-p p=0.05, and three hybrid configs. All at 16k-22k context sweep, measured perplexity every 400 tokens.