llama 3.3 70b q4_k_m perplexity spikes differently on code with comments vs no comments
been testing perplexity degradation on code with inline comments vs stripped code. used same rust codebase, 18.6k tokens with comments, 14.2k tokens stripped. with comments: perplexity holds steady around 11.4 until 17.8k, then spikes to 16.2 by 18.6k without comments: perplexity starts higher at 12.1, stays flatter, only hits 14.8 at 14.2k so comments help coherence in long context but hurt it in short context? or maybe the token density thing is throwing this off and i'm comparing apples to oranges. tested on m2 ultra 192gb, llama.cpp b2701, temp 0.7, top_p 0.9, top_k 40 what's the right way to control for token count differences here?
so.... is this a rope scaling issue or something else.... the fact that perplexity spikes differently suggests the model is treating comments as a different kind of token stream which is weird....
rope scaling would hit both uniformly. this is weird.
this is interesting - need the actual perplexity numbers at same token counts though. comments add context but also noise, curious which dominates past 15k
yeah this matches what i saw last week - comments add local context which helps short-range coherence but past ~12k the extra tokens just become noise. post the numbers though
+1 post the perplexity measurements
1. need the perplexity curves at 5k intervals 2. also post which llama.cpp commit because the rope scaling changed between b1500 and b1700
ok so +1 to posting the curves but also which llama.cpp commit? rope scaling changed between b1500-b1700 and that affects perplexity measurements past 12k
rope scaling break would affect both uniformly though
post token counts for both versions at same logic density