4
mi/buildingBuilding with AIGgrokkinggreg79·1mo ago

does rope scaling affect coherence threshold or just let you fit more tokens

I've been testing llama 3.3 70b q4_k_m with different rope scaling factors to try and push the coherence threshold past 19.1k where it normally breaks on my setup. I'm seeing conflicting results and I'm not sure if I'm measuring the right thing. With default rope (no scaling), coherence breaks at 19.1k tokens on code generation tasks - the model starts inventing plausible but wrong function names and package imports. With rope scale factor 2.0, I can fit way more tokens in the context window (tested up to 38k) but coherence still breaks around 19.1k measured by the same tests. So it's like rope scaling lets me fit more tokens but doesn't actually improve the model's ability to use long context effectively? But then I tried rope scale 1.5 (instead of 2.0) and coherence threshold moved to 19.8k, which is a real improvement. So there's some sweet spot where rope scaling helps coherence but too much scaling hurts it? I'm probably doing something wrong with how I'm configuring rope scaling in llama.cpp. Using `--rope-freq-scale 0.5` for 2x scaling. Is there a better way to configure this or am I just misunderstanding what rope scaling actually does?

Post ID#0811
Merit4
Replies3
SectorMI/BUILDING
[Add a comment]
Checking session…
[3 comments]
Eevaleve64·1mo ago

just lets you fit more tokens. coherence threshold stays same

3
Aacdcalex49·1mo ago

both. scaling lets you fit more but degrades quality past trained length

3
Rredteamko1.5k·1mo ago

ok so both? or just token capacity

3