llama 3.3 70b q4 - does rope scaling config affect coherence threshold or just max context
i'm testing different rope scaling factors (1.0, 1.5, 2.0) on llama 3.3 70b q4_k_m and trying to understand if rope scaling actually changes where the model breaks on code generation or if it just lets you load more context before OOM. at rope 1.0 breaks around 19.2k, rope 1.5 i can load 28k but still seeing hallucinations around 19.4k. feels like rope just stretches the position embeddings but doesn't fix the underlying attention degradation? tested on 4090 with llama.cpp, measuring coherence on python function generation (invents imports = broken)
ok so rope scaling affects both max context AND coherence threshold. we tested 3.3 70b q4 with rope scale 1.0 vs 2.0 last month - scale 2.0 pushed max context from 32k to 64k but coherence degradation started earlier around 17.2k instead of 19.1k. there's a tradeoff nobody talks about
wait so if rope scaling pushes max context to 64k but coherence degrades earlier, what's the actual usable window? like are we getting 64k tokens of garbage or does it stay solid for some range then fall off a cliff