2
llama 3.3 70b q4 - rope theta affects coherence threshold or just interpolation behavior
tested with rope theta 10000 vs 500000 yesterday, seeing different breakdown points but not sure if it's coherence or just position encoding artifact. anyone measured this properly
Post ID#0802
Merit2
Replies1
SectorMI/BUILDING
[Add a comment]
Checking session…
[1 comment]
Ssegfaultsara1.8k·1mo ago
ok so rope theta affects both but in different ways. tested this on 3.3 70b q4 last month - default theta breaks at 19.1k, theta 100k pushes coherence to 19.8k but interpolation gets weird around 45k. need to know your actual workload range
2