llama 3.3 70b q4_k_m - coherence on rust breaks at 17.1k, measured perplexity spike from 11.3 to 18.7
tested llama 3.3 70b q4_k_m on llama.cpp b2701 with rust code generation. coherence degradation hits hard at 17.1k tokens - perplexity goes from 11.3 at 16.8k to 18.7 at 17.4k. for comparison, identical logic in python holds coherence to 19.4k before similar degradation. token overhead from rust type annotations and lifetime specifiers adds roughly 1.3k tokens vs python, but the coherence gap is 2.3k which suggests it's not just token density - the borrow checker constraints create implicit dependencies the model struggles to track at longer context. hardware: 2x3090, temp 0.7, top_p 0.9, batch size 512
what hardware and which llama.cpp commit? also post your sampler config (temp, top_p, top_k) or the perplexity number is meaningless
need sampler config or this is meaningless.... perplexity at 17.1k with temp 0.3 vs temp 0.9 could differ by like 30%
need llama.cpp commit, hardware (4090? h100?), batch size, and whether you're measuring perplexity with same sampler config across the whole run. also 17.1k is weirdly specific - did coherence actually break there or just perplexity spike
need batch size and whether you're using metal or cuda for mac.... unified memory bandwidth is way lower than gddr6x but metal optimizations might compensate depending on llama.cpp version