llama 3.3 70b q4_k_m loses coherence past 38k context despite 128k claims
tested llama 3.3 70b q4_k_m on llama.cpp b4680 with prompts from 8k to 80k tokens. coherence is solid until about 38k, then it starts repeating phrases and forgetting instructions from earlier in the context. by 50k it's basically useless - just loops on the last few sentences or hallucinates that it already answered. the 128k context window claim is marketing. real usable context on q4 quant is maybe 40k max
ok so this is the thing that drives me insane about model releases. they claim 128k context but real performance falls apart at 30-40k depending on quant and task. we tested llama 3.3 70b q4_k_m on summarization tasks and coherence degrades hard past 36k - it starts dropping key facts from early in the context and hallucinates connections that don't exist. the benchmarks they use for the 128k claim are probably needle-in-haystack which is not representative of real tasks at all. nobody is just retrieving a single fact from long context, people are doing reasoning and synthesis and that breaks way earlier.
post the llama.cpp version and quant. also does it fail on specific tasks or just general coherence
tested llama 3.3 70b q5_k_m on vllm 0.6.3 past 40k and coherence holds until around 51k. llama.cpp definitely falls apart earlier
tested llama 3.3 70b q4_k_m on same context range.... coherence falls apart at 36.2k for me on llama.cpp b4729. which backend are you using