deepseek v3 q4 hallucinates way more than llama 3.3 at same context size... seeing this or just me
running deepseek v3 q4_k_m and llama 3.3 70b q4_k_m on identical eval tasks (json schema generation, sql queries, function calling) at 16k context. deepseek invents plausible garbage way more often - like 40% schema violation rate vs 8% for llama at same context size.... both models trained to similar context windows so why is deepseek so much worse at staying grounded
yeah seeing way worse coherence on deepseek past 16k. llama stays solid until like 23k for us but deepseek drops hard around 17.2k on same workload
seeing same thing on our setup, deepseek drops way earlier. we switched back to llama for anything past 16k bc the coherance degredation was brutal on deepseek
ok so yeah deepseek coherence drops way earlier than llama for us too.... seeing similar threshold around 16.8k vs llama staying solid until 19k+. probably rope config or attention pattern difference
yeah deepseek coherence threshold is way lower than llama in our testing too. we ran identical structured output tasks at 17.2k yesterday - deepseek v3 q4 started inventing garbage around 16.4k, llama 3.3 70b q4 stayed solid until 19.6k. probably rope scaling config difference but also curious if it's training data distribution - llama was explicitly trained on longer context during pretraining and deepseek might not have the same long-context emphasis. would be interesting to test both at multiple context sizes (12k, 16k, 20k, 24k) with identical schemas to see if the gap widens linearly or if there's a specific threshold where deepseek falls off
ok so tested deepseek v3 q4 vs llama 3.3 70b q4 yesterday at 18.2k context on same json schema tasks. deepseek invented plausible garbage way earlier - around 16.8k vs llama staying solid until 19.1k. not just you, the coherence drop is real and it's frustrating bc the benchmarks don't show this at all
tested both at 18.4k yesterday, same workload. deepseek invents garbage around 16.9k, llama stays solid until 19.3k. probably rope config or attention pattern difference.
what exact deepseek version and llama.cpp build? also curious if rope config difference explains this - llama trains to 128k with specific rope scaling, deepseek might have different base freq
probably rope freq. llama uses 500k base, deepseek might be different.
probably rope freq base difference yeah. llama trains to 128k with 500k rope base iirc, deepseek might be using different config. could be wrong tho - would need to check actual model cards