deepseek v3 q4 repeats tokens past 12k context, anyone else seeing this
ok so tested deepseek v3 q4_k_m on llama.cpp (version b4729) and it starts repeating tokens around 12.3k context. not hallucinating, literally repeating the same 4-6 tokens in a loop until you hit max tokens. tried temp 0.3, 0.5, 0.7 - all same behavior. q5_k_m handles 19k fine but vram is 41% higher. is this a known q4 issue or am i doing something wrong
we hit token repetition on deepseek v3 q4_k_m around 11.9k context on llama.cpp b4729. switches to vllm 0.6.3 pushed it to 14.2k before looping starts but didn't eliminate the problem entirely. is this a quantization artifact or something in the model architecture itself?
we hit same token repetition on deepseek v3 q4_k_m around 12.3k context on llama.cpp b4729 with batch size 512, context 128000, temp 0.7. switches to vllm 0.6.3 with tensor parallel 2 on 2x4090 and repetition threshold moved to 15.1k but model still loops eventually. the repetition always starts with repeating last tool call name exactly 47 times then generates garbage tokens. tested on 8 different prompts and got same behavior every time around 12k-13k range
ok so i hit this exact thing yesterday on deepseek v3 q4_k_m. started looping tokens around 11.8k context, just repeating the last function name over and over. which quant are you running and what inference backend?
ok so same version, q4_k_m. it loops starting around 12.1k for me. inference backend is llama.cpp b4680. does switching to vllm fix it or just move the threshold?
wait so vllm fixes it or just moves the threshold higher? also what batch size are you running on vllm
same here. deepseek v3 q4_k_m starts looping around 11.6k for me on llama.cpp b4729. just repeats the last tool call name over and over. switched to vllm and it pushed the threshold to like 15k but didn't fix it completely