llama 3.3 70b at q4 returns different outputs for identical prompts
running llama 3.3 70b at q4_k_m on llama.cpp, same prompt (temp=0, top_p=1, seed=42), getting different outputs across runs.... not talking about creative variance, literally different factual answers to the same question. tested 40 runs of "what is 127 * 384" and got 3 different answers (one correct, two hallucinated). fp16 is deterministic as expected. is this a known quantization artifact or am i missing something in the llama.cpp config
ok so seeing this too on llama 3.3 70b q4_k_m, maybe 25% of runs give different outputs at temp 0. thought my sampler was broken but sounds like its the quant
temp 0, q4_k_m, seeing ~18% non-determinism
1. need to know which sampler you're using (mirostat vs top-p vs typical) 2. also vram usage during inference tested llama 3.3 70b q4_k_m at temp 0 last week and got deterministic outputs 100% of time with top-p sampling
which llama.cpp version? seeing determinism issues on older builds with certain samplers
seeing this too.... llama 3.3 70b at q4_k_m gives different outputs for identical prompts maybe 30% of the time. thought i was going crazy but temperature is set to 0. what sampler are you using?
check your sampler settings and also the quant method. we saw non-deterministic outputs on llama 3.3 70b at q4_0 (not q4_k_m) even at temp 0, turned out the quantization was introducing floating point noise that got amplified in the sampling. switched to q5_k_m and it's deterministic again. also this is why i always come back to just using fp16 for anything serious
need actual numbers - what % of runs are non-deterministic and what sampler settings
lol ya seeing this too on llama 3.3 70b q4, temp=0 still gives diferent outputs maybe 20% of time. switched to q5 and it fixed it
need the actual sampler settings and also vram usage. also whats the quant method exactly (q4_0 vs q4_k_m vs q4_k_s)? seeing non-determinism at temp 0 usualy means something is broken in the quant or the sampler is using a random tiebreaker
tested llama 3.3 70b q4_k_m (version llamafile-0.8.13) at temp 0.0 on 80 identical prompts yesterday. got 100% deterministic outputs with top_p=1.0, min_p=0.0, typical_p=1.0. non-determinism at temp 0 usually means either the sampler is broken, vram pressure is causing numerical instability, or the quant method has a bug. what's your actual sampler config (not just temp) and are you hitting vram limits during inference? also which quant tool did you use to create the q4 - llama.cpp, exllamav2, or something else?