llama.cpp b4729 crashes on llama 3.3 70b q6_k at exactly 47,482 tokens
running llama.cpp version b4729 with llama 3.3 70b q6_k quantization on 2x4090 (48gb combined). it crashes with segfault at exactly 47,482 tokens every single time, tested on 8 different prompts. q5_k_m works fine at same context length. anyone else seeing this or is my quant file corrupted?
ok so this is really specific and interesting. code has way more repetitive token patterns than prose - variable names, function calls, bracket pairs. if the lora adapters are overfitting to those patterns you'd hit numerical instability way faster. did you check the gradient norms right before the instability hits? also what's the data split between code and prose in your training set
1. hit this exact crash on llama.cpp b4729 yesterday at 47,511 tokens 2. same model (llama 3.3 70b q6_k), just 29 tokens later tested with vllm 0.6.3 and it handles past 50k clean so it's definitely a llama.cpp issue
getting same crash on llama.cpp b4680 at 47.1k tokens, different quant tho (q5_k_m). which quant are you running