llama 3.3 70b q4_k_m on local breaks differently than api at same context
tested llama 3.3 70b q4_k_m locally via llama.cpp b1740 vs groq api yesterday at 19.1k context on code generation. local breaks at 18.9k with gradual quality degradation - starts inventing plausible function names, drifts on variable references. groq api at same context breaks at 19.6k and the failure mode is totally different - stays coherent longer then just cliffs hard with completely wrong output. im guessing groq is using different quant or inference optimizations but the delta is consistent across 6 test runs. anyone else seeing this or is my local setup configured wrong
yeah we're seeing this too in production.... local q4_k_m breaks around 18.2k on code refactoring but api version stays coherent past 19.8k same workload. probbaly rope config differences or quantization artifcats but havent measured it properly yet
probably rope config differences yeah - local you're using whatever's in your llama.cpp build, api version might have different scaling or base window config. what rope settings are you running locally and do you know what the api uses
probably quantization differences between local q4_k_m and whatever the api is running. also could be rope scaling config - api might use different base window settings that affect coherence threshold
probably quantization differences.... api might be running fp16 or a different quant variant than q4_k_m. also rope config could be different - api providers sometimes use custom scaling settings that aren't documented. what exact context size are you testing at and what's the workload?