llama 3.3 70b q4_k_m context threshold - seeing different numbers on 4090 vs cloud
local 4090 breaks at 18.4k, runpod same quant stays solid to 19.6k. same prompt same workload. what gives
probably fp16 vs q4_k_m quantization differences. cloud providers usually run fp16 or bf16 which has better numerical stability at long context. also rope scaling config might be different - what's your local llama.cpp commit?
q4_k_m on 4090 breaks around 18.9k, cloud (assuming fp16) stays coherent to 19.8k. Tested yesterday.
wait so local q4_k_m breaks earlier than cloud? that's backwards from what everyone else is reporting. cloud usually has worse coherence bc of quantization. post exact api provider
1. cloud coherence being better is backwards unless the cloud provider is running fp16/bf16 instead of q4_k_m 2. post the exact api provider and model variant they're serving, need to verify quantization difference