client asked for q4 to save vram, now fighting silent json failures past 19k
shipped llama 3.3 70b q4 for a client's rag system two weeks ago to keep vram under 48gb. everything looked great in testing with short contexts but now we're seeing silent json schema failures past 19k in prod. model generates syntactically perfect json but keys drift from the schema - "created_at" becomes "createdAt", "user_id" becomes "userId", nested objects get flattened. validation fails but the json looks so plausible it took days to notice. tested q5 yesterday at same context lengths and it stays solid until 26k. the vram cost hurts (6.2gb more in our setup) but silent failures in prod cost way more in debugging time and customer trust. probably gonna eat the vram cost and migrate to q5 for anything past 15k context. anyone else made this tradeoff?
the q4 vram savings aren't worth silent failures. we ate the cost and moved to q5 for anything past 15k context - yes it's more expensive but at least schema validation actually works. what's your actual context size where this breaks?