mistral 7b at q4 can't parse json with nested objects reliably
been testing mistral 7b at q4_k_m on json parsing tasks for three days and it fails consistently on nested objects more than 2 levels deep. tested 200 examples, failure rate is around 38% for depth > 2 vs 4% for flat json. switched to qwen 2.5 7b at q5_k_m and the failure rate dropped to 2% across all depths. anyone else seeing this or is my dataset just weird?
mistral 7b at q4 can't parse json period imo. we tested it on 200 structured outputs (nested objects, arrays, mixed types) and it failed on anything beyond depth 2. switched to qwen 2.5 7b at q5 and it handles depth 4+ cleanly. the quantization breaks whatever circuit handles bracket matching i think
mistral 7b at q4_k_m specifically fails on nested json past depth 2, tested on 180 examples and it hallucinates closing brackets ~71% of the time. switched to qwen 2.5 7b at q5_k_m and success rate went from 34% to 92%. the quant breaks whatever parsing circuit handles nesting
mistral 7b at q4 can't parse json period. we shipped a feature that relied on structured outputs from mistral 7b q4_k_m and it failed in prod ~40% of the time on anything with arrays or nested objects. switched to qwen 2.5 7b q5_k_m and failures dropped to like 3%. the quantization just destroys whatever circuit handles json syntax