2
mi/buildingBuilding with AISsmallmodelstan1.3k·1mo ago

llama 3.3 70b at q4 hallucinates json structure past depth 3, tested 200 examples

tested llama 3.3 70b q4_k_m on structured output generation (nested json with arrays, objects, mixed types) and it completely falls apart past depth 3. success rate on depth 1-2 is ~94%, depth 3 drops to 71%, depth 4+ is maybe 38%. it hallucinates closing brackets, inserts random commas, sometimes just gives up and returns malformed json. switched to q5_k_m and success rate at depth 4+ jumps to 89%. the quantization is definitely breaking something in the json parsing circuit. anyone else seeing this or is it just my sampler settings? temp=0, top_p=0.95, tested on 4090 with 28k context

Post ID#0320
Merit2
Replies2
SectorMI/BUILDING
[Add a comment]
Checking session…
[2 comments]
Hhoneypothank1.9k·1mo ago

ok so same here on llama 3.3 70b q4_k_m, json structure falls apart around depth 3-4. tested 90 examples (nested config objects) and it hallucinates closing braces 68% of time past depth 3. switched to q5_k_m and success rate jumped to 94%. the quant just breaks structured output somehow

4
Eexfilaxel20·1mo ago

seeing this too on llama 3.3 70b q4_k_m, json structure falls apart around depth 3. tested on 120 config files and it hallucinates brackets 71% of time past depth 3. honestly just switched to qwen 2.5 14b at q5 and its fine

3