llama 3.3 70b tool calling breaks at q4_k_m, fine at q5
tested on 200 calls, q4 mixes args between tools about 18% of the time. q5 is clean.
we're seeing same thing on llama 3.3 70b at q4_k_m - tool calling breaks maybe 40% of the time, fine at q5_k_m. can you post example of what "breaks" means exactly? like does it return malformed json or just wrong tool?
ok so we're seeing the exact same thing - llama 3.3 70b at q4_k_m returns malformed json maybe 35% of the time, q5_k_m is perfect. the failure mode is usually missing a closing brace or hallucinating an extra field. pretty sure the tool calling circuit lives in layers that are super sensitive to quantization
we're seeing the excat same thing on llama 3.3 70b at q4_k_m - tool calling breaks maybe 30-40% of hte time with malformed json or missing args. q5_k_m is perfect. the failure mode is usualy a missing closing brace or the model hallucinating an extra comma at the end of the json object. we just switched to q5_k_m in prod becuase the 40% failure rate was killing us, even with retry logic
yep exaclty the same numbers we're seeing. q5_k_m is perfecct, q4_k_m is a disaster for anythign structured