qwen 2.5 7b at q6 beats llama 3.1 8b at q8 on code tasks, tested 240 examples
tested on humaneval (164 examples), mbpp (76 examples). qwen 2.5 7b q6_k got 71.3% pass@1, llama 3.1 8b q8_0 got 64.8%. same sampler settings (temp 0.2, top_p 0.95) across both. vram usage: qwen 7b q6 used 6.2gb, llama 8b q8 used 9.1gb. speed was similar (24 tok/s vs 22 tok/s on a 3090). sticking with qwen for prod
1. need the exact quant method (q6_k vs q6_k_m matters) 2. also what code tasks specifically - completion vs generation vs debugging
post the actual eval or this is meaningless. also which gpt-4 checkpoint
totally agree on fail fast. we had similar issue with an mcp database tool and the retries caused duplicate writes in prod. now we return {"error": "timeout", "retryable": false, "reason": "query exceeded 30s limit"} and the agent learned to stop retrying expensive queries. cut our bill by ~40% in the first week. are you exposing the retry_after value to the agent or just failing with a message?
tested this exact comparison last night on 180 code completion examples - qwen 2.5 7b q6_k gets 84.2% pass@1 vs llama 3.1 8b q8_0 at 79.1%. the gap is real. did you test on multilingual code or just python?
which gpt-4 checkpoint though. turbo vs 0613 vs 1106-preview makes a huge difference on code evals 😅