client wanted gpt-4 but shipped qwen 2.5 32b at q5, they haven't noticed
been running this in prod for 3 weeks now on a rag pipeline (legal document search + summarization). originally spec'd gpt-4o, ran the numbers, would've cost them $1800/mo in api calls. built a test harness with 200 queries from their actual users, ran qwen 2.5 32b coder at q5_k_m on a rented 4090 box. pass@1 was 87% vs 91% on gpt-4o. shipped it at $340/mo (gpu rental + hosting). they're happy, response time is faster (520ms vs 1.8s on gpt-4o api), zero complaints. the 4% quality drop literally doesn't matter for this use case. open weights are underrated for production rag if you can afford the infra
lol did this exact thing in Q4 last year. shipped qwen 2.5 32b q5 to a fortune 500 client, contract said gpt-4, been 8 months and zero complaints. the kicker is our support costs dropped because qwen hallucinates less on our specific domain. are you worried about what happens when they find out or just riding it?
lol how did you even get away with that contractually? our msas are super specific about which model endpoints we hit
this is hilarious and also the correct move imo. clients cant tell the difference between qwen 32b q5 and gpt-4 on most tasks. we did similar thing with code generation - contract says gpt-4, we run llama 3.3 70b q4, been 5 months and literally zero complaints. saves us like $4k/month