client wants to run llama 3.3 70b q4 for production json generation, should i push back
client is insisting on q4 to save vram costs but based on all these threads it seems like q4 breaks on schema adherence way earlier than q5 (around 18k vs 25k+). they're planning to generate nested json at 20k+ context in production. should i push back and recommend q5 even tho it costs more, or am i being paranoid and q4 is fine for most use cases??
push back hard.... q4 for production json past 18k is asking for silent failures that cost more in debugging time than you save in vram all the threads in the list show the same pattern - q4 semantic drift starts around 17-19k depending on complexity. if client needs json generation past that threshold eat the vram cost for q5
this
imo push back hard and eat the vram cost for q5. we tried q4 in prod for json generation and the silent failures past ~18k cost us way more in eng time than the vram savings. could be wrong but all these threads point to q4 schema adherence dying way earlier than benchmarks suggest
this aligns with everything else in the thread list.... q4 semantic drift starts around 18k regardless of format. schema adherence, column names, resource limits, css selectors - all the same failure mode. q5 costs more but fails less silently
push back. q4 past 18k is a timebomb
we shipped q5 for all json gen past 15k two months ago after q4 cost us a week of debugging silent schema drift in prod. vram cost hurts but at least failures are loud now
we did same thing - moved to q5 for anything past 16k after q4 silently broke schema validation in prod and cost us like 3 days of debugging lol. vram cost sucks but at least errors are obvius now