is anyone actually using structured outputs in production or is everyone still parsing json
we switched to structured outputs (openai, the response_format thing) two months ago and it has been great. no more retry loops for malformed json, tool calls are cleaner, cost is maybe 3% higher but we save that in retries. but i am only one seeing this? everyone still talks about parsing and validation like is 2023. am i missing something, is there downside i have not hit yet
we use structured outputs in prod for extracting user intent from support tickets and honestly it works like 91% of the time on haiku which is good enough for our use case. the other 9% we fall back to regex parsing which is janky but works. the big thing is you have to log every failure and actually look at the patterns or youll never know why its breaking
structured outputs in prod, works great for extracting fields from user input. still parse json for legacy endpoints tho
ok so we're using structured outputs in prod for extracting invoice fields and it works great on sonnet but we had to add fallback json parsing anyway because sometimes the structured output just... doesn't validate against our schema? like the model returns valid json but not valid according to the schema we sent. happens maybe 2% of the time. super annoying. are you seeing this or just us
lol we have the same fallback. structured outputs work like 96% of the time on sonnet but that 4% will ruin your day if you don't catch it
we have the exact same setup - structured outputs with fallback json parsing. that 4% failure rate is consistent across sonnet versions for us too, around 96.2% success. do you log what causes the failures?
structured outputs cut our parsing errors by like 80% but we're on haiku not opus, latency matters more than perfection for us. what model r you running
we're on haiku too.... switched from opus because latency was killing us. 3.8s vs 1.1s average and accuracy only dropped like 4%