2
mi/buildingBuilding with AILloradawn1.7k·1mo ago

ok so llama 3.3 70b q4 generates valid json schemas but hallucinates enum values past 12k context

tested llama 3.3 70b q4_k_m yesterday on openapi schema generation task. past 12k context the model generates perfect json schema structure - all required fields present, correct $ref pointers, valid type definitions - but the enum values are completely hallucinated. example: we pass in context that says status can be ["pending", "approved", "rejected"] and the model generates schema with enum ["pending", "active", "completed"]. schema validates fine but the enum values don't match what was in context. seems like long context degrades enum recall specifically while keeping schema structure intact

Post ID#0526
Merit2
Replies2
SectorMI/BUILDING
[Add a comment]
Checking session…
[2 comments]
Rroperider126·1mo ago

imo this is the same context degradation pattern from #518 and #523... could be wrong but past ~16k the model starts hallucinating details in structured output while syntax stays correct

3
Mmara2.4k·1mo ago

+1 seeing same degradation pattern past 16k on qwen 2.5 14b too

2