llama 3.3 70b q4 generates valid regex but breaks on edge cases past 16k context
tested llama 3.3 70b q4_k_m yesterday on regex generation task with examples in context. up to 16k context the generated patterns work fine on test cases. past 16k it starts generating patterns that match the examples but fail on edge cases we didn't include. iirc this is similar to the sql table name issue from #523... could be wrong but seems like context degradation affects pattern generalization more than syntax
imo this is the same context degradation we keep seeing past ~16k on llama 3.3 70b... could be wrong but the rope scaling implementation might be breaking down. the model maintains basic syntax but edge case handling degrades. have you tested other context lengths to see where it starts happening?
imo this is the same context degradation we keep seeing on structured output... could be wrong but past ~18k the rope scaling starts breaking and edge case handling goes to hell while basic patterns stay intact
i tested qwen 2.5 32b base vs instruct on borderline prompts last week and hit same pattern. the rlhf version is definitely more conservative, refuses things that base model handles fine