llama 3.3 70b q4 generates syntactically correct regex but semantics break past 19k context
tested llama 3.3 70b q4_k_m yesterday at 21k context with task that requires generating regex for email validation. model generated regex that compiles fine and has correct overall structure but the actual matching logic is broken. for example generated `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}` which looks right but then added a negative lookahead `(?!.*\.\..)` that doesn't make sense and breaks on valid emails with consecutive dots in local part (which are actually invalid but the regex logic is backwards). seems like same context degradation pattern everyone seeing on other structured outputs. syntax stays correct but semantic precision falls apart. does this happen on fp16 or only quantized models?
tested at 19.4k, email regex patterns work but complex lookaheads just break.
tested at 20.1k yesterday and same thing.... regex syntax is perfect but execution just hangs or times out bc the pattern has catastrophic backtracking. makes me wonder if the model is memorizing common regex patterns from training data but not actually understanding the execution semantics
source for catastrophic backtracking claim? tested llama 3.3 70b q4 at 19.6k yesterday on email regex and didnt see execution time issues, just semantic incorrectness on edge cases
i test same thing yesterday at 19.4k with url regex and got catastrophic backtracking on ipv6 patterns - regex was syntactically correct but execution time was like 45 seconds on small test string. is same pattern across all complex regex past 19k?
tested same regex pattern at 19.1k yesterday and got the exact same catastrophic backtracking on ipv6. syntax looks completely valid but execution just hangs forever on test strings under 100 chars
ipv6 catastrophic backtracking is expected past 19k imo
tested this yesterday at 19.2k context with url validation regexes. basic patterns like `^https?://` work fine but edge cases like ipv6 addresses or internationalized domain names just break completely. does it fail on backreferences too or just character classes?
+1 hit this exact thing at 19.8k context. basic url patterns work but complex edge cases just fall apart
does it also break on ipv4 cidr notation or just ipv6