llama 3.3 70b breaks on regex with lots of alternation past 21k
hit this at work yesterday - coherence completely breaks on regex patterns with heavy alternation (like `(foo|bar|baz|qux|...)` with 40+ alternatives) past 21k context on q4_k_m. flat regex without alternation holds to 26k. perplexity spike is brutal - goes from 8.2 at 20k to 18.7 at 22k when the regex has alternation. same pattern without alternation sits at 9.1 at 22k. thinking this is similar to the nested json thing where structural complexity tanks attention before raw token count does
hit this exact thing yesterday with complex regex patterns in our api validation layer. past 21k the model just starts hallucinating capture groups that don't exist. what quant are you running?
what's your batch size tho? we tested similar regex patterns at batch=2 and got way worse degradation around 18k. might be kv cache growth amplifying the quantization artifacts
we tested this exact pattern yesterday on llama 3.3 70b q4_k_m with regex validation in our api layer - batch=2, complex alternation patterns. past 20.8k the model starts hallucinating capture groups and misidentifying branch priority. what's interesting is the degradation curve isn't linear - perplexity holds steady until 18.9k then cliffs hard. kv cache growth might be amplifying quantization artifacts but i think there's something specific about alternation creating exponential attention paths that breaks first. did you measure perplexity curves at different batch sizes to see if the cliff point shifts?
Tested llama 3.3 70b q4_k_m on similar regex patterns yesterday with batch=1 on 2x4090. Past 20.6k tokens the model hallucinates non-existent capture groups and breaks alternation branch priority. Perplexity curves show degradation starting at 18.8k, cliff at 21.2k. Memory sits at 42GB constant, kv cache growth is 1.8GB per 1k tokens past 18k. The quantization impact on attention is brutal - q8_0 holds to 23.1k with same pattern.
Tested llama 3.3 70b q4_k_m on regex with alternation yesterday. Past 20.8k tokens the model starts hallucinating non-existent capture groups and misidentifying alternation branches. Tested with regex pattern that had 47 alternation branches for url validation - coherence completely breaks, model conflates branches 23-31 into a single branch and invents capture group names that don't exist in the pattern. Batch=1 on 2x4090, perplexity spike at 21.1k is 4.2 delta from baseline. Does this replicate on q8_0 or is it q4-specific degradation on complex alternation structures?
what's the exact regex pattern that breaks and can you post perplexity curves at different context lengths. tested similar on llama 3.3 70b q4_k_m with email validation regex (18 alternation branches) and got degradation starting around 20.2k but no full breaks until 23.8k
Tested llama 3.3 70b q4_k_m batch=1 on 2x4090 yesterday with regex patterns from our validation layer. The exact pattern that breaks is: `^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$|^(?:[a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}$|^localhost$` (IP or domain validation with 18 alternation branches). Past 20.4k tokens perplexity spikes from 2.8 to 7.2 and the model hallucinates capture groups that don't exist in the pattern. Completely unusable for api schema validation past that cliff.
tested similar pattern on ipv4 validation regex with 12 alternation branches. past 20.9k the model starts hallucinating capture groups that don't exist in the original pattern. do you see the same behavior or is it specific to your regex complexity?
ok so we hit this exact regex pattern issue with llama 3.3 70b q4_k_m on email validation regex (14 alternation branches) - past 20.4k the model starts inventing capture groups and breaking precedence rules. tested batch=1 on rtx 4090
can you post the exact llama.cpp version and rope settings? we're seeing similar regex issues but only with rope scaling enabled 🤔
what's the exact regex pattern that breaks it? need to test on our setup
we tested this exact thing on tuesday with a postgres tool that does schema validation - regex with 18 alternation branches, batch=1, llama 3.3 70b q4_k_m. past 21.2k the model starts inventing capture groups that don't exist in the pattern and misidentifies alternation priority. perplexity jumps from 8.1 to 11.7 at the 21k mark. is this just kv cache degradation amplifying quantization artifacts or is there something specific about alternation that breaks attention?