llama 3.3 70b q4_k_m - tokenizer handles unicode normalization differently than 3.1, breaks at lower context on text with composed characters
tested same prompts (mix of english and text with diacritics) on llama 3.1 70b vs 3.3 70b q4_k_m. 3.3 breaks coherence ~2.1k tokens earlier when input has composed unicode characters (é vs e + combining accent). normalized to NFC before tokenization and gap disappeared. is this documented anywhere?
hit this exact bug with llama 3.3 70b q4_k_m on french documentation two weeks ago. tokenizer normalized é to e+combining-accent differently than 3.1 (U+00E9 vs U+0065+U+0301) which broke token alignment and coherence dropped hard around 18.2k instead of expected 22k. tested with identical sampler config (temp 0.7, top_p 0.9). this should have been caught in regression testing before release
hit this exact issue with accented characters in french api docs.... tokenizer normalized é differently than 3.1 and coherence broke around 18k instead of the expected 22k.... really frustrating because the docs say "improved unicode handling" but it's actually worse for non-ascii text....
hit this exact same issue with our french api docs two weeks ago.... tokenizer normalized é to e+combining-accent in 3.3 but kept it as single-byte in 3.1.... completely broke our context measurements.... ended up having to re-benchmark everything with nfc normalization forced....
ok so this is a tokenizer regression not a model architecture thing? that seems like something that should have been caught in testing
It's both imo. Tokenizer regression that should've been caught in testing AND a model architecture thing - the attention pattern at layer 18-22 seems way more sensitive to token boundary shifts in 3.3 than 3.1. Could be wrong but I think the unicode normalization change exposed an existing fragility.