tokenizer artifacts in llama 3.1 are breaking my intent classifier in weird ways
spent the morning debugging why our intent classifier (llama 3.1 8b q4_k_m) suddenly started failing on inputs with em dashes and certain unicode quotes. turns out the tokenizer is splitting them into 3-4 tokens each and the model just... loses the thread. specifics: "user wants to update their profile — including email" gets tokenized as ["user", "wants", "to", "update", "their", "profile", "—", "including", "email"] but the em dash becomes tokens [2347, 29892, 322] and the model treats it like a hard sentence boundary. accuracy dropped from 89% to 71% on inputs with this pattern. iirc this is a known thing with llama tokenizers but i can't find any good writeups on workarounds. preprocessing to replace em dashes with " - " fixes it but feels cursed. anyone else hit this or am i just bad at prompting?
1. check if your examples include tokens that cross BPE boundaries weirdly 2. llama 3.1 has known issues with certain unicode ranges breaking intent
check tokenizer version, llama 3.1 had a patch in sept
Check also if you have unicode normalization issue. We had same problem with llama 3.1 in september, some intents were getting split wrong because of NFD vs NFC encoding in training data)
honestly tokenizer artifacts are the least surprising thing about llama 3.1 at this point. we spent three days debugging what turned out to be a BPE boundary issue in production and I still don't really understand tokenizers if I'm being honest
tokenizer artifacts breaking intent classifiers is the least surprising thing i've read today. post the training data encoding
ok so also check if you're hitting the tiktoken edge case where certain unicode gets split weird and breaks intent parsing. we had this exact problem with llama 3.1 and it turned out our training data had zero-width characters that survived tokenization
tokenizer artifacts breaking classifiers is the least surprising thing i've read today. post your training data and the actual tokenization output so we can see what's being split wrong