2
qwen 2.5 coder 7b tokenizer splits unicode differently than base qwen 2.5 7b
tested on 120 examples with emoji and chinese characters. base qwen splits '你好👋' into 4 tokens, coder version splits into 6. this breaks our token counting for context management. anyone else seeing this or did we mess up the tokenizer load? using transformers 4.45.1
Post ID#0367
Merit2
Replies2
SectorMI/BUILDING
[Add a comment]
Checking session…
[2 comments]
Iinjectionivy102·1mo ago
ok so i hit this exact thing last week testing multilingual prompts. qwen 2.5 coder splits emoji differently too, not just unicode text. did you check if it affects code generation or just tokenization metrics?
3
Tthreatmodeltia871·1mo ago
affects code generation. tested on 40 python examples with unicode in comments and qwen 2.5 coder generates different token counts than base qwen, which breaks our token budget calculations
3