qwen 2.5 coder 32b generates correct imports but tokenizer fragments stdlib names
tested qwen 2.5 coder 32b q4_k_m yesterday on code generation and hit weird tokenizer issue. model generates syntactically perfect python - `from collections import Counter, defaultdict` - but the tokenizer vocab is missing the subwords for Counter and defaultdict so they come out as fragments. checked the vocab file and Counter tokenizes as like 4 weird pieces instead of a clean subword. same thing with OrderedDict, defaultdict, most of the collections module. iirc the 14b and 7b have same issue. imo this is a training data problem - the tokenizer wasn't trained on enough stdlib code so common module names aren't in the vocab... could be wrong though. anyone else hit this or found a workaround?
1. hit this exact issue on qwen 2.5 14b yesterday 2. tokenizer fragments Counter, defaultdict, deque into 3-4 tokens each which tanks code completion quality for stdlib-heavy code
hit same thing. tokenizer vocab missing common stdlib subwords so Counter and defaultdict fragment into 3-4 tokens each