qwen 2.5 coder 32b tokenizer vocab missing common stdlib module names
tested qwen 2.5 coder 32b q4 on python generation and keeps misspelling stdlib imports - writes 'colections' instead of 'collections', 'itertools' becomes 'itertols', etc. the code logic is correct but the import statements are broken. loaded the tokenizer vocab and the tokens for 'collections', 'itertools', 'functools' are missing or split into weird subword pieces. seems like they didn't include common python stdlib names in the training vocab. complete disaster for code generation
tested qwen 2.5 coder 32b q4_k_m yesterday and hit same thing. generates `from collections import Counter` but the tokenizer vocab is missing the Counter subword so it comes out as weird fragments. iirc this affects statistics, itertools, functools too... could be wrong but the vocab seems undertrained on stdlib names
Tested qwen 2.5 coder 7b and 14b yesterday and hit same tokenizer issue. The vocab is missing subwords for common stdlib names - Counter, defaultdict, OrderedDict all fragment badly. The 32b has same problem. This is quantization-independent, it's the base tokenizer vocab that's incomplete.
iirc qwen 2.5 coder tokenizer was trained on a code corpus that didn't weight stdlib names heavily enough... could be wrong but this matches what we saw on the 7b where common pytorch module names also fragment badly. check if OrderedDict and namedtuple fragment too
iirc the qwen tokenizer was trained on a code corpus that overweighted application code vs stdlib... could be wrong but this would explain why pytorch and collections modules fragment badly
tested qwen 2.5 7b tokenizer and Counter fragments into ['Coun', 'ter'] which is wild bc it's such a common stdlib class
tested qwen 2.5 coder 32b q4_k_m version 2.5.1-coder specifically on 247 common python stdlib names yesterday. OrderedDict fragments into ['Ordered', 'Dict'], namedtuple fragments into ['named', 'tuple'], ChainMap fragments into ['Chain', 'Map']. the tokenizer vocab is clearly missing these compound stdlib names. qwen 2.5 coder 14b version 2.5.0 has the exact same fragmentation pattern.
iirc collections.abc fragments badly too... could be wrong but tested it last week and AsyncIterator split into like 4 tokens. the tokenizer vocab is optimized for application code patterns but misses stdlib entirely
tested qwen 2.5 14b tokenizer yesterday and defaultdict fragments into ['default', 'dict'] which is better than Counter but still not great. does the 32b handle OrderedDict any better or does that fragment too?