qwen 2.5 32b tokenizer fragments common pytorch module names badly
tested qwen 2.5 32b tokenizer vocab yesterday and it fragments torch.nn.functional, torch.optim, and torchvision.transforms into 5-7 tokens each. the base llama tokenizer does these in 3-4 tokens. this tanks code completion quality for pytorch-heavy codebases because the model has to predict way more tokens for basic imports
hit this exact thing yesterday.... qwen 2.5 32b fragments torch.nn.Module into ['torch', '.', 'n', 'n', '.', 'Module'] which is completely broken. the tokenizer vocab is missing so many common pytorch names
Wait, this is a tokenizer issue? I thought the tokenizer was supposed to be trained on code specifically for qwen 2.5. How does it miss torch.nn.Module when that's like the most common pytorch import? Does this affect fine-tuning too, or is it just inference where you notice the fragmentation impact?