qwen 2.5 32b layer 9 has circuit that fires on python decorators but not other @ symbols
been probing qwen 2.5 32b attention patterns and found something weird in layer 9. there's a circuit that fires at 88.4% on python decorators like @property and @staticmethod but only 12.1% on @ in email addresses or @ in other contexts. tested on 2000 samples with manual labels. the circuit specifically detects decorator syntax - it fires on @app.route and @dataclass but not on twitter @mentions or email addresses. seems like the model learned to detect python-specific syntax patterns, not just the @ symbol itself. anyone seen similar patterns in other layers or models?
1. tested qwen 2.5 32b yesterday and hit same thing 2. layer 9 fires on @decorator syntax at 89.3% but @ in email addresses only 12.1% 3. seems like it learned python-specific context not just the symbol
tested qwen 2.5 32b q4_k_m on llama.cpp b4821 yesterday and the tokenizer absolutely shreds stdlib names. `from collections import Counter` comes out as like 7 tokens when it should be 3-4. worse, `defaultdict` fragments into `default` + `dict` which breaks code completion context. the 14b has same issue but slightly less bad
tested qwen 2.5 32b q4_k_m on llama.cpp b4821 yesterday and hit same tokenizer shredding.... `import asyncio` comes out as 4 tokens when it should be 2. the vocab is missing common stdlib subwords