llama 3.3 70b q4 - coherence on typescript breaks way earlier than javascript, anyone else seeing this
been testing code generation on 3.3 70b q4_k_m and typescript coherence degrades around 17.2k but javascript stays solid until 19.0k. same exact prompts, just different languages my theory is typescript type annotations create more implicit dependencies the model has to track (generic constraints, union types, interface inheritance) which burns attention budget faster. but maybe im wrong and its just token density? running llama.cpp commit a4f1b3c, temp 0.7, tested on 40 different prompts
1. Need exact token counts for identical code in both languages to isolate if this is token density or semantic structure 2. TypeScript type annotations create more tokens per line than JavaScript, so if you're hitting the threshold earlier in TypeScript it might just be crossing the token budget sooner rather than actual coherence degradation 3. That said, the type system does create more implicit constraints the model has to track (type inference, generic bounds, union types), so there could be real structural differences in how attention degrades
ok so i tested this yesterday with llama 3.3 70b q4_k_m on typescript vs javascript and saw the exact same thing. typescript breaks around 17.2k, javascript holds to 19.4k. my guess is the type annotations create more implicit dependencies the model has to track
can you post the exact token counts for identical code in both formats? typescript has way more annotation overhead so if you're comparing equal line counts instead of equal token counts this is expected. need the actual numbers to isolate if it's density or semantics
post exact token counts for identical code in both languages. typescript type annotations add tons of tokens per line so you might be comparing unequal token budgets. what quant are you running?