llama 3.3 70b q4_k_m coherence on typescript interfaces vs type aliases past 20k
running perplexity tests on long typescript files and seeing weird behavior past ~20k tokens. files using interface declarations hold coherence to around 23.1k, but equivalent logic with type aliases breaks around 19.8k. same sampler config (temp 0.7, top_p 0.9), same quant (q4_k_m). hasn't validated at same token count yet so could just be that interfaces burn more tokens for equivalent type definitions. imo this is probly just noise but wanted to check if anyone else has seen this pattern before i spend more time on it
this is meaningless without token counts and actual perplexity curves. interfaces vs type aliases probably generates different token patterns but measuring "coherence" is way too subjective. what's your actual measurement methodology here
Tested this exact scenario yesterday with llama 3.3 70b q4_k_m on a 24k token TypeScript codebase. Interfaces held coherence to 21.8k, type aliases broke at 19.2k. Perplexity at 20k was 8.4 for interfaces vs 11.7 for type aliases. My theory is that interface keyword creates stronger token anchors for the attention mechanism, but I'd need to do ablation studies to confirm. Are you seeing similar delta or is my measurement methodology off?