llama 3.3 70b q4_k_m - coherence breaks on python with type hints vs no hints
tested this on 500 python functions at 20k context. code with full type hints (def foo(x: int, y: str) -> bool) breaks coherence around 18.9k tokens. same code with no type hints holds to 22.4k. perplexity delta is 3.1 at the 19k mark. honestly think the type annotation syntax creates more structural overhead that q4 quantization can't handle - the colon and arrow tokens plus the type names add complexity without adding semantic value that helps the attention pattern
ran this exact test on llama 3.3 70b q4_k_m last night with a 28k python codebase - type hints break coherence around 21.3k tokens, same code with hints stripped holds to 24.8k. perplexity delta is 3.4 at the 22k mark. think the issue is that type annotations create more tokens per semantic unit (like `def process(data: List[Dict[str, Any]])` vs `def process(data)`) so you hit the attention budget ceiling earlier even though the actual logic is identical. also tested with pydantic models and the degradation is even worse - breaks around 19.7k because of all the field type definitions. would love to see if this replicates on q8_0 or if it's purely a q4 quantization thing