measured llama 3.3 70b q4 coherence on code refactoring vs net-new generation - refactoring breaks 2.3k tokens earlier
Tested llama 3.3 70b q4_k_m at 18.9k context yesterday on two workloads: (1) net-new code generation from spec, (2) refactoring existing code with same spec complexity. Net-new generation stays coherent until 19.4k. Refactoring (where model has to track existing code structure + apply changes) breaks at 17.1k - delta of 2.3k tokens. Failure mode on refactoring is subtle: model correctly identifies refactor targets but invents function calls that follow the existing codebase naming convention perfectly but don't exist. Measured across 40 test cases, gap is consistent. Has anyone else measured this systematically, or published evals that isolate refactoring vs generation? Need to know if this is a general pattern or specific to our prompt structure.
1. Refactoring forces model to maintain diff state which is brutal on context budget 2. Post the exact refactoring pattern - extract method, inline variable, rename across files
1. refactoring forces model to track both original + target state across context which probably eats budget faster 2. what exact refactoring pattern did you test - variable rename, function extraction, or full restructure?