llama 3.3 70b q4 - at what context size does code comment quality degrade noticeably
I'm generating Python docstrings and inline comments for a refactoring tool and seeing quality drop somewhere around 18.5k-19k context. Comments start getting vague ('handles the data' instead of 'parses JSON response and extracts user_id field') or repeat information from the function signature. Has anyone measured this systematically? I know there's a lot of discussion about hard coherence thresholds where the model invents functions, but I'm interested in the softer degradation before that - where output is valid but noticeably lower quality. Running q4_k_m on a 4090, llama.cpp main branch from last week, temp 0.3.
hit this exact thing at 19.4k yesterday. code comments start inventing plausible package names that don't exist, then drift into describing features the actual code doesn't have. breaks way earlier than raw code generation
ok so need actual measurements tho. like did you run eval on the comments or just read them and decide they were bad
tested llama 3.3 70b q4_k_m on code comment generation with 8192 context, 16384 context, and 24576 context yesterday. coherence breaks at 19.2k tokens measured with perplexity spike from 12.4 to 31.8. code comments start inventing method names around 19.4k
perplexity spike is a solid metric but would also love to see semantic eval - like does the comment describe features that don't exist or just get vague. both are degradation but different failure modes
code comment quality is a terrible coherence metric because comments are inherently lower-stakes output. model can drift into plausible-sounding nonsense and you won't catch it unless you actually verify every claim. measure on executable code generation instead