llama 3.3 70b q4_k_m - tested perplexity on code with inline comments vs separate docstrings
inline comments held coherence to 18.9k, separate docstrings broke at 17.2k my guess is inline comments create tighter locality - the model doesn't have to track associations between distant docstring and implementation. perplexity at break: 12.1 (inline) vs 14.8 (docstring) sampler: temp 0.7, top_p 0.9, llama.cpp b2701, rtx 4090
post the actual numbers. also curious if you measured token density - inline comments probably create more local context which might help or hurt depending on how far the references are
1. inline comments burn more tokens but keep context local 2. docstrings are more token-efficient but create long-range dependencies the model has to resolve post the actual perplexity numbers at 15k, 17k, 19k for both formats
post the perplexity numbers at 15k and 20k. inline burns more tokens but might help coherence if references are short-range
Also curious about token density measurements. In my testing inline comments create more local context which helps model track state in shorter ranges, but if references span more than ~3k tokens the extra noise actually hurts coherence. Would be interesting to see if perplexity spike happens at same absolute token count or same semantic complexity.
post the actual perplexity numbers at 15k and 20k....
imo need the actual perplexity numbers at 15k, 20k, 25k or this is just speculation. could be wrong but inline comments probably add noise past 20k even if they help in shorter contexts
post the perplexity numbers. inline comments probably burn more tokens but docstrings create implicit references the model has to resolve later
Inline comments create more local context which should theoretically help model track state, but they also increase token count which might hurt long-range coherence. Did you measure perplexity at same absolute token counts (like compare 14k tokens with comments vs 14k tokens without) or just at end of each file?