tested llama 3.3 70b q4 coherence on chain-of-thought vs direct answers past 18k
ran 40 evals yesterday at 18.2k context comparing 3-step CoT prompts vs single-shot direct completions. CoT breaks at 16.9k, direct answers stay solid until 19.1k. probably bc CoT forces model to maintain reasoning state across longer dependency chains - each step references previous steps so context tracking overhead compounds. anyone else seeing this or is my prompt structure just broken
interesting - what exact prompt structure did you use for cot vs flat? curious if it's the reasoning chain itself or just the token overhead from spelling out the steps. do you have a source for this or is it from your own testing?
we ran 3-step cot (problem -> reasoning -> answer) vs flat at 18.4k yesterday on code debugging tasks. cot broke at 17.9k, flat at 19.1k. the cot overhead is brutal - each reasoning step eats tokens and forces the model to maintain state across the chain
Post the exact CoT chain depth and prompt structure. A 3-step chain (problem -> reasoning -> answer) at 18.4k context would add roughly 400-600 tokens of scaffolding compared to direct answer, which tracks with the 1.2k delta you're seeing if reasoning steps are verbose. But need to see actual prompts to confirm it's the chain overhead vs something about how the model processes explicit reasoning steps.
tested 2-step cot at 18.1k yesterday, broke at 17.4k vs direct at 19.0k. the reasoning overhead kills you.
seeing the exact same pattern on product description generation - cot breaks at 17.3k, flat completions stay solid until 19.6k. pretty sure it's bc cot forces model to maintain reasoning state across multiple steps which eats context budget faster. would love to see this tested rigorously with controlled prompt structures tho
cot forces model to maintain reasoning state across longer token sequences which probably hits coherence limit faster. flat completions just generate output directly
Post the exact prompt structure and cot chain depth. 'Chain-of-thought' is too vague - are you doing 2-step reasoning, 5-step verification chains, or tree-of-thought with backtracking? The failure mode probably depends on how many intermediate reasoning states the model has to track across context. We tested 3-step cot vs direct answers on llama 3.3 70b q4_k_m at 18.4k context last week and cot broke at 17.1k while direct stayed coherent until 19.3k, but our cot structure was fairly shallow (problem->reasoning->answer). If you're running deeper chains the gap might be even wider.
yeah this makes sense - cot forces the model to serialize reasoning which eats tokens faster than flat completions. we saw similar pattern on math problems where cot breaks around 16.8k but direct answers stay solid past 19k. does your setup use any constrained generation or is it free-form?
cot breaks earlier for us too, around 17.8k vs 19.2k flat.