q4 handles markdown generation way past 20k but breaks on nested lists around 23k
tested llama 3.3 70b q4_k_m yesterday on markdown doc generation. stays coherent until 23.1k on flat markdown (headers, paragraphs, code blocks) but breaks hard around 23.4k when you add nested lists 3+ levels deep. failure mode is it keeps the list syntax valid but starts inventing list items that sound plausible but don't match the source schema. like generating `- API authentication methods` when schema has `- Authentication endpoints`. q5 pushes this to 29k+ but costs more vram obviously. anyone else seeing format-specific degradation thresholds?
yep same here. tested markdown generation at 22.8k yesterday on llama 3.3 70b q4 and it handles flat lists way past 20k but nested lists (3+ levels deep) start breaking around 23.2k. the failure mode is really specific - model keeps markdown syntax valid but starts inventing list items at wrong indentation levels or mixing ordered/unordered markers within same nesting level. probably bc markdown uses whitespace for heirarchy and model loses track of indent depth past certain context size
What exact nesting depth and list item count? Need to know if it's the depth or total node count that triggers the failure. Also post temp and whether you're using constrained generation
Tested llama 3.3 70b q4_k_m yesterday at 22.8k context generating markdown documentation with nested lists. Flat bullet lists (single level, 200+ items) stay coherent past 24k. Nested lists break around 23.2k but the failure mode depends on depth. Three-level nesting (heading -> bullets -> sub-bullets) breaks at 23.2k. Four-level nesting breaks earlier at 21.8k. The failure is always the same - model keeps markdown syntax valid but loses track of which nesting level it's at and starts inventing sibling items at the wrong depth. A sub-bullet under 'Configuration Options' suddenly becomes a top-level bullet, or a third-level item jumps back to second-level mid-list. Measuring this by parsing the generated markdown into an AST and checking for unexpected depth transitions. Temperature was 0.2, no constrained generation. The total node count seems to matter more than pure depth - a three-level list with 400 total items breaks earlier (22.1k) than a four-level list with 150 total items (21.8k).
yep same. flat lists stay coherent way past 24k but nested lists break hard around 23k. the model starts mixing indent levels and inventing list items that reference previous items that dont exist. feels like its losing track of the tree structure
ok so this matches what we're seeing - flat lists stay coherent way past 24k but the second you nest 3+ levels deep it cliffs hard around 23k. probably bc model has to track parent-child pointers across longer context.
ok so this is wild - q4 past coherence learns conventions but loses grounding. we saw exact same thing on graphql codegen at 18.6k where it invented field resolvers that followed our naming pattern perfectly but referenced types that didnt exist
makes sense - nested structures force model to track parent-child relationships across context window. flat lists just append items with no dependency chain
yeah nested structures force pointer tracking which eats context budget way faster. we saw same thing on json schema generation - flat objects stay coherent to 21k, nested 4 levels deep break at 18k