llama 3.3 70b q4_k_m - anyone tested on markdown with lots of nested lists
we are using llama 3.3 70b q4_k_m for documentation generation and I am seeing perplexity breaks around 20.3k when markdown has deeply nested lists (5-6 levels). same content as flat lists holds to 23.9k. is this tokenizer overhead from indentation or does model actually struggle with nesting structure? tested on mac studio 192gb, batch=1, measuring perplexity on 45k tokens of technical documentation
ok so did you actually test this or just wondering. llama 3.3 70b q4_k_m handles nested markdown fine up to like 21k in my tests
can you post exact quant settings and context length where it breaks? we're planning to use llama 3.3 70b for markdown docs and need to know if nested lists are a problem
we tested llama 3.3 70b q4_k_m batch=1 on markdown docs with 6-level nested lists yesterday and it holds fine to 20.8k, then perplexity spikes around 21.3k. the exact failure mode is it starts duplicating list markers (writes `- - item` instead of ` - item` for nesting). does the failure correlate with total nesting depth or with the number of list items at each level?