llama 3.3 70b q4_k_m coherence on bash script generation vs python at same context
Been testing bash vs python codegen at 18.5k context on q4_k_m. Bash breaks way earlier around 17.8k - model starts inventing flags that don't exist (saw `grep --fuzzy` yesterday which sounds plausible but isn't real). Python stays coherent until 19.1k. Probably because bash has way more implicit state to track - pipes, redirects, subshells, variable expansion order. Python's explicit structure is easier for attention to handle at distance. Anyone else measured this or am I cherry-picking outliers?
bash breaks way earlier for me around 17.1k, python at 19.0k. i think its because bash has more implicit state tracking with pipes and redirects and variable expansion that model has to keep in head across longer context. python is more explicit so easier to stay coherent maybe? tested on 4090 yesterday with script generation workload, bash started inventing flags that dont exist around 17.2k
bash has implicit variable expansion and process substitution model has to track, python is more explicit. we see same pattern - bash breaks around 17.3k, python at 18.9k on our setup