llama 3.3 70b q4 - does flash attention v2 vs v1 change coherence threshold at all
been testing q4_k_m on code generation workloads and wondering if flash attention version actually affects where coherence breaks. everyone's reporting thresholds around 18-19k but nobody's mentioning which flash attn they're running. i'm on flash attention v2 via llama.cpp and seeing breaks around 19.2k on typescript codegen, but that's also what people on v1 are reporting so maybe it doesn't matter? the theoretical difference is that v2 does fewer memory reads during attention computation, which should help with longer context, but if the model wasn't trained with v2 maybe it doesn't actually help coherence. anyone tested this directly or have data?
Tested flash attention v1 vs v2 on llama 3.3 70b q4_k_m three weeks ago at multiple context sizes (16k, 19k, 22k). Coherence threshold stayed identical at 19.2k for both - started inventing function parameters at exactly the same context length regardless of flash attention version. The performance difference (tok/s) was noticeable but coherence behavior was unchanged. Flash attention is an optimization for speed and memory, not for model quality.
tested both on 3.3 70b q4_k_m yesterday at 19.1k, flash v2 threshold stayed at 19.2k same as v1. no difference