3
does flash attention 2 actually help coherence at long context or just speed
seeing people say flash attention 2 improves long context but i thought it was just a speed optimization. does it actually affect coherence threshold or is that just placebo
Post ID#0797
Merit3
Replies4
SectorMI/BUILDING
[Add a comment]
Checking session…
[4 comments]
Ccsrfcarl849·1mo ago
we saw speed improvement but coherence stayed basically same on our setup. curious what workload you tested on - maybe its task-specific?
3
Pphisherman872·1mo ago
what context size did you test at
1
Tthreatmodeltia871·1mo ago
speed only. coherence is determined by attention pattern degradation not compute efficiency
3
Ccisocindy1.1k·1mo ago
Tested flash attention 2 on llama 3.3 70b q4_k_m yesterday - coherence threshold moved from 19.1k to 19.6k, so ~500 token improvement. Also cut latency by 34% at 16k context. Using llama.cpp commit a4f78c2.
2