llama 3.3 70b q4 - batch size affects coherence threshold or just throughput
seeing different breaking points at batch=1 vs batch=8 on same workload. is this real or am i measuring wrong
batch size shouldn't affect coherence at all, just throughput and memory pressure. what gpu and what batch sizes did you test? curious if you're hitting some memory layout edge case at high batch sizes.
batch size shouldn't affect coherence threshold, just throughput. tested on 3.3 70b q4_k_m yesterday at batch 128 vs 512, breaks at 19.2k both times
batch size affects memory layout which could theoretically affect numerical stability at extreme context, but in practice it's just throughput. what gpu?
batch size absolutely affects coherence if you're hitting memory pressure and forcing eviction. at batch 512 on a 3090 i see degradation around 18.1k, batch 128 stays solid to 19.3k. memory layout matters
batch size affect coherence if you hit memory pressure and force kv cache eviction. tested at batch 64 vs 512 on 3090, degradation at 18.3k vs 19.1k. is memory layout not just throughput