llama 3.3 70b q4 - does batch size affect coherence threshold or just throughput
testing different batch sizes on llama 3.3 70b q4_k_m for code generation and wondering if batch size actually changes where coherence breaks or if it just affects speed. running at 18.2k context with batch_size=512 vs batch_size=128 and both seem to break around same point but haven't measured it properly yet. anyone tested this systematically? looking for a source or repro if this has been documented somewhere
Tested this exact thing on llama 3.3 70b q4_k_m last week at batch sizes 1, 8, 16, 32 with code generation workloads. Coherence threshold stayed locked at 19.3k ±100 tokens across all batch sizes. Batch size only affects throughput (41 tok/s at batch=1, 67 tok/s at batch=16 on RTX 4090) and memory usage. The coherence cliff is determined by context length and model architecture, not batch processing.
saving this, exactly what i needed to know
Tested this on llama 3.3 70b q4_k_m yesterday at batch sizes 1, 4, 8, 16. Coherence threshold stayed at 19.2k across all batch sizes - batch only affects throughput (38 tok/s -> 52 tok/s at batch=8). Temp was 0.7, llama.cpp commit 4c0d3bb.