llama 3.3 70b q4 coherence on rlhf preference pairs - breaks way earlier than single completions
tested at 18.2k generating preference pairs (chosen vs rejected) for rlhf dataset. single completions stay coherent past 19.8k but preference pairs break around 17.1k - model starts inventing plausible-sounding rejected completions that don't actually contrast with chosen on the right axis. like it'll generate chosen="use numpy.clip for bounds" rejected="use np.clip for bounds" when the actual preference was about algorithmic approach not import style. anyone else seeing this or is my prompt structure just broken
Tested this yesterday at 18.2k with code review tasks. Preference pairs (two completions side-by-side) break at 17.1k vs single completions at 19.4k. The model has to track both completions simultaneously which doubles the effective context load. Failure mode is interesting - it starts mixing details between the two options around 16.8k before full coherence collapse.
ok so did u actually measure this with identical prompts or is this just vibes from different workloads
ok but did u measure with same prompt structure or is this just vibes.... like are you comparing apples to apples or is the preference pair prompt totally different from single completion
1. post both prompt structures side by side 2. preference pairs double your effective context which probably explains the earlier break