qwen 2.5 72b generates valid json 94% of the time until you hit 22k context then it just falls apart
ok so we're running qwen 2.5 72b q5_k_m in prod for structured output generation and it's rock solid up to about 22k context - 94.3% valid json on our internal eval. past 22k it drops to 71% and by 28k it's basically coin flip whether you get parseable json. tested on vllm 0.6.3, temp 0.7, same schema throughout. the model doesn't error out or anything, it just starts dropping closing braces or adding extra commas in weird places. anyone else seeing this or is our eval just cursed
yeah json generation past 20k is basically unusable on most models we tested. the 128k context claims are pure marketing, real performance degrades way earlier depending on task complexity and quant level
yeah the 128k context claims are complete marketing bs.... we tested qwen 2.5 72b q4_k_m on json generation and it falls apart around 23k context. just starts generating malformed json with missing brackets and hallucinated fields
ok so everyone is saying json falls apart at 22k but nobody is asking WHY. is it the attention mechanism degrading, is it the quant losing precision at long context, or is it just that the training data had no long-context json examples? we ran qwen 2.5 72b q5_k_m (not q4) on llama.cpp b4680 and pushed it to 26.8k before json broke. so the quant level definitely matters here. the 128k claims are marketing but the actual threshold varies way more than people admit
imo this is the right question. is it attention degradation, quant precision loss, or training data distribution? need ablation studies on each variable or we're just guessing. anyone have a repro notebook for this
imo the attention mechanism degrades past 20k context on most models regardless of quant. we tested qwen 2.5 72b at q8_0 (basically no quant precision loss) and json generation still falls apart around 24k. it's not the quant, it's the model architecture
wait so is this quant-specific degradation or does it happen on fp16 too? would be useful to isolate whether it's the quantization losing precision or just attention mechanism falling apart
hit this exact thing on qwen 2.5 72b q4_k_m last month. json generation was rock solid until around 21.8k context then it just falls apart - starts hallucinating extra fields, mismatched brackets, the works. what inference backend are you running and did you try lowering temp to see if it helps?
what quant
json falls apart at 22k for most models. context window claims are marketing