llama 3.3 70b function calling breaks past 40k context but docs say 128k
shipping a feature that uses llama 3.3 70b for tool calling with long context (legal doc analysis, needs 60-80k context). the model is supposed to handle 128k context but function calling accuracy falls off a cliff past 40k. tested on 50 examples with context ranging from 20k to 80k tokens. at 20k context the model picks correct tool 91% of time, at 40k it drops to 78%, at 60k it's down to 52%. past 80k it basically guesses randomly. the failure mode is it picks tools that are semantically related but wrong, or it hallucinates parameters that aren't in the context. feels like the attention mechanism is breaking down for tool calling specifically because summarization tasks work fine at 80k. anyone else seeing this or did we misconfigure something?
the 128k claim is always marketing. real context handling depends on quantization, inference backend, and batch size. we run llama 3.3 70b in prod and function calling reliability drops hard past 35k regardless of what the docs say
tested llama 3.3 70b q4_k_m on vllm 0.6.3 and function calling degraded around 41.2k context, not 35k. which version of vllm are you running and what batch size?
1. which llm.cpp version 2. does it fail silently or throw errors tested llama 3.3 70b q4_k_m on similar context range last week and function calling broke around 38k for me, way below the claimed 128k
hit this exact thing. silent fails around 38k, no error just garbage outputs
what quant and inference backend? also does it fail silently (returns garbage) or throw actual errors. tested llama 3.3 70b q5_k_m on llama.cpp (version b4603) and function calling degraded around 42k but didn't break completely until 58k