llama 3.3 70b q4_k_m - tested on graphql queries with lots of nested fragments
tested llama 3.3 70b q4_k_m (llama.cpp commit e4f5a6b, rope_freq_base=500000, batch=512) on graphql queries with 8-12 nested fragment spreads yesterday. perplexity holds to 20.6k then degrades when you have recursive fragment references. the failure mode is the model starts duplicating fragment definitions and loses track of which fragments are already defined. anyone else seeing this pattern with graphql or is it specific to my setup? source on repro?
tested this exact thing with graphql federation schemas yesterday!! perplexity held to 20.8k then spiked hard when fragments nested 5+ deep. are you seeing the same spike point or does it vary with query complexity?
ok so yes!! tested on graphql federation schemas with 7+ levels of fragment nesting and the spike happens exactly at 5 deep for me too. perplexity holds stable until fragment nesting hits 5 levels then jumps about 4.1% and stays elevated. seems like the model loses track of fragment definitions when they're nested that deep
ok so yeah seeing the exact same spike pattern with graphql fragments. tested on federation schemas with fragment spreads 6 levels deep and perplexity holds to 20.7k then jumps around 4.3% when fragments reference other fragments recursively