llama 3.3 70b q4 invents plausible-looking imports that don't exist past 18.6k
Past roughly 18.6k context, llama 3.3 70b q4_k_m starts generating import statements that follow the correct syntax and naming conventions but reference modules that don't exist in the actual codebase or standard library. Example from a FastAPI project: `from fastapi.middleware.rate_limit import RateLimitMiddleware` - sounds completely plausible, follows FastAPI's module structure, but doesn't exist. The actual rate limiting middleware lives in third-party packages like slowapi. Another: `from typing import OrderedSet` - follows typing module conventions, sounds like it should exist alongside OrderedDict, but typing only has Set and no ordered variant. The invented imports are scary accurate in terms of style and naming patterns, which makes them harder to catch in code review. They fail at runtime but pass the "does this look right" test. Tested on llama.cpp a4c3b45 with free-form generation (no constraints). Measuring coherence by tracking import statements that fail resolution against known package manifests. Post your llama.cpp version and exact context size if you've seen this. Need to know if 18.6k is consistent or if it varies by setup.
what quant exactly. getting different results on q4_k_m vs q4_k_s at same context
what quant exactly.... seeing different coherence thresholds on q4_k_m vs q4_k_s at basically the same context window. would love to know if the hallucinated imports follow naming conventions from real packages or if they're completely random
1. hallucinated imports usually follow real naming conventions from the stdlib or popular packages 2. post the exact quant and context size where this starts happening