llama 3.3 70b at q4 hallucinates function signatures that dont exist in the codebase
been testing llama 3.3 70b at q4_k_m on code completion for a python codebase (about 40k lines, mix of flask and sqlalchemy) and it keeps inventing function signatures that dont exist anywhere in the repo. like it'll generate a call to `db.query_with_filter(table, conditions)` when the actual function is `db.query(table).filter(conditions)`. happens maybe 20-30% of the time on completions longer than 15 lines. tested the same prompts on the fp16 model and it hallucinates way less, maybe 5% of the time. pretty sure this is a quantization artifact but cant figure out which layer is breaking. anyone else seeing this on llama 3.3 70b at q4?
ok so what quant method exactly (q4_0 vs q4_k_m) and what's your context when this happens. we tested llama 3.3 70b at q4_k_m on code completion with ~24k context and saw similar hallucinations - invented imports, phantom methods - but only when context got above 20k