deepseek v3 generates valid code but fails to execute it in the next turn
imo this is weird behavior - deepseek v3 q4_k_m generates syntactically correct python code in one turn, but when you ask it to execute or debug the code in the next turn it hallucinates errors that don't exist. tested on llama.cpp b4729 with temp 0.7 across 20 different code generation tasks. anyone else seeing this or is my setup broken
need way more context. does it fail at parsing the generated code, does it throw a runtime error, or does the code execute but produce wrong output? also are you using the same context window across turns or does it reset
also check if they're running same context length. retrieval quality tanks when context size changes even if chunk size stays same
1. what does 'fails to execute' mean - syntax error, runtime error, or logical error 2. are you using the same system prompt across turns or does it change tested deepseek v3 q4_k_m on multi-turn code gen last week and it forgets variable names from previous turns around turn 4-5
also need to know if you're using tool calling or just code generation. if tool calling, does the model generate the code correctly but then fail to parse its own output when trying to execute it
this sounds like it could be a multi-turn context management issue. when you say 'fails to execute in the next turn', does the model correctly reference the code it generated in turn 1, or does it hallucinate that the code doesn't exist or was never written? also are you using tool calling mode or just prompting the model to generate and execute code in separate turns?
hit this exact thing on deepseek v3 q4 yesterday. generated valid python in turn 1 (a function to parse json), then in turn 2 when i asked it to run the function on test data, it completely hallucinated that the function signature was different and passed wrong args. the code it generated in turn 1 was syntactically perfect, but in turn 2 it lost track of what it wrote and invented a different version of the same function with different param names. feels like context management issue
need way more detail here. does 'fails to execute' mean syntax error, runtime error, or the code runs but produces wrong output. also are you using multi-turn with tool calling or just generating code blocks
hit this last week testing deepseek v3 q4_k_m on multi-turn code gen. the model generates syntactically valid python in turn 1, then in turn 2 when you ask it to run the code, it hallucinates that it already executed and makes up fake output. turns out the model is trained on conversation logs where code execution happened, but the actual execution was done by a separate system. so it learned to predict the conversational pattern ("here's code" -> "i ran it, here's output") without learning to distinguish between actual execution and simulated execution. tested with explicit system prompt saying "you cannot execute code, only generate it" and it still hallucinates execution results about 40% of the time