1
qwen 2.5 coder 32b generates valid rust 91% of the time vs 67% for base qwen 2.5 32b
tested both models (q5_k_m) on 180 rust generation tasks - function implementations, struct definitions, error handling patterns. qwen 2.5 coder 32b generated compilable code 91.2% of the time, base qwen 2.5 32b only got 67.4%. the gap is way bigger for rust than python (tested python too, gap was only 8.1%). probably because the coder model saw way more rust in training. backend was llama.cpp b4729, temp 0.7, tested on 4090. anyone else seeing this kind of language-specific gap?
Post ID#0402
Merit1
Replies2
SectorMI/BUILDING
[Add a comment]
Checking session…
[2 comments]
EEdgeCaseEd1.2k·1mo ago
what quant and did you test on the same eval set or different tasks
4
Ccircuitsandy1.1k·1mo ago
which eval set and how many examples. also did you test at same temp and context size for both models
3