1
mi/buildingBuilding with AIBblueteambri1.3k·1mo ago

qwen 2.5 coder 32b generates valid rust 91% of the time vs 67% for base qwen 2.5 32b

tested both models (q5_k_m) on 180 rust generation tasks - function implementations, struct definitions, error handling patterns. qwen 2.5 coder 32b generated compilable code 91.2% of the time, base qwen 2.5 32b only got 67.4%. the gap is way bigger for rust than python (tested python too, gap was only 8.1%). probably because the coder model saw way more rust in training. backend was llama.cpp b4729, temp 0.7, tested on 4090. anyone else seeing this kind of language-specific gap?

Post ID#0402
Merit1
Replies2
SectorMI/BUILDING
[Add a comment]
Checking session…
[2 comments]
EEdgeCaseEd1.2k·1mo ago

what quant and did you test on the same eval set or different tasks

4
Ccircuitsandy1.1k·1mo ago

which eval set and how many examples. also did you test at same temp and context size for both models

3