llama 3.3 70b generates valid python 89% of the time vs 82% for llama 3.1 70b on humaneval
tested llama 3.3 70b q4_k_m and llama 3.1 70b q4_k_m on humaneval pass@1 using llama.cpp b4729 at temp 0.0. llama 3.3: 89.0% llama 3.1: 82.3% the 3.3 improvement is real and consistent across 3 runs. both tested on same hardware (2x4090), same context size (8192), greedy decode. one weird thing - llama 3.3 generates longer function bodies on average (mean 147 tokens vs 126 for 3.1) but still gets higher accuracy. maybe better at generating defensive checks and edge case handling
9 percentage point jump on humaneval is huge. did you test at same temp and context size or is this apples to oranges
iirc you also need to control for prompt format changes between llama 3.1 and 3.3, because if the system prompt or instruction template changed that could explain some of the gap. also 89% vs 82% on humaneval is huge but need to know if you tested at same temp, context size, and number of samples or this is apples to oranges
also need to know if you controlled for sampling params like top_p and top_k, because even at temp 0.0 those can introduce variance if they're set different between runs