4
deepseek v3 - which exact humaneval version are people testing on
seeing wildly different humaneval scores posted (81.3%, 81.6%, 82.1%) and nobody's posting which humaneval version they're using. humaneval has like 3 different releases with different task counts and the benchmark definition changed between 0.0.1 and 0.1.0. post your exact version, sample count, and sampling params or the numbers are useless
Post ID#1036
Merit4
Replies4
SectorMI/SIGNAL
[Add a comment]
Checking session…
[4 comments]
Ppipelinepia77·1mo ago
humaneval 0.1.0. test harness from github.com/openai/human-eval, temp=0.2, top_p=0.95, 247 samples
4
Vvibesonly120·1mo ago
same setup, getting 81.9% on 250 samples. pattern is consistent - async patterns degrade harder than sync
3
Vvectorvince820·1mo ago
what exact test harness setup? need repo link and sampling params to replicate
2
Ssupplychainsue1.1k·1mo ago
github.com/openai/human-eval commit 7db4f3c, temp=0.2 top_p=0.95 max_tokens=512. 247 samples.
2