gemini 3.5 pro shows ~94% gpqa, whos actually verified this outside googles blog post
july GA and every writeup just repeats the same number. 94% GPQA is a huge claim and i have not seen one independent run. assuming its contaminated or cherry picked until someone posts a held out eval. anyone run their own set on it yet or are we all just trusting the launch graph
nobody outside google has posted a clean rerun that i can find, which is the tell. every open weights model gets picked apart on the public split within a week and the number always sags a few points off the launch blog. till someone runs the exact config on the held out set its just googles number, same as it was for everyone else
yeah the tell is nobody reran it clean. iirc gpqa also moves a few points just from prompt format, so 94 off their own blog means basically nothing until theres a public split run
94% off google's own blog is a marketing number until someone reruns it on the public split. gpqa also leaks depending on how you prompt it. has anyone posted an independent run with the exact config, or is it still just the launch post going around
this. 94% off their own blog is a marketing number until the public split gets rerun. and gpqa contamination isnt hypothetical, saw runs where just changing the prompt format moved the score like 6-8 points on the same model. post the exact config or its noise dressed as a result
the pattern that makes me distrust the 94 is that its always the vendor blog and never a third party rerun on the public split. coming from the data side, if a number only exists in the press release and nobody independent can reproduce it within a week, i treat it as marketing until proven otherwise. gpqa also moves several points just on prompt formatting, so even an honest 94 might not survive a config change. has literally anyone posted the exact eval harness they used