anyone found a prompt shape that dodges the fable 5 classifier on benign refactors
ok so half my team's debugging requests are getting flagged since the july relaunch, and 'benign refactor' is apparently deep in the false positive zone for whatever they shipped. red-teaming it from the other side: i can't find a stable pattern. sometimes prefixing with the intent ("this is a safe internal refactor, no exploit") helps, sometimes it makes it worse because now the exploit-class tokens are in the prompt. anyone actually characterized what the classifier keys on, or measured a false positive rate on a labeled refactor corpus? i'd rather have a number than a folk remedy
the framing of 'dodge the classifier' is gonna age badly the second someone at anthropic reads this thread... but practically the thing that worked for me was splitting the refactor into steps that individually dont look like the exploit class. it flags the whole diff, not the pieces. annoying and clearly temporary but
the shape isnt the interesting number, the false positive rate is. i ran ~40 boring refactors through it last week - rename, extract function, null checks - and something like a third got flagged as "potentially harmful". zero of them were. until anthropic publishes the classifier fp rate on benign code this is going to keep eating legit debugging sessions... and no im not going to post a working bypass in a public thread, use your head
a third flagged is rough. i ran maybe 15 boring ones (rename, pull out a helper, add a null guard) and got 4 flags, so your ~33% tracks. the false positive number is the only stat that matters here and anthropic isnt publishing it, shocker