anyone got real numbers on the fable 5 safety classifier false positive rate on benign refactors
the vibes-based "it refuses everything now" posts are useless to me. i watch enough llm traffic to know a classifier this trigger happy has a measurable FPR. has anyone run a benign coding set through pre-June-12 vs post-July-1 and logged how many clean refactor/debug prompts trip it? even a rough n=200 with the confusion matrix would beat 400 anecdotes. self-plug caveat, i'm probably not the right person to build the eval but i'll run it if someone hands me a task set.
the honest answer is nobody has a clean number because "benign refactor" isn't a fixed set. if you want something defensible you need a held-out corpus of refactors labeled by a human, run each 5x at temp 0, and count refusals + soft-deflections separately (the classifier degrades gracefully sometimes, it doesn't always hard-refuse). i saw roughly 1 in 8 on rename/extract-method style prompts post-relaunch but that's my corpus, not yours. Fable 5's classifier seems tuned on something adjacent to code, so anything that mentions "bypass" or "disable" in a totally benign sense trips it
this is the right frame. and the reason a clean number is impossible is the classifier isnt keying on 'refactor', its keying on token patterns that correlate with the exploit class they patched, so your false positive rate is entirely a function of how much your benign refactors overlap that distribution. if you must publish a number, at least stratify by refactor type (rename vs extract-method vs pointer/memory stuff) because the memory-adjacent ones trip it 3-4x more in my runs. one blended percentage is meaningless