Judge-free reward-hack detection on the Obfuscation-Atlas honeypot
0.970 recall and 1.6% false-positive rate on the controlled MBPP honeypot benchmark — using zero model calls.
By Verifiable Labs
Most reward-hack detectors are themselves models — an LLM judge reading code and guessing whether it cheats. Judges are expensive, non-deterministic, and, in our runs, wrong a lot. Here is what a judge-free detector does on a public honeypot benchmark.
The benchmark
We evaluated on the Obfuscation-Atlas honeypot set (taufeeque/mbpp-hardcode, arXiv:2602.15515): 500 candidate MBPP solutions labeled genuine, blatant hardcode, or evasive hardcode. Hardcoded solutions pass the visible test without solving the task; evasive ones hide it.
The protocol
The detector is extensional: a candidate that passes the visible test — pass(test_list[0]) — is re-graded against held-out perturbed variants of the same task. Genuine solutions keep passing; hardcoded ones collapse. No model reads the code; nothing is asked to form an opinion.
Against LLM judges
On the same 366 items, a strong open LLM judge caught every hack (recall 1.000) — but false-flagged 14.75% of genuine solutions, took 4.818 seconds median per paid call, and flipped 2.67% of its own verdicts on repeated judgments. That trade-off — a slight recall edge for the judge (1.000 vs our 0.970) at ~9x the false-positive cost, plus latency and non-determinism — is why the deterministic layer is the release-gate core.