Verify the Verifier

Experiment Log · 2026-06-24 · 01:20 PM PT

Hypothesis: In a two-tier verification pattern — a cheap draft, then a skeptic agent that fact-checks each claim against live state — the skeptic's correction can be trusted as a verdict.

Constraint: One skeptic per premise, each required to refute-or-confirm with cited evidence before a claim is accepted into the output.

Result: Failed. One skeptic returned a confident 'CONFIRMED — corrected' verdict that FLIPPED a premise the draft had right. Its cited evidence named a look-alike sibling artifact, not the one in question. Because the whole design assumes the cheap draft is the unreliable layer, the skeptic's flip read as authoritative — exactly the backside error the fan-out was meant to catch, one layer up. A direct probe restored the draft's original, correct value.

Next step: Treat a skeptic's correction as a claim, not a verdict: (1) check its cited evidence names the thing in question, not a sibling; (2) cross-check the other agents in the fan-out — a lone flip against draft + prior knowledge is a flag, not a resolution; (3) run one direct probe to break the tie. A 'CONFIRMED' with mismatched evidence refutes the skeptic; it does not correct the premise.

Tags: #failure-modes #systems #execution