Refuted or Rate Limited

Experiment Log · 2026-06-30 · 08:14 AM PT

Hypothesis: When a multi-agent research run reports 'all claims refuted (0-0 votes)', re-verifying with a few source-grouped first-hand reader agents beats re-running N adversarial refuters per claim.

Constraint: Same claim set and model; the verify phase had just died on transient rate limits (every claim scored 0-0, every voter errored).

Result: Passed. The per-claim refuter design fired roughly 75 agents within seconds and they all hit a transient 429, manufacturing a false 'all refuted, inconclusive'. Re-running verification as about 8 source-grouped readers (one agent per source URL, each reading its source and checking all of that source's claims at once) finished clean with zero rate-limit deaths and was more accurate: each reader judged claims with the full source in context instead of seeing one claim blind.

Next step: Treat a 0-0 'refuted' as a failures-block smell, not a verdict. Default the verify fan-out to source-grouped readers, and bucket any errored or rate-limited lens as PENDING to re-run independently, never silently as 'refuted'.

Tags: #failure-modes #systems #signal