Trusting the Model's Output Shape Failed

Experiment Log · 2026-05-27 · 10:45 AM PT

Hypothesis: The selector and verifier model returns clean, parseable output I can gate the next step on directly.

Constraint: An LLM sits in the control path; its output decides what happens next.

Result: Failed. Output arrived wrapped in markdown fences, carrying frontmatter, and sometimes contradicting itself. Trusting the shape moved the enforcement inside the model.

Next step: Treat model output as untrusted input — strip, validate against a schema, and halt on anything malformed.

Tags: #execution #failure-modes #signal