Dan Mercede: Living Signal Surface

Budget the Week You Act In

Sat, 11 Jul 2026 07:20:00 +0000

Budget the Week You Act In

Short Essay · 2026-07-11 · 12:20 AM PT

Claim: A scheduler that plans on the last day of a period must budget the period it will act in, never the one it is standing in. remaining(now) on a boundary day is an off-by-one-period bug wearing a compliance costume.

I shipped a weekly planner this week that runs Sunday evening and proposes Monday-to-Sunday work. Every proposed row checks a rate-limit ledger before it earns a slot, and the ledger keys its buckets by ISO week.

The first design read remaining(now). Sunday at 17:23 is still the old week. The ledger correctly showed that week at cap, so the planner would have zeroed every weekly surface in every plan it ever produced. Forever. And it would have looked right the whole time: "at cap" is a normal, healthy answer, not an error.

An adversarial review pass caught it before a line of code existed. The refutation took one file read: the live ledger held the ending week at cap while the week the plan targets sat untouched. The fix is boring on purpose: compute the target period start from the run date (so a catch-up run after a reboot still targets the right week), read the bucket at that date through the shared bucket-key helper, and keep the plan-time read informational while the fire-time atomic reserve stays the only authoritative gate.

The test is the part worth stealing. One boundary fixture, two assertions: ending-period-at-cap still yields next-period rows, and a full next-period bucket yields zero. Green looked exactly like the bug here, so the fixture has to discriminate the two weeks. "Respects caps" as a test name proves nothing; which week's cap is the whole ballgame.

Every generate-at-boundary planner (Sunday content planner, month-end batcher, end-of-day queue builder) needs a boundary fixture: ending-period-at-cap must still yield next-period work, and a full next-period bucket must yield zero.

Tags: #failure-modes #systems #execution

Refute Your Own Claims

Fri, 10 Jul 2026 17:00:00 +0000

Refute Your Own Claims

Experiment Log · 2026-07-10 · 10:00 AM PT

Hypothesis: With 67 green unit tests and a passing live end-to-end run, the claims in the README and submission for Engram (my Qwen Cloud hackathon memory engine) would survive an adversarial pass instructed to refute every claim against primary evidence.

Constraint: 75 discrete claims extracted verbatim; independent reviewer agents; primary evidence only (code, tests, git history, live HTTP probes); every claim verdicted HOLDS or OVERCLAIM.

Result: Failed. 21 of 75 flagged. Two were real defects hiding behind green tests: an argument-order swap feeding the contradiction-adjudication prompt, and a cosine gate set to 0.55 by feel when the README's own example pair (I just moved to Denver vs I live in San Diego) measures ~0.49 at 256 dims with text-embedding-v4, so the showcased supersede never fired live. Probed real embeddings, reset the gate to 0.45, re-verified the pair end to end on the deployed route. Repo: github.com/OrionArchitekton/engram.

Next step: The refute pass is a standing gate before any submission. Embedding thresholds get set from live-probed cosines of the actual example pairs, never intuition.

Tags: #failure-modes #execution #workflow-ownership

Quota Is Not a Parse Error

Fri, 10 Jul 2026 08:35:00 +0000

Quota Is Not a Parse Error

Short Essay · 2026-07-10 · 01:35 AM PT

Claim: A pipeline that consumes LLM output needs a third terminal state: a quota-starved reviewer is not-reviewed, which is neither a parse error nor findings.

Overnight, five runs of my PR review pipeline reported the same thing: the Codex review phase exited with status parse_error, schema invalid, zero findings. The obvious read is a broken parser or a drifted schema. The actual payload was one sentence from the vendor: "You've hit your usage limit ... try again at 2:00 AM."

Nothing was malformed. Nothing was reviewed either. The reviewer never ran; the "review output" was a quota message wearing a review envelope, and my pipeline filed it under parse failure. Worse, the run status rolled up as findings-remain, which reads like the reviewer saw problems. It saw nothing.

Pass and fail are verdicts. A quota-starved model produces neither, so classify it as not-reviewed, the same class as a timeout, and carry the vendor's stated reset time as the retry schedule. My pipeline already treated timeouts as non-terminal; the quota message deserved the same treatment and instead got promoted to a defect.

The cost of misclassifying is paid twice. First you debug a parser that is not broken. Then you spend your bounded re-run budget (mine allows one verification re-run per PR) before the quota resets, and the re-run burns on the same quota wall.

One grep fixed the triage: check the parse warnings for a usage-limit string before adjudicating any parse_error. Abstention is not a verdict, and an unreviewed PR is not a reviewed-clean PR.

Fold abstentions into failure states and you debug phantom parser bugs, trust findings that reviewed nothing, and burn bounded retries before the quota resets.

Tags: #failure-modes #signal #execution

Schema Is a Second Surface

Thu, 09 Jul 2026 08:20:00 +0000

Schema Is a Second Surface

Short Essay · 2026-07-09 · 01:20 AM PT

Claim: On a prerendered SPA, a copy reframe has to land on two independent surfaces, not one. The visible React copy auto-bakes into the crawler HTML. The head JSON-LD and social meta do not: they are hand-maintained, and they are exactly what answer engines read without executing your JavaScript.

A reframe of a small site network looked done. Visible copy updated everywhere, tests green, CI passing. Then review caught a stale line no test asserted: the JSON-LD `WebPage.name` in the page head still carried the old title, while every rendered surface had moved on.

That is the trap of a prerendered SPA. Your React components render twice. Once into the DOM a human hydrates, and once into a static body baked into the HTML so crawlers see content without running your JavaScript. Edit the component and both move together. The document head is a different animal. The title, the Open Graph tags, the JSON-LD entity graph: all hand-maintained, and nothing about editing your visible copy touches them.

So a copy change has two surfaces. Miss the second one and you publish a contradiction. The page a person reads says one thing; the structured data a search engine or an answer engine ingests says another. Answer engines lean on that structured data precisely because it is explicit and needs no JavaScript to parse. A stale `name` field there is not a cosmetic miss. It is a split identity signal you are feeding, on purpose, to the machines that summarize you.

Three independent reviewers flagged the same line before it merged. The guard suite flagged none of it, because the identity test asserted the backlink was present, not that the title was current. Structural tests check shape. They do not check that your prose and your schema agree.

The fix was one line. The habit is cheaper than the fix: when you reframe copy, grep the schema too. Diff the head JSON-LD, not just the page. The rendered view is the surface you look at. The structured data is the surface the machines look at, and it is the one you will forget.

Reword the page and skip the structured data and you ship a split identity: humans read the new positioning, crawlers read the old one. When you reframe copy, grep the schema, not just the rendered page.

Tags: #failure-modes #systems #signal

Certify vs Reproduce

Wed, 08 Jul 2026 16:20:00 +0000

Certify vs Reproduce

Short Essay · 2026-07-08 · 09:20 AM PT

Four review layers, each sensing a different modality. Independent ones do not share blind spots.

Claim: A reviewer that reasons about your code and one that runs it have different blind spots: certification is an argument, reproduction is a fact.

A security reviewer read my credential scrubber and certified it: no overlap mis-redaction. One review layer later, a different reviewer ran the code and reproduced a partial secret in the logs. Same function, opposite verdict.

The two reviewers were not unequal in skill. They had different blind spots. A reviewer that reasons about your code builds an argument for why it is correct. A reviewer that executes it observes what actually happens. An argument can be sound and still miss the exact input the code meets in production. Reproduction does not get that option: the secret either leaks or it does not.

This is why a single review pass is not a safety net for AI-written code. The passes have to be independent, because independence is what stops their blind spots from lining up. Certification tells you a careful reasoner could not find the bug. Reproduction tells you whether the bug is there. Keep both, and keep them separate.

The full walkthrough, four review layers and five defects each caught by only one, is here: https://www.danmercede.com/guides/why-agent-code-needs-layered-review

For AI-written code, stack independent review layers so one layer's blind spot is another layer's catch.

Tags: #failure-modes #execution #security

Vet the AI PR Fleet

Tue, 07 Jul 2026 21:35:00 +0000

Vet the AI PR Fleet

Short Essay · 2026-07-07 · 02:35 PM PT

Claim: Merging a queue of AI-authored pull requests is not a rubber stamp. Three failure modes hide in the queue, and each needs verify-before-you-trust, not blind application.

Merging a batch of machine-authored pull requests looks mechanical until three failure modes surface in the same session.

First, contradictory contracts. Two PRs written against different base commits can encode opposite rules for the same interface. One says a source must be a clean checkout and fails closed; another assumes the source can be a loose bundle and falls through. Whichever lands first silently invalidates the other's premise. The tell is the second PR's own test failing against the newly merged base. Do not force the merge with ours or theirs. Stop and hand the design question to a human.

Second, confident false positives. A review bot can flag a high-severity bug that is not real. One insisted a shell here-string leaves a stray newline stuck to the last array element. It does not: `read` consumes that newline as the line delimiter, so the last field is clean. A ten-second empirical check settles it. Verify a finding before you apply it, because a wrong fix regresses correct code with full confidence.

Third, the fix that already exists. When a PR still shows open findings, your own automation may have committed the fix locally and then failed to push it, blocked by a repo gate on a missing dependency. Reconcile the PR head against the local branch before you touch anything, because they diverge. Adopt the existing fast-forward commit instead of re-authoring the same change from scratch.

The through line is that a fleet of AI-written PRs earns trust one verification at a time, and the checks live at three layers: cross-PR contract conflicts, bot-finding accuracy, and unpushed automation state. The merge button is the last gate, not the first.

Treat a fleet of machine-written PRs like untrusted input: gate the merge on empirical checks at three layers, not on the open-PR count going down.

Tags: #failure-modes #execution #systems

Recon vs Prior Decisions

Mon, 06 Jul 2026 07:20:00 +0000

Recon vs Prior Decisions

Short Essay · 2026-07-06 · 12:20 AM PT

Claim: A fresh recon-and-synthesis agent sees only live system state, not the decisions you already made about it, so it will confidently recommend something you already rejected.

Fan a recon pass out over live state, let an agent synthesize it, and the result is grounded, plausible, and blind. It answers "what is the state" but not "what has already been decided about it." So it surfaces the obvious move, say "route all the inference to the idle GPU," when an earlier analysis already found the real constraint was model reliability, not spare capacity, and rejected exactly that. The synthesis is well cited and wrong.

The fix is cheap. Either seed the synthesis with prior-decision context (run a memory or lessons search first and feed its digest into the synthesis prompt), or treat the raw synthesis as a draft and reconcile it against your decision docs yourself before you act. The reconciliation is the deliverable, not the recon.

A second shape of the same blindness shows up in wide research fan-outs: a broad multi-part question spends its budget where sources are dense and quietly under-covers the sparse sub-questions. Do not accept the half-answer, and do not rerun the whole thing. Fire a focused re-run scoped to just the starved parts, and read the abstention as a gap to close rather than an answer.

Reconcile the raw synthesis against your prior decision docs before acting; the reconciliation, not the recon, is the trustworthy output.

Tags: #execution #systems #signal

Cut Agent Web-Read Tokens

Sun, 05 Jul 2026 22:40:00 +0000

Cut Agent Web-Read Tokens

Experiment Log · 2026-07-05 · 03:40 PM PT

Raw HTML can burn 80 percent of an agent's context. Extract-first fetching keeps only the signal.

Hypothesis: Converting a page to markdown before the model reads it, and fetching in a subagent, cuts an agent's web-read token cost without losing content.

Constraint: One real page, raw HTML vs clean markdown, same tokenizer.

Result: Passed. 9,541 tokens raw vs 1,678 as markdown, an 82% cut on that page (20-30% typical). The extraction is deterministic markdown, not a model summary, so it cannot hallucinate. A keyless reader CLI runs in a subagent, so raw HTML never reaches the main context. A live smoke test caught a browser User-Agent 403 the unit tests missed.

Next step: Full stack and the reader CLI in the guide: https://www.danmercede.com/guides/giving-your-agent-web-access

Tags: #economics #execution #systems

PR-Merge Train Gotchas

Sun, 05 Jul 2026 07:15:00 +0000

PR-Merge Train Gotchas

Short Essay · 2026-07-05 · 12:15 AM PT

Claim: Merging a stack of PRs into one branch has host-side mechanics that silently skip half your queue if you script it naively.

Merged fourteen PRs into one repo today in a single pass. If you script that as "loop over the open PRs, merge each one," you will silently skip half of them. Here is why, and the five things that actually decide whether the train runs.

**Each merge invalidates the others.** The moment one PR lands, every other open PR's mergeable state flips to UNKNOWN for a few seconds while the host recomputes against the new base. Read `mergeable` once and your loop sees UNKNOWN, calls it "not ready," and skips a perfectly clean PR. Treat mergeability as eventually consistent: poll each PR until it settles to MERGEABLE or CONFLICTING before you act on it.

**Admin-merge bypasses more than review.** An admin merge skips the required-review rule and the "branch must be up to date" rule, but only when branch protection has enforce_admins set to false. A PR that is behind but clean merges fine. Only a true content conflict stops you. Check the protection setting before you trust what the state string is telling you.

**Order the train by a conflict graph.** Build it from each PR's changed-file list. Merge the largest set of PRs touching disjoint files first, for zero rebases. PRs that share a file cascade-conflict, so sequence them last. That turned a five-PR cluster all editing the same manifest into four clean merges and exactly one resolution.

**Resolve without rewriting history.** When two PRs edit the same lines and force-push is off the table, merge the base branch into the PR branch and union the hunks. Do not rebase. A squash-merge flattens the extra merge commit anyway. Union so both sides' tests still pass: two of mine grepped for different literal regex branches, so both had to survive the merge.

**Stand down on concurrent work.** One PR had another process actively reviewing and merging it. I left it alone instead of racing. It finished and merged itself, correctly, while the rest of the train ran. The cheapest conflict is the one you decline to start.

None of this lives in the merge button. All of it bites the first time you automate the queue.

Treat mergeability as eventually consistent and order the train by a file-conflict graph, or the merge button lies to your loop.

Tags: #execution #failure-modes #systems

Benchmarks Evict Hot Models

Sat, 04 Jul 2026 07:15:00 +0000

Benchmarks Evict Hot Models

Experiment Log · 2026-07-04 · 12:15 AM PT

Hypothesis: Benchmarking a locally-served LLM is read-only and won't disturb whatever model is already loaded on the GPU.

Constraint: One 24GB GPU, a single ~20GB model pinned always-hot via OLLAMA_KEEP_ALIVE=-1, comparison calls fired from a throwaway script.

Result: Failed. The benchmark passed its own per-request keep_alive and also loaded a second model to A/B. Both evicted the pinned model, because only one ~20GB model fits on a 24GB card. The next real request ate an ~80s cold reload, visible as load_duration in the response. keep_alive is per-request, the -1 default only holds when you do not override it. Second trap the same run surfaced: the quantized MoE returned done_reason of length with an empty response string, so an eval_count greater than zero check scored it a success while it produced nothing usable.

Next step: Treat resident-model residency as mutable runtime state a probe can perturb. Re-warm the canonical model with keep_alive of -1 after benchmarking, never interleave two models on one GPU expecting both to stay hot, and assert on the output text, not the token count.

Tags: #infra #failure-modes #systems

Don't Index Your System of Record

Fri, 03 Jul 2026 15:40:00 +0000

Don't Index Your System of Record

Thought Snippet · 2026-07-03 · 08:40 AM PT

Went to drop a PDF into an agent's knowledge vault today and hit the right instinct by accident: the vault's ingest crawler explicitly excludes its own directory from the crawl. Obvious in hindsight. A recall layer, the RAG corpus or the wiki an agent searches, is advisory. It is lossy, re-summarized, allowed to drift. Your registers and config are authoritative. Index the authoritative thing into the advisory thing and you invert that: a stale photocopy of the truth now sits in the retrieval path, and the agent cannot tell which copy governs. Keep the lanes apart. External third-party knowledge goes in the recall corpus. Your source of truth stays in its own home and gets read directly, never re-indexed as advisory. If your pipeline crawls a directory tree, exclude the knowledge base from its own crawl.

Tags: #systems #governance #signal

Silence Is Not Absence

Thu, 02 Jul 2026 07:30:00 +0000

Silence Is Not Absence

Experiment Log · 2026-07-02 · 12:30 AM PT

Hypothesis: An extraction pipeline that returns nothing means the target page carries no server-rendered content worth reading.

Constraint: Headless curl of a JS-heavy document viewer, piped through sed and a grep-family matcher, with stderr captured into the same output file as stdout. Every stage able to fail independently.

Result: Failed. The empty page was my pipeline failing, not the source being empty. The context regex blew the matcher's complexity budget (ugrep: error ... exceeds complexity limits) and 104 bytes of that error text was the entire captured output; a first read of the file even reported it empty. A dumber second pass on the same URL found a 52KB document sitting behind a signed URL embedded in the HTML, with escaped ampersands (\u0026) that also had to survive decoding. The source was never empty. The extractor was broken twice, and each failure read as no-content.

Next step: Treat empty as unproven rather than absent: check exit codes per pipe stage and byte counts at every hop before concluding a source has nothing. Same failure class as a rate-limited search API returning [] that gets read as zero results.

Tags: #failure-modes #signal #execution

The Limit Kills the Report, Not the Work

Thu, 02 Jul 2026 07:10:00 +0000

The Limit Kills the Report, Not the Work

Short Essay · 2026-07-02 · 12:10 AM PT

Claim: A background agent that dies on a rate limit has not necessarily failed its task.

Yesterday a builder subagent returned exactly one line: a session-limit notice. Thirty-two tokens, no report. The obvious move is to re-dispatch. I probed first: the worktree existed, the branch was pushed, the pull request was open. The work was done; the limit killed the report, not the work. Re-dispatching blind would have opened a duplicate PR against the same branch, and outward actions like pushes, PRs, and posts are not idempotent.

The discipline: before re-driving any limit-killed or crashed agent, probe its intended artifacts live. Check the branch with git ls-remote, list PRs by head, stat the files it was told to write. Then finish only the missing tail inline. The same rule scales up: a workflow journal that caches completed agents makes resume cheap, and per-step commits make git log the ground truth for what actually happened, regardless of which reporter died.

Treat 'the agent died' and 'the task failed' as separate facts to verify independently. The cheaper the kill signal (a limit, a timeout, a crash), the more likely the work outran the report.

Tags: #failure-modes #execution

Recheck What You Rejected

Wed, 01 Jul 2026 07:45:00 +0000

Recheck What You Rejected

Short Essay · 2026-07-01 · 12:45 AM PT

Claim: A negative decision stored in an agent's memory ('we evaluated X and rejected it') is the most dangerous kind of stale fact, because it reads as settled and quietly stops you from re-checking.

Today a build nearly started from a false premise. Two sources agreed a monitoring tool had been evaluated and rejected months back: a quick scouting pass, and the agent's own memory note from that earlier decision. Both said "we don't use it."

A live probe said otherwise. The tool was in production: agents running on three hosts, config checked into the repo, a merged pull request wiring it up. The rejection was real when it was written. An adoption decision reversed it later, and the note never caught up.

Open questions get re-checked. They read as unfinished, so something in you goes and looks. A recorded rejection does the opposite. It reads as closed, so you skip the probe and inherit a months-old snapshot as current truth. The more settled a stale fact sounds, the less it gets verified, which is exactly backwards.

The fix is cheap. Treat "we rejected X" or "we don't use X" from memory, a handoff, or a plan as a hypothesis, not a fact. Re-probe the live world: running processes, config on disk, merged PRs. Give a recalled negative the same suspicion you give a "still broken" bug report. Live evidence decides; the note only nominates.

Before building on 'we don't use X', re-probe live for X's actual presence. Only affirmative live evidence retracts the note.

Tags: #failure-modes #systems #signal

Prefill demos server-side

Tue, 30 Jun 2026 23:45:00 +0000

Prefill demos server-side

Experiment Log · 2026-06-30 · 04:45 PM PT

Hypothesis: An automated, headless demo pipeline can record a stateful web app's flow shot-by-shot as scripted, with no changes to the app.

Constraint: Each shot runs in a fresh browser context (no carried state), and the pipeline trims each segment to its narration length.

Result: Failed. Shots that triggered a live async result and then narrated it showed only the loading spinner: the rendered result landed after the narration window and got trimmed off, and later shots ran against a blank page because state did not carry. The fix that made it deterministic was a server-rendered demo mode (a query param read on the server) that seeds prefilled inputs AND a frozen-but-real result directly into the server-rendered HTML, so the content is present on load with no hydration race. A browser-side effect that prefilled after mount lost the race against capture. Second fix: long captions cover the very thing you narrate, so render one result card full screen per shot instead of all of them.

Next step: Make the server-rendered demo mode and one-card-per-shot the default in the demo tooling. Unrelated footgun from the same run: kill dev servers by port, never by matching the process command line, because that also hits your own running script and kills the run, and a stale server still holding the port makes a fresh build silently serve the old code.

Tags: #failure-modes #execution #systems

A Report Is Not Recon

Tue, 30 Jun 2026 15:22:00 +0000

A Report Is Not Recon

Short Essay · 2026-06-30 · 08:22 AM PT

Claim: A research report's accuracy about the outside world does not transfer to its accuracy about your own system. The trend scan can be impeccably sourced while every 'so adopt it here' line rests on an unchecked guess: that the named target exists, that you have not already built it, that this is even the right home. Those guesses read as facts and produce confident, wrong work.

Before turning a report into work, recon each recommendation against live state: does the target exist, is it already-have, partial, or net-new, is the home right? Reclassify from evidence, not from the report's framing. One recon pass over a single report this week caught six such backside errors, each of which would have shipped a wrong build: a named code target that did not exist, a doctrine note proposed for a health-check tool's slot, an eval suite called 'already built' that solved a different problem.

Tags: #execution #failure-modes #systems

Refuted or Rate Limited

Tue, 30 Jun 2026 15:14:00 +0000

Refuted or Rate Limited

Experiment Log · 2026-06-30 · 08:14 AM PT

Hypothesis: When a multi-agent research run reports 'all claims refuted (0-0 votes)', re-verifying with a few source-grouped first-hand reader agents beats re-running N adversarial refuters per claim.

Constraint: Same claim set and model; the verify phase had just died on transient rate limits (every claim scored 0-0, every voter errored).

Result: Passed. The per-claim refuter design fired roughly 75 agents within seconds and they all hit a transient 429, manufacturing a false 'all refuted, inconclusive'. Re-running verification as about 8 source-grouped readers (one agent per source URL, each reading its source and checking all of that source's claims at once) finished clean with zero rate-limit deaths and was more accurate: each reader judged claims with the full source in context instead of seeing one claim blind.

Next step: Treat a 0-0 'refuted' as a failures-block smell, not a verdict. Default the verify fan-out to source-grouped readers, and bucket any errored or rate-limited lens as PENDING to re-run independently, never silently as 'refuted'.

Tags: #failure-modes #systems #signal

Tests Spend Prod Budget

Mon, 29 Jun 2026 17:40:00 +0000

Tests Spend Prod Budget

Short Essay · 2026-06-29 · 10:40 AM PT

Claim: A test that reserves against a real shared-state file (a daily-budget ledger, a counter, a lock directory) silently spends your production budget and goes flaky across runs. The cause is almost always a path that is an import-time constant: the test isolates its own outputs but not the shared-state path, so every run writes to the same file the live system uses. Two failures land at once. Production's budget gets consumed by CI, and repeated suite runs accumulate reservations until live-write assertions fail on a green-yesterday diff that never changed. Fix it at one chokepoint, not per test: a single autouse fixture that redirects the shared-state path to a temp directory, because an import-time constant cannot be redirected per test any other way. The tell is a suite that passed yesterday failing on count or budget assertions while the code under test looks innocent. Inspect the real state file before you debug the diff.

Production-shared mutable state reached through a non-injectable path is a test-isolation bug waiting to happen. Make the path injectable, isolate it once, and the same tests stop both corrupting prod and flaking.

Tags: #failure-modes #execution #systems

Runtime Governance: How It Works

Mon, 29 Jun 2026 15:00:00 +0000

Runtime Governance: How It Works

Diagram · 2026-06-29 · 08:00 AM PT

Continuously evaluate, decide, and enforce at runtime, without slowing down your apps.

Tags: #governance #execution #systems

The Two-Plane Architecture

Mon, 29 Jun 2026 15:00:00 +0000

The Two-Plane Architecture

Diagram · 2026-06-29 · 08:00 AM PT

One box, one reverse proxy, zero public inbound ports: a private mesh for everything else.

Tags: #systems #infra #execution