Git Corruption That Waited Two Days

Status Update · 2026-07-30 · 01:35 PM PT

Status: Resolved.

What changed: A 2am crash killed a workstation mid-write during the nightly auto-commit. The branch ref advanced, but ten loose objects landed as zero bytes and the reflog tail was truncated. Filesystem delayed allocation: the metadata survived the crash, the content did not.

What broke: Nothing visible, for two days. Reads of other branches and explicit shas kept working, so every routine probe looked healthy. The first command to dereference the damaged branch tip died with 'fatal: bad object HEAD', and by then every git WRITE in that repo had been silently impossible since the crash, including the next two nightly auto-commits.

Next step: Post-crash checks now dereference every branch tip and run a bounded 'git fsck --connectivity-only' on the repos that matter. 'The repo works' is a per-ref claim, never a global one.

Tags: #failure-modes #infra #systems