review-loop-exhaustion $4.22 · 87m · review_required_human
Migrate legacy auth middleware to new policy engine
State path intake → survey → draft → build → inspect → inspect → inspect → review_required_human
Inspector flagged subtle integration-test regressions across three sequential builds. Recovery logic now caps inspector iterations at 5 and escalates to operator with a diff summary, instead of looping indefinitely on the same class of regression. The PR shipped a week later under explicit human review; the failure mode shipped first.
trust-boundary-fabrication $2.87 · 54m · review_required_human
Wire new export resolver into the module graph builder
State path intake → survey → draft → build → inspect → build → inspect → build → inspect → review_required_human
The Builder claimed reviewer action items were resolved and supplied plausible-looking file contents that did not exist on disk. The Inspector accepted the claim without verifying against the branch. Fixed by requiring file-verified evidence: the already_correct manifest now includes SHA-checked file contents, and the Inspector cross-checks every Builder claim against the actual branch state before accepting it.
circuit-breaker-missing $11.34 · 3m · blocked
Implement dependency graph for cross-issue coordination
State path intake → survey → draft → build → build → build → build → build → blocked
84 tasks re-enqueued in 3 minutes against the same ENFILE error — each retry recreated the same failing worktree state. No circuit breaker existed for repeated identical build failures. Fixed by error-signature circuit breaker (#1089): three consecutive failures with the same error signature trip the breaker, block the issue, and require a manual operator unblock before re-entry.
subprocess-silent-hang $0.31 · 312m · worker_timeout
Refactor webhook dispatcher to async enrichment queue
State path intake → survey → draft → build
The Claude Code subprocess froze with no stdout or stderr output for over five hours. A single wall-clock timeout would have killed long-running but productive work. Fixed by dual-timeout defense: a wall-clock timeout caps total execution time, and an inactivity timeout kills the process if no data events arrive within 300 seconds. Both mark the result as transient and re-enqueue for retry.
workspace-git-corruption $1.98 · 29m · review_required_human
Add per-file confidence scoring to the merge conflict resolver
State path intake → survey → draft → build → worker_error → build → worker_error → review_required_human
Two workers sharing a host-mounted .git directory wrote concurrent worktree entries, corrupting .git/config and causing force-with-lease push rejections. Fixed by a three-step fallback in workspace cleanup — git worktree remove, then rm -rf on the directory, then git worktree prune — and by moving toward per-worker clone isolation to eliminate shared .git state entirely.
state-label-drift $6.13 · 218m · review_required_human
Ship structured dependency graph to replace HTML comment metadata
State path intake → survey → draft → build → inspect → intake → survey → draft → build → inspect → intake → survey → draft → build → inspect → review_required_human
GitHub's ETag-cached issue-list endpoint returned 304 Not Modified for several poll cycles after colony:blocked was applied. The sprint-master kept re-enqueueing tasks, driving three complete intake-to-inspect cycles before the cache invalidated. Fixed by making Postgres pipeline_issues.state the authoritative source for all pipeline routing decisions and demoting GitHub labels to a human-visible projection synced as a side effect.
Trust boundary Fabricated evidence
Agents under pressure can claim review items were fixed with plausible file contents that do not exist on disk. Colony requires file-verified evidence and Inspector checks against the actual branch state.
Review loop Unparseable LLM output
When the Inspector returns truncated JSON, empty verdicts, or prose fallbacks, retrying usually repeats the same failure. Colony retries once, then escalates instead of burning cycles.
Termination Looping failure paths
PR conflicts, build failures, and blocked retries can cycle indefinitely without explicit counters. Colony wraps these paths with circuit breakers and human-unblock escape hatches.
Subprocess Silent hangs
Claude Code subprocesses can freeze with no stdout or stderr. Colony uses both wall-clock and inactivity timeouts so productive long runs survive and silent hangs get killed.
Workspace Shared git state
Workers sharing a host-mounted .git directory created corruption, stale worktrees, and force-with-lease collisions. Colony moved toward per-worker clone isolation.
State Eventually consistent labels
GitHub labels are not a reliable distributed state machine. Colony made Postgres authoritative and treats labels as a projection for human visibility.
