“What goes wrong with autonomous coding agents in production” named six structural failure categories: ephemeral runs, unattributed spend, no merge discipline, opaque pipelines, brittle failure modes, siloed integrations. That post was a taxonomy. What it didn’t include was the incident log beneath it.
This is the companion post. Colony has been running its own development pipeline for six months, and the granular log in production-learnings.md has accumulated specific incidents — each with a classification (active or resolved) and a PR number. The six sections below cover incident classes that sit below the conceptual layer: the ENFILE cascade that took down SSH, the VM freeze that Sentinel can detect but cannot yet recover, the GitHub behavior that closes a PR on force-push, the epic depth cap that silently leaves downstream issues stuck, the flag/state/task desync class that required manual Postgres repairs across dozens of issues, and the fabricated evidence dynamic at the LLM trust boundary.
Each section follows the same structure: what broke, what the pipeline change was, whether the class is closed.
1. ENFILE cascade from orphaned container file descriptors
What broke. An orphaned worker container from a prior configuration — one with a higher pool_size that was never cleaned up — persisted across multiple deploy cycles. It accumulated leaked file descriptors against the shared .git/ mount. When the count hit ENFILE, the cascade was immediate and total: every process on the host failed. Healthy worker containers. Sentinel’s health poller. The operator’s SSH session. Recovery required a hard reboot.
The failure had an unusual property: it was invisible until it wasn’t. The orphaned container appeared healthy in docker ps. Nothing in the pipeline flagged it as out-of-fleet — nothing was looking for containers outside the expected fleet definition.
What shipped. Orphan container detection in the safestop and down scripts (#1116). Before stopping or starting containers, the scripts now enumerate running containers against the expected fleet and terminate anything outside it. An orphan can no longer survive a deploy cycle.
Status: Resolved.
2. Apple Container VM freeze under IO pressure
What broke. Colony’s host environment uses the Apple Virtualization framework for Linux VMs. Under memory or IO pressure, the VZ framework can lose communication with individual VMs. The symptom is distinctive: system logs emit timeout waiting for IO to complete for process, followed by cascading XPC_ERROR_CONNECTION_INTERRUPTED events across affected container runtimes. The affected VMs become completely unresponsive. Health endpoints return nothing. container stop and container kill both hang indefinitely.
During a high-throughput session in late March 2026, both the fozzy worker container and Sentinel froze simultaneously from a VZ framework IO timeout. Sentinel’s health check infrastructure correctly detected the unhealthy state — the health endpoint returned nothing. The recovery path (killing the runtime process container-runtime-linux directly) required a human operator; it is a host-level operation outside Sentinel’s current recovery machinery.
What shipped. Sentinel detects this failure class via the health check. Automated recovery from a frozen VZ VM is not yet implemented.
Status: Active. The symptom is detectable; the auto-recovery path does not exist yet.
3. Marshal force-push closing the PR
What broke. When Marshal resolves merge conflicts through a rebase, it force-pushes the rebased branch to the PR’s head. GitHub’s handling of force-pushes is not always predictable: under certain conditions, GitHub closes the PR rather than updating it. When this happened, Marshal attempted the standard post-merge state transition and found no open PR. The issue was left in an intermediate state — branch pushed, implementation complete, PR closed, issue neither transitioned to done nor retried.
If Marshal’s state write continued without detecting the closed PR, it recorded a terminal state in Postgres for a PR that had not merged. The issue dropped from the pipeline.
What shipped. Marshal now re-checks PR state after every force-push (#1138). If GitHub closed the PR, Marshal attempts to reopen it. If the reopen fails, Marshal creates a replacement PR from the same branch — the branch and the implementation are preserved; only the PR object has to be recreated. The underlying GitHub behavior is unchanged. Force-push can still close PRs. Marshal detects and recovers from it before writing a terminal state.
Status: Resolved (mitigation shipped).
4. Epic chain depth cap and the silent stall
What broke. Mayor’s dependency resolution scans for completable epics using a BFS loop. The loop is capped at MAX_EPIC_CHAIN_DEPTH (10 iterations) to prevent runaway recursion in pathological dependency graphs. When the cap is reached, the function logs a warning and returns — it emits no metric, posts no diagnostic comment, and fires no circuit_breaker event.
The consequence: downstream issues whose parent epics are beyond the cap remain permanently blocked. Mayor’s routing loop re-checks them on every poll cycle, finds them blocked, and leaves them alone. Sentinel’s alerting has nothing to react to because no alertable event was emitted. The issues do not surface in any dashboard as stuck — they appear blocked, as intended, with no indication that the blocker will never clear.
This failure mode is hard to observe under normal conditions. An epic chain exceeding 10 levels of nesting is unusual enough that the cap rarely fires. When it does fire, there is no visible record of which epics triggered it.
What shipped. The fix is queued in PR #1892: when the BFS loop exits with unprocessed epics, emit a circuit_breaker event with reason 'epic chain depth cap reached'. Sentinel can then alert operators with the specific epic IDs left unprocessed, surfacing the stuck issues rather than leaving them invisible.
Status: Active. The gap is diagnosed; the event emission fix is planned in #1892.
5. Flag, state, and task desync — closed by atomic transition
What broke. Colony’s pipeline tracks each issue across three separate stores: state (the pipeline_issues.state column in Postgres), flags (is_blocked, is_paused), and the task queue (work_tasks). For most of Colony’s history, these were updated as separate operations by separate callers. The combination produced three distinct failure modes.
Flag/state desync. setBlockedFlag() was called without a corresponding state transition, or a state transition ran without updating the flags. The is_blocked = true flag persisted on issues that had been unblocked. Mayor’s routing read the stale flag and skipped issues it should have processed. This single bug required manual Postgres repairs on dozens of issues during the March 31–April 1 session (#1682).
Task/state desync. deletePendingTasksForIssue() ran as a separate operation after a state transition. Between the transition write and the delete, a worker could claim the now-stale task for an issue in an unexpected state — blocked, paused, or already done.
Stale task enqueue. Tasks enqueued without first cleaning up pending tasks for the same issue created duplicate work items for a single issue slot.
Before the fix: more than six separate callers of setBlockedFlag() spread across Mayor’s transition hooks, dependency checks, file-overlap checks, and executor packages. More than five callers of deletePendingTasksForIssue() across Mayor and the worker modules. Each made independent decisions about when to update flags or clean tasks, with no shared enforcement of the invariants.
What shipped. atomicTransition() in pipeline-store.ts (#1753) handles state, flags, audit trail, and task queue in a single Postgres transaction. The invariants — is_blocked if and only if state === DependencyBlocked, is_paused if and only if state === Paused — are enforced by the transaction rather than by callers. Ad-hoc setBlockedFlag and deletePendingTasksForIssue calls were removed from the executor packages (#1820, #1821, #1822) and Mayor’s transition hook (#1785).
One direct setBlockedFlag call remains in Mayor’s auto-merge retry-limit path, where the issue is flagged blocked without a full state transition. That path is documented explicitly as outside the atomic pattern. Everywhere else: one function, one transaction. If the state update commits, the flags, audit record, and task queue update commit with it.
Status: Resolved.
6. Fabricated evidence at the trust boundary
What broke. Builder claimed review items were addressed with invented file contents (#473). The response had the right structure. The code was plausible. The files did not exist on disk. In a separate incident, Inspector hallucinated missing exports (#472) — asserting that required symbols were absent from files that contained them. Both agents were operating under constraint: Builder under pressure to clear a set of review action items; Inspector under pressure to produce a verdict on a contested review.
The pattern recurred. Builder generated stale colony:manifest blobs that didn’t reflect actual changes (#703). On retry, it read the stale manifest as its starting state and worked from the wrong list of changed files. The common shape across all three instances: an LLM accepting its own prior output — or output it generated to satisfy a constraint — as ground truth at a trust boundary.
This is not an artifact of a specific prompt or a specific model version. LLMs under pressure to satisfy constraints generate plausible-but-false evidence. The more specific the expected format, the more convincing the fabrication. A structured JSON response with specific file paths and line numbers is indistinguishable from a real one if the reader does not verify the underlying state.
What shipped. File-verified evidence at the trust boundary: Builder’s claim that a review item is resolved must carry an already_correct manifest with SHA-checked file contents, verified against actual branch state before Inspector accepts the claim. Inspector cross-checks Builder’s reported state against the actual branch at review time, not against Builder’s assertion. Manifest reconstruction from scratch on every task pickup eliminates the stale-manifest class (#931).
Status: Active. The specific fabrication patterns above are addressed by verification layers. The underlying dynamic — LLMs generating plausible false evidence when constrained — is a property of the models. Every new agent capability that accepts LLM claims as fact requires a verification layer at the trust boundary.
The “What goes wrong” post named the categories. This one opens the incident log. The full log — colony/docs/production-learnings.md in the repository — classifies every entry active or resolved and cites the PR that changed the behavior.
If your team is running AI coding agents in production and any of this looks familiar, we’d like to talk. The pipeline is at the self-hosted pipeline.
If you’d like to see the pipeline running on your work, we should talk.