Human Task, signal, and Agent-approval recovery

Use this when a run is parked at a Human Task or Agent tool approval, a task looks orphaned or overdue, a signal appears lost or duplicated, notifications are not arriving, or an indexing wait never wakes. Human Tasks are the control-plane projection of a parked run — the run's checkpoint and wait token remain the source of truth, so recovery never means editing task rows to force a resume.

Confirm topology

1. GET /api/v1/human-tasks?filter=open — open tasks the caller can monitor (read-only projection; no draft/form values, no resume secrets).
2. GET /api/v1/human-tasks/{id} — one task's lifecycle, work state, assignee, dates, and outcome.
3. GET /api/v1/runs/{id} — the linked run should be waiting while its task is open.
4. Run detail / task inbox in the UI — the safe reason, policy/version identifiers, and the allowed recovery action for a parked, blocked, or failed step.

A parked run holds no worker lease and no live process, and its compute segment is closed with a waiting-<kind> reason, so a sleeping task does not reserve or bill compute. A resume opens a fresh segment. If a waiting run still shows an active lease, treat it as a lease bug, not a task bug — see the workflow and worker recovery runbook.

Parked task never resumes

Deadlines are advanced by sweepHumanTaskDeadlines on each worker tick (reminders, escalations, hard timeout). Reminders and escalations update task/audit state and notify the target but do not resume the run. Use injected/advanced clocks in tests; never sleep for a real deadline.

Signal lost, early, or duplicated

Signals (human responses, external callbacks, indexing completion) are durable receipts keyed by workspace, correlation, and idempotency key, storing only a bounded payload and digest.

• Early signal (arrives before the wait is visible): buffered on acceptance; wait activation rechecks buffered receipts, so a valid response is never lost.
• Duplicate signal: the receipt idempotency key + task/checkpoint revision CAS make a replay converge — the run resumes at most once.
• Externally presented resume secrets are distinct from internal wait IDs, stored as digests, never returned by list APIs, and never logged. A leaked link cannot be replayed after consumption.

Diagnose with the run's checkpoint revision and the receipt's accepted/consumed state. A consumed receipt with a resolved token is healthy; a consumed receipt without a resolved token is the invariant to escalate.

Notification outbox retry and exhaustion

The transactional outbox commits with the checkpoint; dispatch starts only after commit and is retryable with claim/lease/attempt metadata and exponential backoff. On reaching maxAttempts a record becomes failed (not silently dropped). The task inbox remains the source of truth, so a failed notification never hides work — reconcile by re-claiming and re-dispatching, and watch the humanTask.outbox.retryExhausted counter for delivery-channel outages.

Agent tool approval replay

A pending Agent tool call projects into the same inbox as an immutable proposal (normalized tool, target, argument digests, risk, plan hash). Approve executes the exact hashed proposal; the runtime revalidates guardrails and current authority immediately before execution. If the node policy, target, or arguments changed, the plan hash no longer matches and a fresh approval is required. Rejecting with feedback creates a new proposal and hash — it never edits the pending action in place. A revoked principal's approval is rejected and the wait remains active.

Cancellation / timeout race

When completion races a timeout or run cancellation, one checkpoint compare-and-set wins and the loser receives a terminal conflict (409/410) without a second resume. The task and run cannot end in contradictory terminal states. Inspect the winning transition from the terminal task deep link, which stays readable to authorized users.

Feature rollback

Human Task projection, MCP typed write tools, Agent guardrail profiles, and Knowledge lifecycle nodes graduate independently. To roll a gate back, disable its emitters while preserving readable state and legacy paths: legacy approval graphs, wait tokens, and knowledgeSearch remain usable when a new gate is off. Do not drop the additive tables — expand-contract migrations keep older readers safe, and the durable receipts/tasks/outbox rows stay as the audit trail.

Observability

Structured counters (bounded labels, never prompts/drafts/documents/tool args/secrets/embeddings):

Every label value is drawn from a fixed, closed domain (enum, age bucket, or <stage>.<rule> machine code) with a safe other fallback, so a counter can never carry a prompt, form/draft, document content, tool argument, credential, token, embedding, or provider payload.

Compute billing while parked

A parked Human Task, wait, or Knowledge indexing wait closes its active compute segment with a waiting-* close reason (waiting-approval, waiting-event, waiting-timer) and releases occupancy, so calendar sleep is never metered or billed. Resume opens a new compute segment. If a segment is left active by a crashed worker, recoverAbandonedWorkflowComputeSegments finalizes it with worker-crash at its last heartbeat rather than billing wall-clock time.