Workflow and worker recovery

Use this when runs stay queued, workers crash mid-graph, waits never resume, or imported n8n workflows will not deploy.

Confirm topology

1. GET /api/v1/workers/status — queue.mode, queue.error, persistence.durable, database, health.reasons.
2. GET /api/v1/workers/doctor — blockers / nextActions (bundle inventory if you pass bundlePath).
3. Settings → Workers: same public fields (no env var names, no secrets).

Memory-only persistence: queued run rows and collaboration sessions die with the process. After restart, empty run lists are expected — re-execute; do not look for the old run ids.

Runs stuck in queued

Queue mode needs a running worker and reachable Redis.

POST /api/v1/workers/smoke-test with { "useQueue": false } proves the web process can run a safe workflow without the worker. Then retry with queue enabled.

Cancel a non-terminal run: POST /api/v1/runs/{id}/cancel (409 if already terminal). Re-run: POST /api/v1/runs/{id}/rerun (201, new run). Both accept Idempotency-Key.

Leases and crash recovery

Queued execution (runtime v2 checkpoints) takes a lease on the run: owner = worker job token, TTL WORKFLOW_RUNNER_LEASE_TTL_MS (default 300_000 ms). Heartbeat interval defaults to half the TTL (WORKFLOW_RUNNER_LEASE_HEARTBEAT_INTERVAL_MS).

• Another worker cannot steal a live lease (leased by another workflow runner).
• After expiry, a worker may acquire and apply crash recovery:
  • queued checkpoint → resume
  • running → retry up to WORKFLOW_RUNTIME_V2_MAX_RECOVERY_ATTEMPTS (default 3)
  • terminal → do not resume
  • waiting → only resume if the wait is due (timer); otherwise leave parked
• Losing the lease mid-HTTP node fails that node (Workflow lease lost).
• Side-effect nodes record an idempotency key {runId}:{nodeId}:external-effect so a retry should not double-send if the checkpoint already recorded the effect.

BullMQ job attempts (3) are separate from checkpoint recovery attempts.

If a worker dies without releasing the lease, wait for TTL (or the next worker after expiry). Do not delete Redis keys by hand in production without knowing you will orphan the Prisma checkpoint.

Failed BullMQ jobs

Settings → Workers separates recent unacknowledged failures from retained historical or acknowledged failures. Only a recent unacknowledged failure degrades worker health; an old retained job remains visible without making a healthy queue look unavailable. The default recency window is 24 hours.

Owner and Admin roles can inspect the workspace-owned failed jobs and choose Retry, Acknowledge, or Remove. Retry is accepted only while the job remains failed. Remove asks for explicit confirmation and cannot be undone. Each action is written to Audit Logs using the exact queue and job identity. Member and Viewer roles see only the workspace aggregate, not job IDs or failure detail.

Before retrying, confirm that the workflow's external mutations are idempotent or safely repeatable. Acknowledge a failure when it is understood and retained only as history. Remove it only after its run and recovery evidence are no longer needed. Jobs whose workspace ownership cannot be proven are omitted and cannot be acted on from the workspace.

Waits never resume

Timer waits: delayed job and scheduler sweepDueTimerWaits on each worker tick. If Redis is down but the web process still ticks schedules, the sweep can still resume timers. Event waits need POST /api/hooks/resume/{token} (single-use). Approvals: POST /api/runs/{id}/approval (session/member rules). waiting is not failed — keep polling GET /api/v1/workflows/{id}/status?runId=.

Review / deploy blocked

POST .../deploy → 409 workflow_review_not_synced when a GitHub repository is configured and derived review status is not synced. Create/update PR (POST .../review, Member+), merge in GitHub, then POST .../sync-from-repo (Admin/Owner). expectedHeadSha on review POST detects branch conflicts (409 if status is not open).

Drift reasons include proposal_branch_changed, draft_changed_after_proposal, base_and_draft_changed. Preview sync: GET .../sync-from-repo?ref=.

Import / placeholder / code

Validate: GET /api/v1/workflows/{id}/validate. Typical blockers: placeholder workflowCall n8n ids, unsupportedN8nNode, Code nodes with the operator flag off, the isolated Linux executor unavailable, or inline secrets, missing credentials, CLI nodes without a verified session, and live transports off. An isolated-code blocker requires the operator to enable FLUXUS_ENABLE_WORKFLOW_CODE=true or repair/redeploy the Linux policy; there is no in-process fallback. To check a deployment, run node --test scripts/workflow-sandbox/verify.cjs inside the web or worker container (a root shell spawns the fixtures as the node user, exactly like the application does).

Import originals: /api/v1/workers/n8n/subflows/import-originals. Bootstrap env credentials: /api/v1/workers/n8n/credentials/bootstrap-from-env. Then n8n smoke-test.

Confirmation checklist

• Workspace API key on all /api/v1/workers/* and execute (execute additionally requires workspace scope, not a user-only key).
• Session Workers UI uses /api/workers/* plus workspaceId.
• Doctor persistence.durable: true before trusting imports across restarts.
• After fixing Redis, re-run smoke-test; do not assume old queued jobs still exist (removeOnFail / restart).
• Treat the failed count as workspace-scoped. Inspect recent unacknowledged failures before deciding whether retained history needs retry, acknowledgement, or confirmed removal.
• Do not paste CLI OAuth tokens or decrypted credentials into tickets.

Next steps