Workers and readiness

Workflow runs either execute in-process (direct mode) or wait in BullMQ for a platform worker (queue mode). Settings → Workers shows the public status the UI and GET /api/v1/workers/status share.

Queue vs direct

REDIS_URL set → queue mode (queue.mode: "bullmq", enabled: true). The worker process (pnpm platform worker / Compose worker) consumes queue workflow-runs. Default concurrency is WORKFLOW_RUNNER_CONCURRENCY (default 4).

REDIS_URL unset → direct mode. POST /api/v1/workflows/{id}/execute runs inline and returns 201 with queued: false.

FLUXUS_E2E=true also forces inline execution even if Redis is configured (test harness). forceQueue is internal to smoke tests.

Execute with Redis: create a queued run, enqueue job id = run id, return 202 and queued: true. If enqueue fails, the run is canceled with reason Workflow queue enqueue failed before dispatch (deterministic runId retries are left queued). Jobs use 3 attempts, exponential backoff from 1s, keep 100 completed / 250 failed jobs.

Status and health

GET /api/v1/workers/status (bearer) or GET /api/workers/status?workspaceId= (session). Optional now= must be a valid date or 400.

Public status includes queue, persistence (durable vs memory), database/migrations, scheduler due counts, health (healthy | degraded | attention plus reasons[]), local AI CLI (no secrets), managed workspaces, agent runtime, side-effect transport names (env var names stripped), deployment readiness labels, active/queued/running counts, recent runs, latest smoke test, checkedAt.

Health turns attention when persistence is not durable, Redis/queue errors, due schedules piling up, required CLI nodes not ready, or deployment blockers. degraded covers milder mismatches (see reasons).

When FLUXUS_WORKSPACE_DERIVED_ACCESS=true, both web and worker also require the durable FLUXUS_WORKSPACE_SURFACE_READINESS_ARTIFACT produced by the same-release readiness audit. The artifact contains the checked authorization-inventory digest and completed cutover attestations; missing, unreadable, or stale artifacts fail startup closed even when environment attestations are present.

Persistence setup statuses: memory_only, database_unreachable, migrations_pending, ready. Hosted environments throw if Prisma is required and DATABASE_URL is empty.

Workspace Owners and Admins can inspect failed jobs in Settings → Workers, retry a still-failed job, acknowledge its retained history, or remove it after confirmation. Manual retries require a shared PostgreSQL database and the retry-claim migration on every web process. Concurrent requests admit one retry for the same failure; if the job fails again after another attempt, that new failure can be retried separately. Acknowledging or removing a job does not require a retry claim.

Local AI CLIs

Claude Code and Codex CLI blocks need a session-scoped CLI home on the worker/web image, not a pasted API key in the graph.

The workflow's Check and credential test report whether the saved workspace or user CLI session can authenticate. Model-catalog availability is a separate check: if the catalog cannot load or a selected model is unavailable, validation reports that issue while retaining the session's authentication result. Reachable called workflows are included when checking which CLI providers the deployed workflow needs; disconnected workflows are not. After a CLI login or credential check, refresh workflow validation to see the current readiness.

Choose a valid fixed model before checking the session: a missing or invalid model blocks credential preparation. Correct that model first; this error does not mean the saved account needs authentication again.

In Settings, signing in, confirming login and checking a session wait up to 15 seconds for a busy workspace credential. A running Command or workflow may still be using it. A conflict means the confirmation was not consumed: wait for that task to finish, then retry. This does not interrupt the running task or replace its credential. Completing login has its own progress state; the card refreshes its saved status without requiring a page reload.

If Claude Code provides a setup token instead of a browser confirmation code, paste it into the same confirmation field. Both Settings and the workflow block recognize the provider's versioned token format and use token capture; browser confirmation codes keep their existing callback flow. The server still validates the value and requires the same account-management authority and recent authentication.

v1 (bearer; workspace from the key):

Session cookie twins live under /api/workers/local-ai-cli/* and take workspaceId as query/body. Invalid kind / action → 400. Verify that is ready persists credentials from the CLI home; failure to persist returns 500. Token capture is Claude Code only (400 otherwise). Public payloads omit home paths and raw tokens.

Doctor

GET /api/v1/workers/doctor compact report: persistence, database, queue (mode/enabled/error, no Redis URL), scheduler counts, worker health, deployment missing labels, local AI CLI, managed workspaces, agent runtime, optional n8n bundle inventory (bundlePath query), blockers, warnings, nextActions. Optional now= same as status.

Session: GET /api/workers/doctor?workspaceId=&bundlePath=.

CLI equivalent: pnpm platform:doctor -- --strict (and --app-url, --workspace).

Smoke tests

POST /api/v1/workers/smoke-test { targetWorkflowId?, useQueue?, waitForWorkerMs? }. Creates a temporary safe workflow + due schedule, processes that schedule, waits for a terminal run (or waitForWorkerMs in queue mode). HTTP 200 passed, 202 attention (still queued, or target not live-ready), 500 failed. A run that has not been observed in a terminal state retains both the temporary workflow and schedule, including after a polling error: the schedule is its execution principal. Inspect the returned runId; remove the retained schedule and workflow only after that run has succeeded, failed or been canceled. A timeout alone does not cancel or finish the run.

Retained smoke resources have no automatic expiry. A failure while recording schedule bookkeeping can return 500 after a run has been accepted; use the returned resource IDs before deciding whether to repeat the check. If the run counter could not be recorded, the still-enabled schedule can admit another diagnostic execution. Check every execution of that temporary workflow before deleting its schedule and workflow. The diagnostic graph only maps synthetic input; it does not invoke the target workflow or external services.

targetWorkflowId only audits that workflow (credentials, env flags, CLI, code secrets, missing/placeholder/reconstructed subflows). It does not execute the target’s side effects.

n8n: POST /api/v1/workers/n8n/smoke-test { bundlePath?, useQueue?, waitForWorkerMs? } — import/sync audit + the same safe smoke. LuminAI aliases: /api/v1/workers/luminai/smoke-test (and matching credential/subflow paths).

Worker process

The platform worker starts only after database reachable, migration table present, no pending migrations, and Redis reachable if REDIS_URL is set. Invalid DATABASE_URL / REDIS_URL fails fast as misconfigured. Otherwise it retries with backoff until WORKER_STARTUP_TIMEOUT_MS (default 300s). Failure logs redact connection URLs.

Tick interval: SCHEDULER_POLL_INTERVAL_MS (min 1s, default 60s). Each tick: due workflow schedules, Confluye Command schedules, timer wait sweep, bounded recovery of stale queued/running workflow runs, A2A task sweep, pin-data retention (FLUXUS_PIN_DATA_RETENTION_DAYS, default 30, 0 disables), mailer/RSS polls, trigger outbox. File backup/GC ticks only in production. Command FIFO recovery redrives active queued command jobs when the command worker is up.

Incoming webhook messages have a separate dispatch loop that checks committed trigger deliveries every second by default (TRIGGER_OUTBOX_DISPATCH_INTERVAL_MS, minimum 1s). Slow maintenance or provider polling does not hold up that loop. The scheduler's outbox drain remains a recovery path.

Signed callbacks to product connections (a run that ended, an end user reply, a mail batch) are sent by their own loops every two seconds by default (RUN_CALLBACK_DISPATCH_INTERVAL_MS, minimum 1s), so a slow product never delays trigger delivery. A delivery is attempted up to 12 times. With the normal backoff (10 seconds, doubling up to 30 minutes) the 11 waits add up to about two hours (2 hours 12 minutes 30 seconds). A product that answers with Retry-After can make each wait longer, up to one hour, so a delivery that keeps asking for the most can stay pending for up to 11 hours (plus the time of the attempts themselves) before it is failed. So a callback that has not arrived after two hours may still be on its way; a product that needs the result sooner reads the run from the API. Mail batches have a loop of their own, because each one reads the mailbox first: a slow mailbox never delays the callback of a run or an end user reply.

Queued Telegram replies also dispatch at startup and every second by default (PLATFORM_NOTIFICATION_DISPATCH_INTERVAL_MS, minimum 1s; the older HUMAN_TASK_NOTIFICATION_DISPATCH_INTERVAL_MS setting is still accepted). Delivery remains durable and deduplicated, with the existing per-chat and workspace rate limits. These intervals control queue pickup, not total response time: workflow execution, provider requests, retries and a busy chat can add time.

Next steps

Native API execution ownership

Native API workflows use an invocation-specific lease owner and reserve one execution segment before dispatch. Duplicate delivery cannot clean up another active invocation. Native expired ownership and uncertain claimed segments are refused instead of blindly replayed; preparation may have failed with no HTTP request. Inspect the run and provider state before a deliberate new action. Committed waits resume only their exact continuation. These rules also protect native ownership with optional runtime-v2 projection disabled. See native recovery; the general retry controls do not reset a native reservation.

Cortex and n8n Public API worker prerequisites

Enable the selected workspace live transport (WORKFLOW_LIVE_CORTEX or WORKFLOW_LIVE_N8N_PUBLIC_API) and use current Human Task/checkpoint ownership with a live execution lease. Configuration never silently acknowledges a request. Metadata/schema availability and n8n edition/version/entitlement/scopes must be checked separately; a free-trial restriction is not bypassed.

The original run actor must confirm the exact resolved request with current workspace AND organization execution access. Selected encrypted account revision, published node/version, run input, live-policy generation, original task ID AND wait token and execution lease remain bound through one winning durable claim, settlement and fresh authenticated output. A replaced task never renews consent. One physical attempt is allowed, including workflow maxTries3; timeout, reset, rate limit or an uncertain used claim never triggers automatic resend, readback, redirect or refresh.

Names, IDs, tags, schema option labels, job/report fields and audit locations may contain personal data retained in ordinary run, checkpoint, Human Task history and downstream mappings or AI. The key uses existing credential encryption; this adds no raw-database encryption, automatic deletion/TTL, scanner or deletion guarantee. Local bounds are 64 KiB input, 1 MiB complete response, depth 32, 20,000 nodes, 4096 reads and thirty seconds for the whole invocation, rather than vendor guarantees. Real provider authentication, selected-instance permissions/schema/version/entitlement, durable DB/Redis recovery, native DNS/TLS/sockets, Review tutorials and real AI request interpretation remain untested; local production-owner tests use simulated dependencies.

n8n destructive Public API actions

The selected n8n Public API subset now has nine ordinary actions. Delete a credential (credential.delete, n8n_public_api_credential_delete), Delete an execution (execution.delete, n8n_public_api_execution_delete) and Delete a workflow (workflow.delete, n8n_public_api_workflow_delete) each send one destructive WRITE: DELETE /credentials/<credentialId>, /executions/<executionId> or /workflows/<workflowId>, prefixed by the saved API root. There is no query or body. Credential/workflow IDs are explicit 1–256 ASCII letters, digits, underscore or hyphen; execution IDs are positive safe decimal strings. No collection, wildcard or returned URL becomes a deletion target.

The existing encrypted manual API-key profile and unverified instance edition/version/scopes/entitlement/free-trial prerequisites still apply. All three require the initially unset schema-2 migration, raw-retention and one-attempt controls plus destructiveAcknowledged:true, then a separate original-actor Human Task showing the saved account, origin/root, exact DELETE path and named resource ID. Imported string "true", 1, null and invalid objects remain visible and unchecked until explicitly repaired; opening or saving another field never grants consent. Current workspace and organization authority, policy/account revision, original taskId and waitTokenId, immutable request and lease/claim/CAS checks still apply, including later authenticated OUTPUT.

A genuinely empty HTTP204 or HTTP200 response maps to data:{resource,id,response:"empty"}, with zero examined/returned records and outcomeVerified:false; it observes a response and does not prove deletion. HTTP200 expected metadata maps to response:"metadata" and bounded metadata, with an exactly matching target ID. Credential metadata is only id/name/type; execution data omits runData/workflowData/customData; workflow data omits parameters/settings/pinnedData/connections/credential references. JSON null, whitespace, malformed/unexpected bodies, wrong IDs and other statuses are withheld. There is no deleted:true, automatic readback, paging, polling, retry or uncertain-effect replay. Map node.<id>.json.data.id, .response or available .metadata.name/.status; names and IDs may contain PII retained in ordinary history and downstream mappings, without a new raw-database encryption, TTL/deletion or scanner guarantee.

Real deletion effects, provider authentication, durable DB/Redis, native DNS/TLS and matching-revision Review remain untested. Use only separately authorized disposable resources for live verification; existing credentials, workflows and executions are not test targets. Credential create, workflow create/update/activate/deactivate/getVersion and Tool/Trigger remain unavailable; nine actions do not establish full-node parity.

DynamoDB typed items

Enable WORKFLOW_LIVE_AWS_DYNAMODB and the workspace DynamoDB live preference only for authorized runs. Current workspace AND organization authority, the selected account/input, original public-resume task/wait/decision and actual acquired lease guard one physical claim, response, settlement and fresh OUTPUT. Expired ownership or an uncertain claim never permits replay. Live AWS, native transport, real DB/Redis restart and Review organization walkthroughs remain untested. See typed requests, examples and limits.

AWS Transcribe job requests and metadata

The partial AWS Transcribe subset supports four ordinary numeric-version-1 actions: transcribe.transcriptionJob.create, transcribe.transcriptionJob.delete, transcribe.transcriptionJob.get and transcribe.transcriptionJob.getAll. Transcript downloading, simplification, automatic language detection, unbounded pagination and Tool/Trigger forms are unsupported. This is partial service coverage, not full-node or provider parity.

Save an existing awsTranscribeManualSigV4V1 account with profile aws-transcribe-manual-sigv4-v1, partition aws, one of twenty pinned standard regions, declared twelve-digit account ID and manual access keys. Named fields and Advanced JSON use the same encrypted, write-only account. manual-static omits token and expiry; manual-temporary requires an explicitly supplied session token and actual UTC expiry, with more than ninety seconds remaining at execution. Save/update/reopen is local and unverified with zero provider HTTP/DNS. Test and bulk Test are unavailable. No ambient/system/role credentials, STS acquisition, refresh, discovery or custom endpoints are used. The declared account does not prove AWS identity.

Import exact n8n-nodes-base.awsTranscribe with numeric typeVersion:1, preserve original parameters and credential references, and repair the schema-2 input deliberately. Select the exact saved account and set the initially unset migrationAcknowledged:true and rawRetentionAcknowledged:true controls. Imported strings such as "true", numbers and null remain invalid, unchecked and preserved until an explicit click or repair. Unknown and inactive fields remain repair items; embedded credentials are refused before source-ledger persistence. Original get defaults to returnTranscript:true; that intent is unsupported and is never silently converted to metadata.

Create requires an explicit unique transcriptionJobName, authorized same-region s3://bucket/key media URI, mediaFormat, detectLanguage:false, supported languageCode and an explicit options object. The original v1 node has no media-format field, so imports require manual format selection. Supported original settings include channel identification, maximum alternatives/speaker labels (2–10), vocabulary name and a complete vocabulary-filter name/method pair. Channel identification cannot combine with speaker labels. Deliberately set the initially unset processingAcknowledged:true: AWS obtains the authorized media, processes speech, may charge costs and retains job/transcript data under its policies; the app never fetches the media URI. A QUEUED or IN_PROGRESS result remains pending. No polling or completed-transcription claim follows submission.

Delete requires the explicit job name and initially unset destructiveAcknowledged:true, then a separate original-actor Human Task showing the exact AWS request. An actual empty HTTP200 body yields empty operationData:{} with responseReceived:true and outcomeVerified:false. It is response evidence only; job deletion completion and transcript object removal are not verified. There is no fabricated success:true, hidden readback or automatic resend.

Get requires returnTranscript:false explicitly. It returns bounded typed job/status/time/media/transcript-location metadata in operationData. Transcript locations remain inert JSON strings, including on COMPLETED jobs: the app never fetches a provider-returned URI or simplifies a transcript. Get All requires explicit returnAll:false, limit 1–100, filters:{} or supported original status/name filters, and boundedPageAcknowledged:true. It returns one partial page envelope with TranscriptionJobSummaries and any actual top-level NextToken. Copy that token into nextToken only for a separate manually approved next request. A short or empty page does not prove exhaustive inventory; the original broken unbounded paginator is not reproduced.

Only the original run actor can approve the exact request within fifteen minutes. Current workspace and organization authority, selected account, published node/version, run input, resolved request, original task ID, wait token, stable durable decision and execution lease are checked through claim, dispatch, settlement and fresh OUTPUT. Replacement never renews consent. One claim permits at most one physical attempt despite workflow retries; timeout, revocation, disconnect, expired lease or uncertain settlement cannot authorize resend. Same-lineage failure cleanup grants no new authority and remains fenced by the real lease.

Local limits are 16 KiB request, 1 MiB whole response, thirty seconds, depth32/nodes20000 and strict unknown-field refusal. Malformed, oversized, unexpected, numerically mistyped or credential-reflecting responses fail closed. Useful job metadata, names and media/transcript locations enter ordinary workflow/checkpoint/Human history and may reach downstream AI. Accepted raw response bytes use the existing encrypted protected Workspace Files mechanism; this adds no raw-database encryption, automatic TTL, plaintext-scanner or deletion guarantee.

En español: selecciona una cuenta manual existente y la acción exacta. Corrige los campos importados y marca cada confirmación deliberadamente. AWS obtiene el archivo autorizado; la aplicación no descarga archivos ni transcripciones. Los trabajos pendientes siguen pendientes, eliminar solo acredita la respuesta y listar devuelve una página parcial con un token manual. Los metadatos y sus ubicaciones pueden conservarse en el historial y llegar a la IA posterior.

Local tests simulate provider/Prisma facts and use actual public resume, Human Task, checkpoint and output owners. Real AWS authentication/permissions/effects, native DNS/TLS, DB/Redis restart, real AI request interpretation and Review workspace/organization Help walkthroughs remain untested because those external environments and credentials are outside this worker's authority. See the official StartTranscriptionJob, DeleteTranscriptionJob, GetTranscriptionJob and ListTranscriptionJobs contracts.

AWS IAM identity and membership actions

The ordinary numeric-v1 IAM user/group subset covers twelve actions only. It uses the dedicated awsIamManualSigV4V1 account: profile aws-iam-manual-sigv4-v1, partition aws, signing region us-east-1, and fixed global https://iam.amazonaws.com/ over HTTPS443. Use named fields or Advanced JSON for manual static keys, or a supplied temporary token with its actual UTC expiry. Save and reopen are encrypted, write-only, local and unverified; Test is disabled. No ambient, system, role, metadata, STS, credential acquisition or refresh is used.

Exactly one requested Action is sent by form POST with Version 2010-05-08. Identity, access and membership changes require explicit acknowledgments and a separate original-actor Human confirmation. DeleteUser/DeleteGroup fail on provider dependencies; they never remove memberships, login profiles, MFA, policies or access keys implicitly. Membership add/remove are separate exact user/group operations. CreateUser supports path, tags and permissions boundary; updates support path and rename only. No policy, role, access-key or login-password CRUD is added. HTTP200 is provider response evidence, never verified effect completion.

Lists require explicit returnAll: false, limit 1–1000 and bounded-page consent. GetGroup also pages Users; IsTruncated and opaque exact Marker remain visible even when Users are omitted. Continue only with another separately approved request. No automatic paging, membership scans, discovery, loaders, post-read, retries, returned-URL fetching or redirects. Original helper, returnAll and includeUsers semantics that cannot be preserved remain visible for manual repair; unknown fields and invalid consent types survive export/reopen. Embedded credentials are refused before source-ledger persistence.

Typed user/group identity, ARN, path, creation time, tags, boundary and explicitly selected membership data plus an exact raw reference enter ordinary run/checkpoint/Human history and may reach downstream AI. Account encryption adds no raw database encryption, automatic TTL or scanner guarantee. A bounded XML decoder rejects DTD, external/custom entities, duplicate or wrong envelopes, malformed members, unknown fields/status and credential reflection before retaining raw data. Current workspace and organization permissions, account, input, policy, immutable original task/wait/decision and lease are rechecked through claim, send, response, settlement and fresh OUTPUT. Uncertain or expired claims cannot resend. Local bounds: 32 KiB request, 1 MiB response, 30 seconds. Real AWS, hosted Review and full node/provider parity remain untested.

AWS S3 ordinary bucket subset

The partial ordinary S3 subset covers only s3.bucket.create, s3.bucket.delete, s3.bucket.getAll and s3.bucket.search: four of twelve actions, with exact numeric source V1/V2 imports analyzed separately. Search is ListObjectsV2 object metadata, not a bucket-name search. Object upload/download/copy/delete, folder deletion, multipart, Tool and Trigger remain unsupported.

Save awsS3BucketManualSigV4V1 / aws-s3-manual-sigv4-v1 with partition aws, an explicitly known us-east-1 or eu-west-1 region, declared twelve-digit account ID and manual static keys or supplied temporary token plus actual UTC expiry. Temporary execution needs over ninety seconds remaining. Named fields and Advanced JSON use encrypted, write-only, unverified local Save/update/reopen with zero provider HTTP/DNS. Test, bulk Test, STS, system/role chains, refresh, custom endpoints and discovery are unavailable.

Every request requires schema2 and initially unset migrationAcknowledged, rawRetentionAcknowledged, regionAcknowledged and billingAcknowledged, all literal true, plus action-specific consent. Raw string "true", 1, null and objects remain visible and unchecked until a deliberate click. Then a separate original-actor Human Task approves the exact request. Current workspace and organization access, account, node, input, policy, original task ID/wait token/stable decision and real execution lease are checked through the one durable claim, send, response, settlement and fresh OUTPUT. Expired or uncertain claims never replay.

Create requires name, additionalFields:{} (or a matching known region) and name/global-namespace/security/write acknowledgments. It sends one PUT: empty body for us-east-1, matching XML LocationConstraint elsewhere. Only provider secure defaults are supported; all original ACL/grant/ownership/tag/object-lock intent stays for explicit repair. HTTP200 empty body yields response evidence with effectConfirmed:false; Location remains inert manual metadata. Creating an existing owned us-east-1 bucket can reset ACLs under AWS behavior; inspect the exact name and permissions manually before approval.

Delete requires name, empty-bucket/global-namespace/destructive acknowledgments and sends one DELETE. The bucket must already be empty of objects, versions and delete markers; nothing enumerates or deletes them or changes protection/ownership. Only HTTP204 empty body is accepted as response evidence, never confirmed deletion. A deleted global name may be reused by another account.

ListBuckets (getAll) needs explicit returnAll:false, limit:1..1000, filters:{} and boundedPageAcknowledged:true. This is a local cap, not the AWS maximum10000. Optional filters.bucketRegion must equal the endpoint region; filters.prefix filters bucket names. Output preserves actual ContinuationToken; copy it byte-for-byte into nextToken for a separately approved request. Unpaginated accounts with approved quota above10000 may be rejected; no auto-page or local slice is used.

ListObjectsV2 (search) needs bucketName, explicit returnAll:false, limit:1..1000, additionalFields:{}, bounded-page and Requester Pays acknowledgment. Supported typed options are prefix, delimiter, encodingType:url, fetchOwner, startAfter and requesterPays; the payer is a header. Dotted names use regional HTTPS443 path style. Output keeps bounded Contents, CommonPrefixes, KeyCount, IsTruncated, keys, ETags, sizes, storage classes, dates and actual NextContinuationToken; tokens and keys remain inert values. No content or returned URL is fetched. Empty XML containers and singleton/many entries normalize without fabricated rows.

Original versions keep exact parameters and credential references. V1 ignores the location region during signing; V2 forwards it and handles dotted search names differently. Both original location reads, null/us-east-1/EU assumptions, returnAll/default/limit and unsupported options require visible manual repair. No hidden GetBucketLocation/HeadBucket/readback, redirect, follow-up, polling or retry occurs. Local32KiB request/1MiB response/thirty-second bounds apply; malformed, duplicate, DTD/entity/XXE, wrong-envelope, oversized or credential-reflecting replies fail before accepted artifact retention. Useful metadata and tokens enter ordinary run/Human history and downstream AI; account encryption adds no raw DB encryption, TTL or scanner guarantee.

En español: selecciona una cuenta manual existente y una región conocida, corrige el origen V1/V2 y marca cada confirmación de forma deliberada. Crear y eliminar solo acreditan respuestas; listar devuelve una página parcial y buscar lista metadatos de objetos. Copia el token exacto para otra solicitud aprobada. No se descargan objetos ni se eliminan sus versiones.

Local tests simulate provider/Prisma facts. Real AWS credentials, permissions/effects, native DNS/TLS, DB/Redis restart, AI request interpretation and Review workspace/organization Help walkthroughs remain untested because they are outside this worker's authority. See CreateBucket, DeleteBucket, ListBuckets and ListObjectsV2.