Operations

Mailer and trigger recovery

Reconcile inbound replays, reply reservations, poller leases, and the trigger outbox.

Mailer and RSS do not execute workflows inline. They persist a source event, enqueue a trigger outbox row, and rely on the platform scheduler tick to poll due sources and drain the outbox.

Keep the scheduler running

pnpm worker:scheduler calls runPlatformSchedulerTick on SCHEDULER_POLL_INTERVAL_MS (default 60s, minimum 1s). Each tick:

Polls due email groups (pollDueEmailGroups) and RSS triggers (pollDueRssTriggers) in parallel (concurrency 4).
Reconciles Prisma trigger sources (reconcileTriggerSources) when the Steel Engine Prisma store is enabled for mailer or RSS.
Drains the trigger outbox (processTriggerOutbox).

pnpm worker:scheduler:once runs a single tick. If the worker is down, inbound signed/generic mail can still persist messages, but polling and outbox delivery stall.

Inbound signed replay

Signed inbound claims mailer-inbound-replay:{routeId}:{sha256(timestamp+signature)} for five minutes, with a 60-second processing lock.

Symptom	What to do
`409` replay	The signature already completed. Do not rotate the secret; send a new timestamp/signature if the provider retries a new body.
`409` in_progress	Another worker holds the lock. Wait for the lock or expiry; do not treat it as delivered until `202`.
`500` after persist	If the reservation status is already `completed`, the handler still returns `202`. If completion cannot be marked, the reservation is released and the client may retry.
Transient ingest error	The handler releases the pending claim so a later signed POST can retry.

Generic inbound (/api/v1/mailer/inbound) has no HMAC replay store. Dedup is route + provider + externalMessageId at ingest.

Reply reservations

Do not retry POST .../messages/{id}/reply with the same Idempotency-Key after MAIL_REPLY_PENDING or MAIL_REPLY_FAILED. The provider may already have accepted the send.

GET /api/v1/mailer/reply-reservations/{id} as Admin/Owner.
Confirm the mailbox: look for the outbound message id stored on a sent reservation.
Only if you accept a possible duplicate send, PATCH { "action": "authorize_retry" }.
Replay the same key and payload. A different payload returns MAIL_REPLY_IDEMPOTENCY_MISMATCH (not reconciliable).

authorize_retry is refused while a completion is in flight, or while a pending lease is still valid (2 minutes). Audit action: mailer.reply.authorize_retry.

If OAuth refresh persistence fails after a successful send, the reply stays successful and providerMetadata.credentialRefreshPersistence may be "failed". Audit: mailer.credential.refresh_failed. Re-auth the credential; do not treat the user-visible send as failed.

Email and RSS poller leases

Due pollers are claimed for 30 minutes. A second tick skips a claimed group/trigger (claimToken null). After success or recorded failure, nextPollAt moves forward by intervalSeconds and lastError is set or cleared.

Symptom	Recovery
Gmail History 404	Poller restarts bootstrap from a fresh profile `historyId`.
Transient attachment/download error	Cursor is not advanced; the next due poll retries.
Permanent attachment rejection	Message is dead-lettered without attachments; poll continues.
RSS HTTP error / >2 MB	`lastError` set; interval still advances.
RSS `in_progress` delivery	Item skipped; ETag/Last-Modified not updated so the feed is re-read.
RSS dispatch failure	Target-bound events retain their activation for outbox recovery. Legacy unbound reservations release for polling retry.
Stale poll claim	After 30 minutes the lease expires and another tick can claim it.

DWD poll/reply failures (config disabled or untested, subject mismatch, token exchange) fail the poll or reply closed. Re-run POST /api/v1/google-dwd/{id}/test and enable only after testStatus: "succeeded". Do not paste service-account JSON into logs.

Outbox reconcile

When Prisma mailer/RSS stores are active, reconcileTriggerSources finds inbound messages with triggerRequired: true and RSS deliveries still marked required, enqueues missing outbox rows (email externalEventId is {provider}:{externalMessageId}; RSS uses the item externalId), then clears triggerRequired. processTriggerOutbox then creates the workflow run or Command delivery.

Target-bound source records retain the activation accepted at ingress. Reconcile does not choose the newest deployment. If that generation has been stopped, outbox admission discards the stale event before starting a workflow or billing it. Publishing a successor cannot revive it. Due polling excludes stopped RSS destinations and mail groups with no enabled route destination.

If a run never appears after 202 inbound or a successful poll, inspect outbox rows and lastError on the polling group or RSS trigger, then run a scheduler tick. Do not POST a new signed inbound with the same signature; use a new timestamp or wait for reconcile of the stored message.

Confirmation checklist

Inbound: 202 plus outbox[]. deduplicated: true means the message id was already stored.
In-app test copy: “queued” vs “was already queued” from resolveMailerInboundTestResult.
DWD: successful test before enable; then bind the Gmail route subject.
Reply: 201 first send, 200 replay; 409 + reconciliable needs Admin authorize_retry.
RSS: GUID/externalId completed once; lastError null after a clean poll.

Next steps

Mailer

Inbox and reply model.

Scheduled Tasks

The same scheduler worker also fires cron schedules.

If one destination is stopped or loses AIMS approval during a shared mailbox poll, other independently authorized routes can still receive the message. The blocked destination receives no new execution from that poll; restoring approval does not replay skipped messages automatically.