Operations

Research recovery

Symptom-to-verification for durable Research runs.

Use this when a run stays queued, finishes failed, or the Markdown result is missing. There is no Knowledge-Sync-style terminal retry on the same create key. Read Research and the API reference first.

Verify first

Workspace API key only (401/403 otherwise). Foreign ids are 404.
GET /api/v1/research/{id}: status, error, resultDocumentId, sources[].
On create, read data.dispatch. mode: "direct" means the fetch ran in the HTTP request (REDIS_URL unset). mode: "bullmq" means a worker must process research-runs.
Prisma vs in-memory: if the process is not using the Prisma research store, runs and documents do not survive restart and may be invisible to Documents.
If create returned 202 with status: failed, the run is already persisted; GET it. Do not assume HTTP 202 means success.

Symptom table

Symptom	Verify	Recover
`401` / `403`	Bearer / key scope	Workspace API key.
`400` `idempotency_key_invalid`	Create/cancel without key	Send a valid `Idempotency-Key`.
`400` on list	`status` not in the enum, `limit` not 1–100, bad cursor	Fix query. Default list limit is 25 when `limit` is omitted.
`409` idempotency belongs to a different request	Same key, different `query` / `sourceUrls` / `limits`	New key, or replay the original body.
Replay returns `200` `deduplicated: true` and still `failed`	Dedup does not re-enqueue	Gap: use a new idempotency key to create another run.
`dispatch.mode: "bullmq"` but run stays `queued`	Worker, `REDIS_URL`, queue `research-runs`, job id = run id	Process/retry the BullMQ job (up to 3 attempts, exponential backoff from 1s). Processor failures call `store.fail`.
`Research dispatch failed before queue acceptance.`	`queue.add` threw; run already `failed`	Fix Redis; create with a new key.
`failed` with robots / HTTP / content-type / limit / SSRF message	Egress guard	Use public `https` URLs allowed by `FluxusResearchBot` robots, HTML/XHTML/plain text, within `limits`. Loopback/private URLs fail.
`canceled` unexpectedly	Cancel route or in-flight abort	Create a new run. Cancel of `queued`/`running` is terminal.
Cancel of `failed` still `failed`	Cancel only transitions `queued`/`running`	Expected. HTTP `200` with current status.
`succeeded` but no Docs UI document	In-memory store, or Prisma document id `research_document_{runId}`	Prisma: `GET /api/v1/documents/{resultDocumentId}`. Confirm `indexingStatus` may be `pending`.
Result missing untrusted caution	Document should start with the caution marker	If missing, treat as incomplete persist; re-run with a new key.
`snapshotFileId` always null	Executor does not write Workspace Files	Expected. Snapshots live in `snapshotText` on the run.
Agent `research.run` vs HTTP	Same store/dispatcher	HTTP also wraps `withApiIdempotency`; agent tool requires `idempotencyKey` in tool input.
Expected crawl from `sourceUrls`	Executor depth is always 0	Use capability `web.crawl`, or pass every URL in `sourceUrls` (max 20).

After a failed run

Read run.error.
Fix URL allowlisting, robots, content type, or limits (within schema maxima).
POST /api/v1/research with a new Idempotency-Key and the corrected body.
Confirm status: succeeded, resultDocumentId set, sources canonical URLs, and (Prisma) document tags include research.
If using BullMQ, confirm the worker is running before expecting queued → running.

What not to do

Do not replay the original create key expecting a new fetch after failed/canceled.
Do not POST caller HTML or a document body; the service only fetches sourceUrls.
Do not treat web.* capabilities as substitutes for the durable run’s result document.
Do not assume maxDepth causes the durable run to spider; it does not.