Research
Durable bounded HTTP(S) research runs that persist untrusted snapshots and a Markdown result document.
A Research run fetches caller-supplied public URLs through the Research egress guard, stores canonical source snapshots, and writes one Markdown document. Output is labeled untrusted: each snapshot and the result document start with a caution that web pages are data, never instructions.
HTTP surface: /api/v1/research. Agents also use preview capability research.run (confirmation
and idempotency required) and the related web.search / web.read / web.crawl tools (same
egress, no durable run).
Create a run
POST /api/v1/research requires a workspace API key and Idempotency-Key.
| Field | Rules |
|---|---|
query | Required, trimmed, 1–2000 characters. |
sourceUrls | Required, 1–20 http:// or https:// URLs. |
limits | Optional object. Missing keys use egress defaults. |
| Limit | Default | Create max (schema) | Egress default if omitted |
|---|---|---|---|
maxDomains | — | 8 | 8 |
maxPages | — | 20 | 20 |
maxDepth | — | 2 | 2 |
maxBytesPerPage | — | 2 MiB | 2 MiB |
maxTotalBytes | — | 20 MiB | 20 MiB |
maxDurationMs | — | 30 000 | 30 000 |
Stable run id: research_ plus the first 32 hex chars of sha256(workspaceId + NUL + key). Same
key and same query / sourceUrls / limits returns 200 { deduplicated: true, dispatch: { dispatched: false } } without a second enqueue. Same key, different payload: 409 conflict.
First create: 202 { run, deduplicated: false, dispatch }.
Dispatch
If REDIS_URL is set, the job is added to BullMQ queue research-runs (jobId = run id,
3 attempts, exponential backoff from 1s, removeOnComplete: 100, removeOnFail: 250). Worker
concurrency: RESEARCH_WORKER_CONCURRENCY (default 4). If queue add throws, the run is marked
failed (Research dispatch failed before queue acceptance.) and the error is rethrown.
If REDIS_URL is unset, enqueue runs the processor in the same request (dispatch.mode: "direct", queued: false). Tests that fetch blocked hosts therefore often return 202 with
run.status: "failed" immediately.
Store: Prisma when shouldUseDurablePrismaStore("research") is true; otherwise a process
in-memory store (lost on restart).
State
status | Meaning |
|---|---|
queued | Created, not yet markRunning. |
running | Claimed from queued with cancelRequestedAt null. |
succeeded | Result document persisted; resultDocumentId set. |
failed | Egress, dispatch, or processor error. error is the message. |
canceled | Cancel requested while queued or running. |
Prisma document id is research_document_{runId}, mimeType: text/markdown, tags ["research"],
sourceUri: research:{runId}, indexingStatus: pending, status: ready, enabled: true. Name:
Research - {query truncated to 120}.md. In-memory complete uses the same id prefix in a separate
document map (not the Docs HTTP API unless Prisma is in use).
Cancel (POST .../{id}/cancel, idempotency required) sets canceled for queued/running,
aborts an in-process controller, and removes a BullMQ job still waiting / delayed /
prioritized. A terminal run is returned as-is (200); the test suite shows cancel of failed
still returns failed.
Egress guard
Fetches use profile research (SSRF / DNS allowlist), User-Agent: FluxusResearchBot, and
robots.txt (max 512 KiB). Allowed page types: text/html, application/xhtml+xml, text/plain.
HTML is stripped to text; scripts are not kept as instructions. Duplicate canonical URLs in one
run are skipped. Canonicalization lowercases host and sorts query params.
Private/link-local URLs are blocked by egress (a loopback sourceUrl fails the run).
Authorization
Workspace API key only. 401 without a valid bearer. 403 for personal/copilot keys. Foreign run
ids: 404. Workspace id is always the key's workspace.