Operations

Knowledge Sync Recovery

Symptom-to-verification runbook for knowledge connector sync.

Use this when a connector stays syncing/failed, a run never leaves queued, or documents do not appear after a reported success. There is no background Knowledge Sync worker: recovery is inspect HTTP state, fix config or credentials, then replay the same Idempotency-Key on POST /api/v1/knowledge-connectors/{id}/runs.

Read Knowledge Connectors and the API reference first.

Verify first

Confirm the API key is workspace-scoped. Personal/copilot keys return 403 and never touch sync state.
GET /api/v1/knowledge-connectors/{id} and GET .../runs/{runId} in the same workspace as the key. Foreign ids are 404.
On GET /api/v1/knowledge-connectors, check data.providers for that providerId: certified: true and capabilities containing knowledge.sync.
Compare connector.status, connector.lastError, connector.cursor, run.status, run.error, run.upsertedCount, and run.tombstonedCount. Counts are the last applied batch, not a live byte meter.
If the HTTP POST .../runs returned an error envelope, still GET the run. Execute persists failed then throws; the HTTP status is not 201.

Symptom table

Symptom	Verify	Recover
`401` / `403` on any connector route	Bearer missing/invalid, or key `scope` is not `workspace`	Use a workspace API key. Do not retry with a personal/copilot key.
`404` on a known connector	`GET` uses the key's workspace, not query `workspaceId`	Call from the workspace that owns the connector.
`400` `validation_failed` on create with `providerId: "notion"` (or any non-allowlist id)	Allowlist is `drive`, `confluence`, `slack`, `github`	Recreate with a registered provider.
`400` `idempotency_key_invalid`	Mutation without `Idempotency-Key`	Send a valid key. Required on create/update/delete connector, create run, and cancel.
`400` on `POST .../runs` mentioning `records`, `cursor`, or `defer`	Body keys other than `options`	Send `{}` or `{ "options": { ... } }` only. There is no deferred dispatcher.
`409` `Provider {id} has not passed certification.`	`listProviders` shows `certified: false` / empty `capabilities`	Do not queue. The process default registry certifies the four built-ins. An ad-hoc uncertified registry cannot sync.
`409` `setup_required` and `lastError` about a missing credential	`config.credentialId` unresolved, no secret, or credential removed mid-sync	Recreate/reconnect a workspace-scoped credential and PATCH `config.credentialId`. Then replay the same run key.
`409` `setup_required` about workspace scope	Credential `scope` is not `workspace`, or `workspaceId` mismatch	Use a credential stored in this workspace. Fetch does not run.
`409` `setup_required` about credential type	Type does not match the provider allowlist (case/punctuation ignored)	Drive: `googleDriveOAuth2Api`, `googleOAuth2Api`, `googleWorkspace`. Confluence: `confluenceApi`, `jiraSoftwareCloudApi`, `jiraCloudApi`, `jira`. Slack: `slack`, `slackApi`, `slackOAuth2Api`. GitHub: `github`, `githubApi`.
`409` `setup_required` listing missing scopes	Secret is JSON with `scope`/`scopes` that omit a required grant	Reconnect the OAuth credential. Drive accepts `auth/drive` as covering `drive.readonly`. Non-JSON secrets skip the scope check.
`409` `Knowledge sync run is not queued.` on execute	Another run holds the connector lease, or this run is `cancelled`/`succeeded`/`failed`	GET both runs. If another is `running` and the lease is live, wait or cancel that run. If this run is `cancelled`/`failed`, replay the same `Idempotency-Key` to re-queue (`retryTerminal`).
Connector stuck `syncing`, run `running`, no progress	Lease default is 5 minutes; heartbeat renews at one-third	Wait until `leaseExpiresAt` is in the past, then POST the same idempotency key again. A newer start fails the stale run with `Knowledge sync lease expired and was recovered by a newer run.`
`409` `Knowledge sync lease was lost during provider fetch.`	Heartbeat `renewRunLease` returned false mid-fetch	Provider abort is expected. Replay the same key after the stale lease expires. Cursor is unchanged if apply never ran.
`409` idempotency reused with different input	Same `Idempotency-Key`, different `options`	Use a new key, or replay the original `options`.
`400` `Stored … cursor is invalid`	Filter set changed (`fileIds` / `spaceIds` / `channelIds` / `repositories`+`pathPrefixes`) so `selectionKey` no longer matches	PATCH filters, then either accept a fresh cursor by clearing stored cursor via a new connector, or restore the previous filter set. Do not inject a caller cursor on `POST .../runs`.
Drive/Confluence `… exceeded the Knowledge Sync fetch budget.`	`addItem` failed; those providers throw instead of returning a partial page	Raise `maxItems`/`maxBytes`/`maxPages`/`timeoutMs` within bounds, or shrink `fileIds`/`spaceIds`.
Slack empty document with `contentTruncated`	Single message JSON larger than `maxBytes`	Raise `maxBytes` (max 50 MiB) or accept the compact record (`text` empty, `originalBytes` in metadata). Mid-channel cursor is valid; next run resumes.
GitHub `tree … was truncated; narrow pathPrefixes`	Recursive tree `truncated: true`	Set or tighten `pathPrefixes` (max 50 unique strings).
GitHub `content exceeded …` on the first record	First blob larger than `maxBytes`	Raise `maxBytes` or exclude that path. Later records pause and persist `progress` on success.
`409` `Cannot cancel a {status} …`	Run already `succeeded`, `failed`, or `cancelled`	Cancel is not idempotent at the service layer. GET the run; do not retry cancel.
HTTP error after a long POST, documents unchanged, cursor still `null` or previous	Apply is one transaction; cursor advances only after persist	Treat as failed apply. GET the run. If `failed`, replay the same key. Do not assume partial documents.
`409` `Knowledge sync document ownership changed during persistence.`	Upsert row count did not match the batch (workspace/connector ownership)	Stop retries until ownership is consistent. Documents are keyed by `(connectorId, externalId)`.
`409` duplicate `externalId` in a batch	Provider batch contained duplicate upserts	Inspect provider output; built-in normalize already collapses by `externalId` (tombstone wins).
Run `succeeded` but RAG is keyword-only	Upserts set `indexingStatus` to `pending`	Wait for indexing. See Knowledge Base. Re-index the document if it stays keyword-only.
`nextSyncAt` in the past, no automatic run	`nextSyncAt` is stored only	Trigger `POST .../runs` manually.
Shared credential: second connector saw a stale secret	Concurrent fetches serialize on `workspaceId:credentialId`	A compare-and-set `conflict` does not fail the run. `not_found` does. Re-resolve the credential if the secret was deleted.

After a failed run

Read run.error and connector.lastError (they should match after a leased failure).
Fix the cause (credential, filters, budget, lease).
POST .../runs with the same Idempotency-Key and the same options.
Confirm run.status is queued then, on the same HTTP call, succeeded (or GET if the client timed out).
Confirm connector.status is succeeded, lastError is null, and cursor moved only if the batch returned a cursor.
Confirm documents: upserts are enabled, linked to the knowledge base, and indexingStatus is pending. Tombstones are disabled and trashed (trashedAt set).

What not to do

Do not POST caller records or cursor. The service fetches from the certified provider.
Do not expect a second in-flight execute on the same connector to steal the lease while it is live.
Do not treat a cancelled run as startable; re-queue through the same idempotency key.
Do not edit docs-site/docs.json or application source to “register” a fifth provider. The registry rejects unknown ids.