AI indexer

The indexer is a private FastAPI service. Go owns the public upload transaction; this service receives the committed upload UUID, reads the upload/object metadata, extracts text, writes chunks, and updates the durable processing job. It does not accept file bytes from clients.

Service authentication and base command

The protected routes inspect a standard Authorization header only when SERVICE_TOKEN is configured. Send Bearer <SERVICE_TOKEN> exactly. When the token is not configured, the dependency permits the request; production deployments should always set it and keep the service off the public internet.


export INDEXER_URL="http://127.0.0.1:8090"
export SERVICE_TOKEN="<private indexer token>"
export UPLOAD_ID="11111111-2222-3333-4444-555555555555"
 
curl --fail-with-body "$INDEXER_URL/health"

Exact HTTP contracts

Method and path	Required headers	Data to send	Successful response
`GET /health`	None	No body.	JSON object: `status` (`ok` or `degraded`), `service`, `database`, `storage_configured`, and `redis_configured`.
`GET /v1/index/jobs/{upload_id}`	`Authorization: Bearer $SERVICE_TOKEN` when configured	UUID path parameter; no body.	The existing processing-job row as JSON. Returns `404` with `detail: "AI processing job not found"` when absent.
`POST /v1/index/uploads/{upload_id}`	Service authorization when configured; `Content-Type: application/json` for a body	Optional JSON `{ "force": false }`.	The object returned by `index_upload`; `force: true` makes a completed upload eligible for re-indexing.
`POST /v1/index/process-due`	Service authorization when configured; JSON content type for a body	Optional JSON `{ "limit": 100, "include_done": false }`.	The object returned by `process_due`, covering eligible pending, failed, and stale jobs.

limit is optional but, if sent, must be an integer from 1 to 1000. force and include_done are booleans and default to false when omitted. Invalid service authorization produces 401; database failures in job operations produce 503 with FastAPI’s detail message.

Requests the Go API sends

After it has stored a valid multipart upload and committed its database row, the Go upload handler starts an asynchronous five-second callback to the indexer. Its body is exactly:


POST /v1/index/uploads/11111111-2222-3333-4444-555555555555 HTTP/1.1
Authorization: Bearer <INDEXER_SERVICE_TOKEN>
Content-Type: application/json
 
{"force":false}

An operations caller can force a known upload after confirming no active worker owns it:


curl --fail-with-body -X POST "$INDEXER_URL/v1/index/uploads/$UPLOAD_ID" \
  -H "Authorization: Bearer $SERVICE_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"force":true}'

For controlled recovery, run a bounded sweep rather than invoking a large unbounded job:


curl --fail-with-body -X POST "$INDEXER_URL/v1/index/process-due" \
  -H "Authorization: Bearer $SERVICE_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"limit":100,"include_done":false}'

Job semantics and supported upload classes

The indexer locks on the upload UUID in Redis, makes or resets the processing-job row, and replaces document chunks in a PostgreSQL transaction. A completed job with chunks is skipped unless force is true. A stale processing job becomes eligible after JOB_STALE_MINUTES; failed jobs are included by the due-job operation. The job transitions to done or failed and carries its source-defined error state rather than returning a fabricated success.

Upload category	Eligible formats
Documents	PDF, DOCX, DOC, TXT, Markdown, RTF
Presentations	PPTX and PPT
Spreadsheets	CSV, XLSX, and XLS

The worker skips images, video, audio, archives, and other unsupported binaries. It downloads with MAX_DOWNLOAD_BYTES, chunks text at a default 1,200 characters with 150-character overlap, and can preserve chunks with zero vectors if the optional embedding provider is unavailable. This makes the extraction job observable without claiming a provider result that did not occur.

Consumer pattern

The public client should poll Go, not the private indexer:


curl --fail-with-body "$KIBEGI_API/ai/status/$UPLOAD_ID/" \
  -H "Authorization: Bearer $ACCESS_TOKEN"

Go returns the durable processing job when one exists; otherwise it returns the safe initial state { "status": "not_started", "chunks_created": 0 }. This keeps private service tokens and object-store topology outside browser and mobile clients.