Ingesting documents

Ingestion is how content gets into CRTX. You upload a file or submit a URL; a background worker parses it, splits it into chunks, embeds each chunk, and upserts the vectors into the collection's Pinecone namespace. Everything is asynchronous and observable — you get a job_id back immediately and poll for progress.

---

#Supported inputs

Limits: files up to 50 MB. File type is detected extension-first (browsers often send wrong MIME types for CSV/XLSX), then by MIME. Unsupported types return 415; oversized files return 413.

---

#Uploading

#From the UI

Open a collection and use the upload zone (drag-and-drop or file picker), or paste a URL. Progress shows as chunks_processed / chunks_total.

#Via the API

File upload (multipart):

curl -X POST "$BASE_URL/ingest/?collection_id=$COLLECTION_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@/path/to/report.pdf"
# → { "job_id": "..." }

URL ingestion:

curl -X POST "$BASE_URL/ingest/url" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/whitepaper", "collection_id": "'"$COLLECTION_ID"'" }'
# → { "job_id": "..." }

collection_id is optional. Omit it to ingest into your personal namespace (no collection), but note that documents ingested without a collection aren't listed or shareable — practically, always ingest into a collection.

URLs must be http/https and point to a public host. Private, loopback, link-local, and .local hosts are rejected (SSRF protection).

---

#Tracking a job

Poll the job endpoint until it reaches a terminal state:

curl "$BASE_URL/ingest/jobs/$JOB_ID" -H "Authorization: Bearer $TOKEN"

{
  "job_id": "…",
  "collection_id": "…",
  "source": "report.pdf",
  "status": "processing",       // queued → processing → succeeded | partial | failed
  "chunks_processed": 240,
  "chunks_total": 512,
  "error_message": null
}

You can only poll jobs you created (enforced by user_id).

---

#What happens inside a job

Download from Storage (PDF/Office/text) or fetch URL
      │  (3 tenacity retries w/ exponential jitter)
      ▼
Extract → list of chunks   [{ text, chunk_type, image_url }]
      │
      ▼
For each batch of 100 chunks:
   embed_documents(batch)               ← retried on OpenAI 429/5xx/timeout
   build vectors with deterministic IDs
   upsert to Pinecone (namespace = collection_id)   ← retried on Pinecone 5xx
   update chunks_processed
      │
      ▼
Persist chunk→vector mapping to document_chunks
Update collection_documents.chunk_count
Set final status + write ingest_logs row

Key properties:

• Batched. Embedding and upserts run in batches of 100 to stay under API limits and keep memory flat on large files.
• Resilient. Each external call (storage download, URL fetch, embedding, upsert) is retried up to 3 times with exponential jitter. A batch that still fails is recorded and skipped — the job continues and ends as partial rather than losing everything.
• Idempotent. Every vector ID is SHA-256(collection_id, source, chunk_index). Re-ingesting the same document upserts identical IDs, so Pinecone overwrites in place instead of duplicating. No cleanup step needed.
• Block-page aware. URL fetches that return a short "access denied / captcha / enable JavaScript" page with HTTP 200 are detected and rejected rather than ingested as content.
• Empty-document aware. A PDF with no extractable text or images, or a page behind a JS/login wall, fails with a clear message instead of creating an empty document.

---

#How each format becomes chunks

• PDF (native). Three passes: (1) tables extracted and serialized to GFM Markdown so structure survives; (2) body text extracted excluding table regions to avoid duplication; (3) embedded images above MIN_IMAGE_BYTES described by GPT-4o vision, uploaded to the images bucket, and stored as image_description chunks with an image_url.
• PDF (scanned). Detected by a low chars-per-page heuristic. Each page is rendered at 150 DPI and described by vision — effectively OCR-plus-understanding.
• Word / PowerPoint / Excel / CSV. Structured extractors preserve tables and schema. Spreadsheets get a schema-summary chunk plus per-row chunks so a question about one row retrieves just that row with its column headers.
• Text / Markdown / URL. Cleaned and split with a recursive character splitter (governed by CHUNK_SIZE / CHUNK_OVERLAP).

Image-description chunks are labelled [Figure/Image] in the model's context at query time, and render inline as sources in the UI.

---

#Deleting documents

Deleting a document (owner only) removes the original from Storage, deletes its Pinecone vectors (looked up via document_chunks), and removes the DB row (its chunk rows cascade away):

curl -X DELETE "$BASE_URL/collections/$COLLECTION_ID/documents/$DOCUMENT_ID" \
  -H "Authorization: Bearer $TOKEN"

Deleting an entire collection purges its whole Pinecone namespace in one call.

---

#Tips for high-quality ingestion

• Prefer native PDFs over scans. Native text extraction is faster, cheaper, and more accurate than page-level vision. Scans still work, but cost more.
• Chunk size is a retrieval trade-off. Larger chunks preserve more context per hit but dilute similarity; smaller chunks are precise but can fragment ideas. See Retrieval tuning.
• Ingest the source, not a summary. RAG answers are only as good as the content you give it. Upload the authoritative document.
• Re-ingest freely. Because vector IDs are deterministic, re-uploading a corrected file overwrites the old vectors cleanly.

Next: Querying & chat →