Operations & scaling

How CRTX behaves in production: the processes, what runs where, what to monitor, and the known limits to address before scaling from tens to hundreds/thousands of concurrent users.

---

#The processes

Run the worker. The Procfile only defines web:; the worker must be started as its own service/replica. This is the most common production mistake — verify it.

---

#Concurrency model

The API is asynchronous, but with one uvicorn process = one event loop, a single blocking call stalls every in-flight request (including token streaming). CRTX mitigates this in the hot path:

• The /query/ endpoint offloads its blocking Supabase calls (access check, config load, history load) to threads before streaming.
• Retrieval (embedding + Pinecone) runs in a thread so the event loop stays free while tokens stream.
• Per-query eval scoring + persistence is pushed to the Arq worker, off the web process entirely.

To actually scale, run multiple API workers/replicas:

uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 4

or horizontally scale the container/dyno. Module-level singletons (Supabase/OpenAI/ Pinecone clients, the cached Pinecone index, the JWKS client cache) are per-process and safe under this model.

---

#Observability

Every query and ingest emits a structured JSON log and persists a row:

• query_logs — per query: request_id, collection_id, user_id, retrieval strategy, Top-K, embedding/retrieval/generation latency, time_to_first_token_ms, total latency, chunks retrieved, retrieval scores, prompt/completion tokens, model.
• ingest_logs — per ingest: job_id, source, load/embedding/upsert latency, chunk count, success flag, and error detail.
• ingest_jobs — live job status (queued/processing/succeeded/partial/ failed) with chunks_processed/chunks_total for progress polling.
• rag_evals — per query: faithfulness + context-relevance scores, latencies, retrieval strategy, Top-K, scorer_error.

What to watch:

Ship these logs to your platform's log drain (Railway/Vercel/CloudWatch) and build alerts on job failure rate and eval-score drops.

---

#Resilience built in

• Retries. Ingestion retries storage downloads, URL fetches, embeddings (OpenAI 429/5xx/timeout), and Pinecone upserts (5xx) up to 3× with exponential jitter (Tenacity).
• Partial success. A failed batch is recorded and skipped; the job ends partial instead of losing everything.
• Idempotent ingestion. Deterministic vector IDs mean re-runs overwrite in place — safe to retry a whole document.
• Graceful streaming failures. OpenAI errors mid-stream emit an error SSE event; client disconnects are logged and stop cleanly.
• Redis-down fallback. If Redis is unavailable, ingestion is disabled (503), but query persistence/scoring falls back to running inline so chat history is still saved (degraded, not lost).

---

#Known scaling limits (address before high load)

From the scaling audit — the architecture is sound; these are the sharp edges:

1. Missing indexes. collections.user_id and collection_members.user_id aren't indexed, so the dashboard's collection list does sequential scans. Add:

   CREATE INDEX IF NOT EXISTS idx_collections_user_id ON collections (user_id);
   CREATE INDEX IF NOT EXISTS idx_collection_members_user_id ON collection_members (user_id);
   

2. list_users() full-table read. The members/shares views call Supabase list_users() (all users) and filter in Python — O(total users) per view, and it silently drops people past ~50 total users (pagination). Fix by looking up only the specific candidate ids.
3. Single uvicorn process. Run --workers N / multiple replicas so one process isn't the ceiling.
4. get_eval_stats reads up to 500 full rows and aggregates in Python. Push aggregation into SQL (a view/RPC) when eval volume grows. Lower severity (dashboard endpoint, not per-query traffic).

Things that already scale fine at ~1000 users: the JWKS cache, the cached Pinecone index, module-level client singletons, and the Arq ingestion pipeline itself.

---

#Capacity notes

• Vector store. Each collection is a Pinecone namespace; growth is horizontal and absorbed by Pinecone.
• Redis. Sized for the job queue (and future rate limiting). Modest.
• Storage. Originals live in the private documents bucket; extracted figures in the public images bucket. Grows with corpus size.
• OpenAI throughput. Your OpenAI rate limits are the practical ceiling on concurrent generation and ingestion embedding. Request higher limits before a big rollout; the built-in retries smooth transient 429s but won't create capacity.

See also: Configuration for the knobs, and Self-hosting for deployment topologies.