Evaluation & quality

The thing that separates CRTX from a black-box chatbot is that every query is scored. Without any ground-truth labels or human annotation, CRTX runs an LLM-as-judge over each interaction and records two metrics — faithfulness and context relevance — so you can trust individual answers and watch quality trends across a whole collection.

---

#The two metrics

Together they localize failures:

• Low faithfulness, high relevance → retrieval was fine but the model drifted or invented. Tighten the prompt or model behaviour.
• High faithfulness, low relevance → the model faithfully worked with bad context. This is a retrieval problem — see Retrieval tuning.
• Both low → the collection probably doesn't contain the answer, or chunking fragmented it.

---

#How scoring works

Scoring runs asynchronously after each answer streams, in the Arq worker, so it never adds latency to the user's response:

1. The query streams to the user and the exchange is handed to the worker (persist_query_result).
2. The worker persists the chat message, then calls gpt-4o-mini twice — once with a faithfulness rubric (question + context + answer) and once with a relevance rubric (question + context) — each returning strict JSON {"score": …}.
3. The scores, latencies, retrieval strategy, Top-K, engine, and the retrieved chunks are written to the rag_evals table.

If scoring fails (e.g. an OpenAI hiccup), the exchange is still saved and a scorer_error is recorded on the row — the score is simply null and excluded from averages. Using the cheaper gpt-4o-mini as the judge keeps evaluation costs a small fraction of generation.

The worker must be running for scoring to happen in the background. If Redis is unavailable, CRTX falls back to running the persist+score inline (degraded but correct) so chat history is never lost.

---

#The dashboard

GET /evals/{collection_id}/stats powers the Evaluation dashboard and returns:

{
  "total_queries": 128,
  "scored_queries": 124,
  "avg_faithfulness": 0.91,
  "avg_context_relevance": 0.86,
  "avg_total_latency_ms": 2450,
  "avg_retrieval_latency_ms": 180,
  "avg_generation_latency_ms": 2200,
  "trend": [
    { "date": "2026-07-01", "avg_faithfulness": 0.89, "avg_context_relevance": 0.84, "count": 40 },
    { "date": "2026-07-02", "avg_faithfulness": 0.93, "avg_context_relevance": 0.88, "count": 44 }
  ],
  "worst_queries": [
    { "id": "…", "question": "…", "faithfulness_score": 0.35,
      "context_relevance_score": 0.40, "total_latency_ms": 3100,
      "engine": "langchain", "created_at": "…" }
  ]
}

Three views to actually use:

• Averages — the headline health of the collection.
• Daily trend — is quality improving or regressing as you add documents and tune retrieval? Watch this after every change.
• Worst queries — the ten lowest-faithfulness questions. This is your bug tracker: each one is a real retrieval or grounding failure to investigate.

#Paginated raw records

GET /evals/{collection_id}?limit=50&offset=0 returns individual eval rows (question, answer, both scores, all latencies, engine, retrieval strategy, Top-K, scorer_error, timestamp) for export, auditing, or your own analytics.

---

#Turning evals into a quality loop

1. Baseline. Ingest a representative document set and run a batch of realistic questions. Note the average faithfulness and relevance.
2. Read the worst queries. For each, open the question and look at whether the right chunks were retrieved. This tells you if it's a retrieval or grounding problem.
3. Change one knob. Switch to mmr, enable reranking, adjust Top-K, or re-ingest with different chunking — see Retrieval tuning.
4. Re-run and compare the trend. The daily trend line shows whether the change helped. Keep it if relevance/faithfulness rose; revert if not.
5. Repeat. Quality is a loop, not a launch.

---

#What evals are and aren't

• ✅ A fast, label-free signal that catches hallucination and retrieval failure at scale.
• ✅ A regression detector — you'll see a dip the day a bad batch of documents lands.
• ⚠️ Not a substitute for human review of high-stakes answers. LLM judges are strong but imperfect; treat scores as a triage signal, not a verdict.
• ⚠️ Judge and generator share a provider — a systematic model bias could affect both. For critical deployments, spot-check with humans and consider a different judge model.

Next: Sharing & collaboration →