The Retrieval Lab answers the question every accuracy switch raises: did that actually help? It runs a fixed set of benchmark questions through a chat's real retrieval configuration — same profiles, same switches, no answer generation, no caching — and scores how well the right content came back. Find it under Tools → Retrieval Lab.
- The cases editor holds your benchmark questions as JSON — save it, or import/export the set as a file.
- The run form executes every case against a chat (by ID) and reports the metrics.
- Export cases JSON downloads the set — it contains only your questions and expectations, never indexed content or credentials, so it's safe to share with support.
Writing cases #
Each case is a question plus what a correct retrieval looks like:
query— the question, phrased the way a visitor would ask it.required_evidence— phrases that must appear in the returned content ("30-day refund period").expected_content_ids/expected_source_ids— alternatively, the exact posts or sources that should match.conflicting_entities— names that signal a wrong match ("Model Plus" when the question is about Model Pro).parent_rollup— optionally force whole-document or chunk-level results for this one case.
Write eight to fifteen cases covering your greatest hits: exact SKUs and error codes, vague follow-up phrasings, questions about easily confused products, and one or two questions your content genuinely doesn't answer (those should come back empty).
Reading the report #
- Hit@K / Recall@K — how often, and how completely, the expected evidence appeared in the top results.
- MRR — how high the first correct result ranked; 1.0 means it was always first.
- Entity conflict rate / empty rate — how often wrong-product content or nothing at all came back.
- Each case row also shows full score provenance — dense rank, keyword rank, fused score, and the entity decision per result — so you can see why a case passed or failed.
Baselines #
Type a name in Baseline before running: the first run under a name is saved as the fixed reference, and every later run with the same name reports deltas against it instead of overwriting it. Save a before-accuracy baseline first, then compare after each switch you enable.
The same runner is available from the command line for scripted checks:
wp wyregraf evaluate --chat=123 --baseline=before-accuracy
What to do next #
Work through the rollout order in Improving Answer Accuracy, re-running your cases at each step.
