Ocean Search Testing

Source authority × phrase fidelity — the scoring model behind SifterSearch's 491-fixture ocean test suite, adapted from the ocean-search-testing framework by David Hunt.

Core Scoring Model

Search quality is the product of two independent dimensions: source authority (how reliable is this document as a source?) and match quality (how well do the user's words appear in this passage?).

The combined score = authority_score + match_score (maximum 8):

Expected result ordering

The ideal ranking is by combined score descending. The critical insight: a primary source with some-words match (3+2=5) should rank equal to a supplemental source with an exact-phrase match (0+5=5), and above a general source with all-words match (1+3=4). Authority wins ties.

Document Authority Weights

Every document in the corpus receives an authority weight at index time. The original framework uses a 1–8 scale (1 = highest); SifterSearch maps this to a 0–10 scale (10 = highest). Key assignments:

Success Metrics

Ordering error (original)

For each query over N books, compute the ideal result order (by combined score descending). Ordering error = how many positions each result is displaced from ideal. Pass threshold: median ordering error ≤ 5% of N.

This is stricter than precision@K: it penalizes any out-of-order result, not just results that fall outside the top K.

SifterSearch adaptations

• Precision@10 — did the expected document appear in the top 10 results?
• MRR — Mean Reciprocal Rank; penalises results that appear late in the list
• Text check — does the matching passage contain the expected phrase?
• Anti-test — top result must NOT be from a forbidden secondary author

Performance target

The original framework requires < 200ms per query on a dedicated Elasticsearch cluster. SifterSearch targets p50 < 500ms, p95 < 2000ms under normal load (Meilisearch on shared hardware).

SifterSearch Implementation Gaps

1. Phrase match boosting (high priority)

Queries of 4+ words should trigger a dedicated exact-phrase Meilisearch pass (matchingStrategy: 'all') with a 3× score boost relative to the vector similarity score. Without this, the hybrid search may rank a semantically related passage above the verbatim source.

Fix location: api/lib/search.js:hybridSearch — add phrase detection and a boosted keyword-first pass when the query matches the phrase pattern.

2. Cross-tradition authority normalization (medium priority)

Currently each tradition calibrates its own 0–10 authority scale independently. For cross-tradition queries, "primary Christian" and "primary Bahá'í" must rank equally. Implement four cross-tradition tiers (A/B/C/D) that supersede the per-tradition scale when no religion filter is active.

3. Ordering error metric (low priority)

MRR already captures most of the information in the ordering-error metric. Adding it to score-search.mjs would allow direct comparison with the original framework's results, but is not required for search quality improvement.

Running the Battery

# Ocean suite (491 fixtures from David Hunt's framework)
npm run test:search-quality:ocean

# Core + ocean combined (543 fixtures total)
npm run test:search-quality:all

# Filter by category
node tests/quality/score-search.mjs --ocean --category=phrase-match

See the Search Quality Battery page for current scores and the failing tests breakdown.

Phrase-Match Fixtures (168)

Near-verbatim phrases that should surface a specific passage. These test whether the exact words from a sacred text can be retrieved — the highest-priority match quality tier.

Concept-Match Fixtures (271)

Conceptual queries with religion filter — primary source should rank first. These test whether the authority-weighted ranking correctly promotes primary scriptures above secondary and general sources for tradition-specific searches.

Cross-Tradition Fixtures (52)

Universal concepts without religion filter — multi-tradition results expected. These test whether cross-tradition authority normalization is working: primary scripture from any tradition should rank equally against primary scripture from another.