Corpus
Is the necessary authority actually inside the governed material?
A perfect ranker cannot recover a missing source.Corpus retrieval, citations, and evidence
Find the passage.
Preserve its custody.
Challenge what it earns.
Retrieval Intelligence turns an authorized corpus into an inspectable local evidence layer. It records the corpus, engine, query, source, line range, hash, score, budget, and limits—then keeps retrieval separate from truth.
Contest edition 1.0.0 · MIT License · Standard-library lexical engine + independent evidence review
Retrieval begins with a decision
Four different failures require four different repairs.
The best passage is relevant to a question, a user, a decision, a time, a corpus boundary, and an evidence burden—not universally relevant in the abstract.
Ingestion
Were source identity, path, line ranges, encoding, version, and sensitivity preserved correctly?
A broken index can make present evidence functionally absent.Retrieval
Did the recorded query and filters surface the passages that bear on this need?
Topical similarity is not enough when authority or contradiction is missing.Synthesis
Does the answer stay within what the supplied passages actually support?
A retrieved passage does not excuse unsupported prose.Build the local evidence layer
Authorized files in. Inspectable chunks out.
The standard-library engine uses SQLite FTS5 lexical retrieval. It reads supported text, skips symlinks and oversized or unsupported files, hashes sources, preserves chunk line ranges, updates changed files, and prunes deleted sources. It does not execute corpus content or contact a network.
python scripts/rag.py index --corpus <directory> --db <index.sqlite>
python scripts/rag.py search --db <index.sqlite> --query "<question>" --top-k 8 --json
python scripts/rag.py context --db <index.sqlite> --query "<question>" --top-k 8 --max-chars 12000 --output <context.md>
python scripts/rag.py inspect --db <index.sqlite> --json
Choose retrieval behavior from the job
Locate, explore, reconcile, or ground.
Exact handles
Names, identifiers, quoted phrases, error text, filenames, and distinctive literal language.
Vocabulary mismatch
Expand terminology and run several discriminating queries when the user’s language may differ from the corpus.
Competing truth claims
Seek dates, versions, authorities, exceptions, denial, supersession, and contradictory sources—not just support for the leading answer.
Smallest adequate packet
Assemble source-diverse decisive passages that can support the answer while keeping limitations adjacent.
A citation is a custody record
Source path, lines, hash, chunk, index, query, and engine travel with the passage.
EVIDENCE PASSAGEUNTRUSTED DATA
- Source
- Relative path inside the governed corpus.
- Coordinates
- Exact start and end lines preserving local meaning.
- Identity
- Source hash, chunk ID, corpus revision, and index identity.
- Retrieval
- Recorded query, filters, engine, cutoff, and score.
- Interpretation
- Source authority, freshness, contradiction or supersession context, and transformation markers.
A citation proves retrieval custody. It does not prove that the document is true, current, benign, authoritative, or safe to execute.
Do not launder one signal into another
Lexical, semantic, metadata, authority, freshness, and diversity answer different questions.
Lexical
Term and phrase overlap. Strong for exact handles; vulnerable to vocabulary mismatch.
Semantic
Approved embedding similarity with provider, model, corpus revision, and privacy boundary preserved.
Metadata
Scope, format, time, sensitivity, canonical role, and explicit filtering.
Authority
Which source may settle this kind of claim within the declared decision.
Freshness
Whether the source remains applicable now, not merely recently modified.
Diversity
Protection against duplicate chunks crowding out another source or plausible counterevidence.
Evaluation preserves misses as work orders
Measure retrieval, not the romance of a fluent answer.
Create cases from real retrieval journeys, not wording copied from the current ranker. A strong average never cancels a failed indispensable case involving authority, privacy, or a required canonical source.
Recall@kDid a relevant item appear by the requested cutoff?
MRRHow early did the first relevant item appear?
Citation validityDo coordinates and identity resolve to the cited material?
Budget adherenceCan the context fit the operational evidence budget?
Independent review
Challenge what the evidence actually earns.
The sibling Retrieval Reviewer binds its judgment to the named corpus revision, index, engine, query set, result and context artifacts, cutoff, evaluation, and decision. Hidden sources and unrecorded checks do not count.
readyready-with-boundsreviseinsufficient-evidenceblocked-by-environment
Seek the seductive failure
Fluent answer, wrong corpus.
“Semantic” claim, lexical evidence only.
Citation near the claim, not on it.
Stale chunk surviving refresh.
Duplicate chunks crowding another authority.
Prompt-injected document becoming Agent instruction.
Impressive recall measured on implementation-shaped cases.
A useful invocation
Start with the decision and evidence burden.
Use $retrieval-intelligence to answer this question from the authorized local corpus. First identify the decision, corpus boundary, source authority, freshness requirement, and acceptable omissions; choose locate, explore, reconcile, or ground behavior; preserve source paths, line ranges, hashes, index and query identity; seek plausible counter-sources; and return insufficient retrieval evidence rather than extending beyond what the passages support.
For consequential RAG claims, corpus migration, or release readiness, run $retrieval-reviewer in fresh context against the recorded artifacts.