The sci-rag methodology

This document is the kit's specification. It describes every design decision that matters, in plain language. You should be able to judge whether the approach fits your field, explain it in a paper, or re-implement it in another stack. The code follows this document, not the other way around.

In one sentence: hybrid retrieval over a single Postgres database, a knowledge graph built from a user-defined ontology, fail-closed license scoping, and an evaluation harness designed to be hard to game.

1 Why this shape

Scientific question-answering has three properties that break naive RAG:

1. The evidence is numeric and tabular. The answer to "What yield should I expect?" is a table row, not a vibe. Chunking that shreds tables, or retrieval that cannot find the number's context, produces confident nonsense.
2. Questions and documents use different words. A user asks "how much straw does the county produce"; the document says "141,000 harvested acres at a 1.1 straw-to-grain ratio". Pure keyword search misses it; pure embedding search blurs it.
3. Corpora mix redistribution rights. A lab's document pile mixes public reports, CC-BY papers, and paywalled PDFs it may hold but not redistribute. A RAG that quotes retrieved text is redistribution, so rights have to be a first-class, fail-closed property of retrieval.

Every choice below traces back to one of those three.

2 One database

Text, chunks, embeddings, full-text indexes, the knowledge graph, and licensing metadata all live in a single PostgreSQL database with the pgvector extension. There is no separate vector store and no graph database.

This is a deliberate trade. A dedicated graph database is faster at deep traversals. But this methodology never traverses deeper than two hops (section 6.3), and one database means one backup, one migration story, one access-control surface, and transactional consistency between a chunk and its graph entries. Measure before adding infrastructure: the seams are there if a corpus ever outgrows this (millions of chunks), but do not pay the operational cost on day one.

3 Ingestion

Every document flows through the same sequence:

parse -> chunk -> classify license -> deduplicate -> embed -> store (one transaction)

• Parse. PDFs go through a structure-preserving parser (Docling when installed, pypdf as the fallback). Markdown parses directly, and plain text passes through. All routes produce the same block model: headings, tables, and prose.
• License. Each document carries a redistribution class declared in the corpus manifest: public, open_commercial, open_noncommercial, restricted, or unknown. Nobody said otherwise means unknown, and the kit treats unknown as unsafe (section 7).
• Deduplicate. Content identity is a SHA-256 over the normalized chunked text, and a unique constraint enforces it. Re-ingesting a file, or the same content under a new filename, is a no-op.
• Store transactionally. A document and all its chunks commit together or not at all; a crash cannot leave half a document behind.

4 Structure-aware chunking

A generic recursive text splitter throws away the two most valuable signals in technical writing: where a statement sits in the section hierarchy, and whether a table survives intact. The chunker keeps both:

1. Normalize the raw text: form feeds, stray page-number lines, end-of-line hyphenation ("ligno-\ncellulosic" becomes "lignocellulosic"), null bytes.
2. Segment into paragraphs; drop fragments under 20 characters (almost always extraction noise).
3. Detect headings three ways: numbered ("2.1 Feedstock handling"), ALL CAPS, and short Title Case lines. Maintain a section path such as "2 Methods > 2.1 Feedstock handling".
4. Detect table-like paragraphs (pipe tables, tab columns, aligned space grids) and emit each table as its own intact chunk, flagged is_table.
5. Merge ordinary paragraphs up to a target of 800 tokens, splitting oversized paragraphs on sentence boundaries, and never merging across a section boundary.
6. Carry roughly 150 tokens of trailing overlap into the next prose chunk (tables hand over at most their last two rows), and never carry overlap across a heading.
7. Prepend the document title and section path to every chunk, so each chunk embeds and reads sensibly on its own.

The 800/150 defaults suit dense technical PDFs; both are parameters.

5 Embeddings

• One embedding model per corpus, with the model identity stamped on every chunk (embedding_version). A model upgrade is then a findable migration ("re-embed rows whose version is stale"), never a silent mixture of incompatible vectors.
• The default dimension is 1536, requested from the model by Matryoshka truncation. This keeps vectors inside pgvector's 2000-dimension HNSW index limit, so nearest-neighbor search uses a real index instead of a full scan. The embedder re-normalizes truncated embeddings to unit length before storage, because truncation breaks the unit norm that cosine ranking assumes.
• The embedder asserts the returned dimension on every call. A model/configuration mismatch fails loudly at the source instead of surfacing later as an opaque database error.
• Queries and documents each embed with their own task hint, which is what asymmetric retrieval means: the same words are encoded differently depending on which side of the search they sit on. Interactive query embeddings cache briefly in process memory under hashed keys, and the raw query text is never a cache key.

6 The five retrieval layers

Five candidate generators run in parallel, each with its own database session and its own timeout, and their ranked lists fuse once.

6.1 Dense vector search

Cosine similarity over chunk embeddings, served by HNSW. The workhorse; carries the highest fusion weight.

6.2 Keyword full-text search

Postgres full-text search over a generated tsvector column (GIN indexed), using websearch_to_tsquery so raw user input is always safe. This layer catches exact terms, identifiers, and chemical names that embeddings blur.

6.3 Knowledge-graph traversal

At ingestion time, an LLM extracts entities and typed relationships from each chunk. A domain ontology you declare constrains it: entity types and relation types with one-line descriptions, in a YAML file. The extractor drops unknown types and dangling endpoints rather than guessing them. Entities are canonical by name, accumulate evidence pointers (the chunks they were extracted from), and retain surface-form aliases actually present in the source. Relationships keep the quoted phrase that stated them and a calibrated confidence score: 1.0 for direct statements, 0.7 for strong implications, and 0.4 for inferences across sentences. Re-extraction merges aliases and preserves the highest confidence observed for a repeated typed edge from the same evidence surface. Edges with different document or chunk provenance remain separate so retrieval scope cannot erase otherwise eligible relationship evidence.

Extraction can still fragment one concept across several names. Run sci-rag graph resolve-entities --dry-run to inspect a conservative three-tier resolution pass: normalized name and alias overlap first, high-similarity same-type names second, and one batched LLM decision for the ambiguous band. Nothing lands until --apply. A merge unions evidence and aliases, repoints relationships, and leaves the old row as a canonical_entity_id tombstone. Every applied merge has a durable row in entity_resolution_audit; --no-llm provides a deterministic-only pass. The doctor reports cheap probable duplicates, and graph GC preserves these tombstones. Because community summaries materialize entity membership and relationships, an applied merge clears them; rebuild with sci-rag graph communities after reviewing the resolution receipts.

At query time, a fast LLM call extracts entity names from the question. The walk follows matching graph entities up to two hops in either direction, and the chunks those entities point to re-enter the candidate pool. By default candidates retain the historical hop-distance ordering. The domain profile may set a minimum relationship confidence, and may rank by the strongest minimum-edge confidence along each path before using hop distance as a tie-breaker. Both controls are off by default and must earn their place through the confidence_weighted versus full_deep ablation. The graph stage can also expand the represented documents by one resolved citation hop in either direction. This is off by default. It uses only document_citations rows whose target resolves to a corpus document, and it applies the request scope to the neighboring document before chunk ranking. The with_citations ablation measures it. Unresolved DOI pointers remain visible provenance but never enter retrieval. This is what makes multi-hop questions work: the connecting entity brings its evidence with it even when the question's words never appear in that text.

Alias strings currently do not carry per-surface document provenance, so only an unrestricted graph walk may expand them. A restricted walk may seed from an exact active or tombstone name only when that literal surface occurs in one of the entity's eligible evidence chunks. Resolution tombstones retain their original evidence pointers for this check. The walk restricts retrieved chunks before ranking, and every traversed relationship must itself carry eligible document or chunk provenance. Restricted evidence therefore cannot seed, extend, or contribute a candidate to the walk.

6.4 Community summaries

Clusters of tightly connected entities usually map onto real themes in a corpus. Deterministic label propagation finds the clusters, an LLM writes a short summary of each, and the embedder embeds those summaries. At query time the layer runs vector search over the summaries and can return them as results. That is how the layer answers "big picture" questions when no single chunk covers them.

One hard rule: a stored summary aggregates evidence from many documents before any caller's scope is known, so this layer disables itself whenever license, source, or exclusion filters are active. A scoped caller must never receive a summary partially built from documents outside their scope.

6.5 HyDE (hypothetical document embeddings)

A fast model writes the short passage a real document would contain if it answered the question. That passage embeds as a document, and vector search runs near it. This bridges question-phrasing and answer-phrasing (property 2 in section 1). The domain profile can steer the passage style per query class (an availability question reads like a resource assessment; a properties question reads like a characterization table). The generated passage is a search probe only: never shown, never cited.

6.6 Fusion

Weighted reciprocal rank fusion:

score(c) = sum over layers of  weight_layer / (k + rank_of_c_in_layer)

with rank starting at 1 and k = 60. Native scores (cosine distance, ts_rank, hop counts) are incomparable across layers; ranks are not. A candidate several layers agree on beats a candidate one layer loved.

Layer	Default weight
vector	1.5
keyword	1.0
graph	0.8
community	0.6
HyDE	1.2

These defaults have held up in production use. To tune them for your own corpus, use the evaluation harness's ablation mode: it reports what each layer actually contributes to hit rate before you touch a weight.

6.7 Profiles and degradation

Two profiles set defaults: interactive (vector + keyword only, short per-stage timeouts, query-embedding cache on) for humans waiting on a spinner, and deep (all five layers, generous timeouts) for agents, batch jobs, and evaluation.

A slow or failing layer degrades rather than breaks. It contributes no candidates, a per-stage trace records its status (timeout, error, empty, skipped, disabled), and the caller sees exactly what ran. Traces are content-free by design (stage, status, duration, candidate count; never query text or chunk text), so they are always safe to log.

7 Scope precedes ranking

A retrieval scope is the caller's rights: allowed license classes, allowed sources, excluded documents. Three rules make it trustworthy:

1. Every layer applies scope inside its own SQL, before ordering and limiting. Filtering after ranking is wrong twice over: an out-of-scope row can crowd an eligible row out of a bounded candidate pool, and excluded content silently shapes what survives.
2. An explicitly empty license allowlist means "return nothing". Fail closed, before any embedding call or database query.
3. unknown is never external-safe. Only public and open_commercial belong on surfaces you do not fully control.

Excluded document IDs use the same mechanism, which is also what makes evaluation holdouts real (section 9). An excluded document is unreachable through every layer, including graph traversal, and a regression test proves it.

Retraction is a fourth scope dimension, and it is the one with a default. Crossref enrichment records whether a document has been retracted, and exclude_retracted drops those documents inside every layer's SQL like any other scope condition. Answering turns it on by default; raw retrieval does not, because inspecting what a retracted paper claimed is a legitimate thing to want and the caller has asked for candidates rather than for an answer. A retraction discovered after ingestion changes the next answer without re-ingesting anything, and doctor reports the count so a corpus cannot quietly acquire retracted sources.

8 Grounded answers

The answer prompt receives numbered sources: the retrieved chunks, each with title, section path, and citation. It also carries three standing orders. Cite every claim inline by number. Prefer the sources' numbers and units over summary. And when the sources do not contain the answer, say so instead of improvising.

The response carries a structured citation list mapping each number to its document, and the evaluation judge checks that honesty rule after the fact.

Between retrieval and prompt assembly, sources may be compressed: question-aware summarization of each chunk, dropping any whose relevance falls below a floor. This shortens the prompt without changing which documents are cited, so the citation list is unaffected. It is off in the model default and on for the shipped demo domain, and the difference is the point. Compression only earns a default where a paired judged-answer evaluation has shown every quality dimension holding while measured prompt tokens fell. Carrying it to another corpus means re-running that gate there, not inheriting the demo's result.

9 Evaluation design

• Ground truth is expert-authored. A seed question holds the question, what a correct answer must say, which documents contain it, and a few distinctive evidence phrases. Ten questions an expert vouches for beat a hundred vague ones.
• Retrieval metrics are mechanical and transparent. A retrieved item is relevant if it comes from a reference document or contains an evidence phrase (whitespace and case normalized). The harness computes hit@5, hit@10, and MRR per layer-ablation config, so every layer has to earn its fusion weight on your corpus.
• The judge is blind. Grading happens in two independent passes. The grounding pass sees the question, the answer, and exactly the sources the assistant retrieved. It scores groundedness, citation accuracy, and completeness against those sources only, and it never sees the reference answer. A judge that does see it will reward reference-matching answers the sources do not support. The correctness pass compares the answer to the expert reference in a separate call, without the sources. Both run at temperature 0. Scores clamp to a 0-to-2 rubric, and a malformed judge response is a failure, never a coerced score.
• Numbers carry their context. Every report carries a corpus fingerprint (document, chunk, and graph counts, embedding versions, latest ingestion time) and the git commit. An eval number without its corpus fingerprint is just a rumor.
• Honesty probes. Questions tagged unanswerable sit outside the retrieval metrics. They are there to check that the system admits gaps instead of inventing answers.

10 What this methodology does not do (yet)

Kept out deliberately, with the seams left visible:

• ~~Cross-encoder reranking of the fused pool.~~ Filled in v0.2: src/sci_rag/retrieve/rerank.py ships a Reranker protocol with an LLM adapter (default, zero new dependencies) and a local cross-encoder adapter behind the rerank extra. It stays off until the with_rerank versus no_rerank ablation justifies it on your corpus. GCP users can implement the same protocol against the Vertex AI Ranking API, meaning the discoveryengine.googleapis.com rank endpoint. Score the pool, return the reordered items, and point retrieval.reranker in domain.yaml at your adapter, through a small subclass of the retriever or a fork of build_reranker.
• Hierarchical communities (communities of communities) for very large graphs.
• Automatic license classification (DOI lookups against registries). The manifest declares rights in v1; a classifier can enrich it later, but must only ever downgrade toward restricted, never upgrade.
• A learned fusion model. Weighted RRF is transparent and debuggable; do not replace it until an ablation table says so.