Retrieval-augmented generation, built around your scientific domain.¶
Sci RAG Kit provides a blueprint for scientific RAG development, from document ingestion, to knowledge graph generation, retrieval and evaluation. Fully extensible and ready to serve over API and MCP.
Start a new project¶
Get started with two commands in the terminal. The setup wizard will ask a series of questions and initialize configuration files around your answers:
Quick mode uses shipped defaults for models, parsing, reranking, infrastructure,
and licensing, and configures any LLM credentials you wish to provide. Choose
Offline for a credential-free first pass or Advanced for full customization.
(A full trace of this terminal session can be found below)
Prefer to read the code before creating a project? Other entrypoints covers a clone, the GitHub template, sci-rag init, and the dev container. Every route ends at the same tree.
The Quickstart goes from installation to a served knowledge base in about ten minutes, and How it works explains what is happening at each step along the way.
What's in the kit?¶
Document ingestion: PDF, HTML, Markdown, and text files become passages that keep their section headings and their tables intact, so a number retrieved from "Table 3" still knows which experiment it belongs to. Follow a document into storage.
Five kinds of search: by meaning, by exact words, through a graph of the field's concepts, through summaries of related concepts, and through a model-written hypothetical answer. The model-dependent layers require a credential. Their result lists merge into a single ranking that says which layer found what. See how a question is answered.
One database: passages, vectors, the full-text index, and the concept graph all live in Postgres. That leaves one thing to run, one thing to back up, and one place where a chunk and its graph entries commit together. Read why.
Rights built in: every document carries a license class, and a request that restricts rights never sees passages outside it, because the filter runs inside each search before anything is ranked. Read the rights rules.
Cited answers: with LLM provider credentials, every claim points at a numbered passage. When the documents do not contain an answer, the kit says so rather than filling the gap from a model's memory. Use REST or MCP.
Measurement: a file of questions with known answers lets the kit score retrieval, grade generated answers, and report what each search layer contributes on the corpus at hand, so a change is judged by numbers rather than by impression. Evaluate your pipeline.
Configure around your domain¶
Point the kit at a new domain by editing plain-text configuration. The domain's concepts, prompt wording, and scoring questions all load at run time without Python changes.
pipx install sci-rag-kit installs the kit. sci-rag new then fills in the
configuration files from the answers given during setup.
domain/ holds everything specific to the domain: the concepts the graph
looks for, the prompt wording, and the test questions. data/ holds the
documents themselves and a one-line-per-document manifest that records who
wrote each one and whether its text may be redistributed. Everything else
is the pipeline, and most projects never open it.
your-sci-rag/
├── domain/ the field: concepts, prompts, test questions
├── data/ the documents and their manifest
├── src/sci_rag/ the pipeline, from ingestion to serving
├── migrations/ database tables
├── tests/ runs offline
├── infra/terraform/ optional Google Cloud deployment
└── docs/ this siteWhere to start¶
Example CLI Setup Wizard Session¶
The Quick session above, in full. scripts/render_cast.py builds it by driving the real setup wizard, so it cannot drift from what sci-rag new asks. make cast regenerates it, and make docs fails when it is stale.
$ pipx install sci-rag-kit
$ sci-rag new
Select Setup
1 - Quick - Six questions, sensible defaults for the rest
2 - Advanced - Every option for full customization
Choose from [1/2] (1): 1
What is your project called?
project_name (My Scientific KB): Membrane Materials KB
One line about your field
description (A short description of your domain.): Membrane chemistry and performance for water treatment
Contact email. Sent with each request to OpenAlex, Crossref, and Unpaywall, which serve identified callers faster. Blank is allowed.
contact_email (): you@lbl.gov
Select environment_manager: Environment manager
1 - uv: Fast Python environments and locking with uv
2 - pixi: Conda packages and Python dependencies in one project
3 - conda: A conventional conda environment plus pip dependencies
4 - venv+pip: Standard-library virtual environment and pip
Choose from [1/2/3/4] (1): 1
Select credentials: How will you reach a model?
1 - google_ai_studio: Shortest local setup; no manual Cloud setup
2 - vertex_ai: Billed through a Google Cloud project you already have
3 - offline: No model calls, graph extraction, or generated answers
Choose from [1/2/3] (1): 1
Google AI Studio API key. Get one at https://aistudio.google.com/apikey. Blank to add it later.
google_api_key ():
Select corpus_source: Where will the first documents come from?
1 - local_files: Add PDFs, HTML, Markdown, or text files from disk
2 - openalex_topic: Discover a legal corpus from an OpenAlex topic
3 - doi_list: Resolve a list of known DOI records
4 - demo_only: Keep the bundled synthetic corpus for evaluation
Choose from [1/2/3/4] (1): 1
Checking the credential with one small model request...
gemini-3.6-flash check simulated for this recording; no model request sent.
Drafting an ontology for "Membrane chemistry and performance for water treatment"...
Entity types Membrane, Material, Contaminant, Process, Property, Application, Organization, Standard
Relation types MADE_OF, REMOVES, HAS_PROPERTY, USED_IN, REQUIRES, COMPARED_WITH
Query classes performance, fabrication, fouling, application
Accept this ontology? [y/n/redraft] (y):
Fetching sci-rag-kit for membrane-materials-kb...
Writing membrane-materials-kb/
removed docs/planning/, scripts/cloud_postgres.py, infra/terraform/dev-database/
domain/domain.yaml 8 entity types, 6 relation types, 4 query classes
data/demo/eval_seed_questions.jsonl ground truth for the demo corpus
domain/eval_seed_questions.jsonl guided blank
.env google_ai_studio, gemini-3.6-flash, gemini-embedding-001
pyproject.toml name, description, extras: none
Makefile commands prefixed with `uv run`
docs/ kit onboarding, player, and cast removed
Dockerfile uv base image
.devcontainer/ ghcr.io/va-h/devcontainers-features/uv:1
rendered 6 files for uv
data/corpus.jsonl commented field shape, ready for your documents
LICENSE BSD-3-Clause
README.md rewritten opening
git initialized, 1 commit
Done. Membrane Materials KB is set up. Next:
cd membrane-materials-kb
make setup # install, start Postgres, create the tables
uv run sci-rag doctor
# copy your PDFs, HTML, Markdown, or text files into data/raw/
uv run sci-rag build data/raw
uv run sci-rag answer "a question in your field"
Then let a model draft the rest of your domain files:
uv run sci-rag draft manifest --folder data/raw
uv run sci-rag draft ontology --from-corpus
uv run sci-rag draft questions --count 10
Each command proposes a file for review. Add --print-prompt to copy its prompt into an assistant you already use.
The walkthrough: docs/bring-your-own-domain.md
Show the Advanced setup
$ pipx install sci-rag-kit
$ sci-rag new
Select Setup
1 - Quick - Six questions, sensible defaults for the rest
2 - Advanced - Every option for full customization
Choose from [1/2] (1): 2
What is your project called?
project_name (My Scientific KB): Membrane Materials KB
Repository directory name
repo_name (membrane-materials-kb):
One line about your field
description (A short description of your domain.): Membrane chemistry and performance for water treatment
Who should the project credit?
author_name (Your name, lab, or organization): Berkeley Lab
Contact email. Sent with each request to OpenAlex, Crossref, and Unpaywall, which serve identified callers faster. Blank is allowed.
contact_email (): you@lbl.gov
Python version
python_version (3.12):
Select environment_manager: Environment manager
1 - uv: Fast Python environments and locking with uv
2 - pixi: Conda packages and Python dependencies in one project
3 - conda: A conventional conda environment plus pip dependencies
4 - venv+pip: Standard-library virtual environment and pip
Choose from [1/2/3/4] (1): 2
Select dependency_file: Where should pixi dependencies live?
1 - pyproject.toml: Keep project and pixi dependencies together
2 - pixi.toml: Keep pixi configuration in its own file
Choose from [1/2] (1):
Select credentials: How will you reach a model?
1 - google_ai_studio: Shortest local setup; no manual Cloud setup
2 - vertex_ai: Billed through a Google Cloud project you already have
3 - offline: No model calls, graph extraction, or generated answers
Choose from [1/2/3] (1): 1
Google AI Studio API key. Get one at https://aistudio.google.com/apikey. Blank to add it later.
google_api_key ():
Select embedding_provider: Embedding provider
1 - google: Semantic embeddings from the configured Google model
2 - local-hash: Deterministic offline vectors for development and tests
Choose from [1/2] (1):
Generation model
llm_model (gemini-3.6-flash):
Embedding model
embedding_model (gemini-embedding-001):
Embedding dimensions
embedding_dim (1536):
Select ontology: Starting ontology
1 - draft_with_llm: Draft field-specific types from your description
2 - keep_demo_example: Keep the worked agricultural-residue ontology for now
3 - blank: Start with an intentionally empty ontology
Choose from [1/2/3] (1): 1
Select corpus_source: Where will the first documents come from?
1 - local_files: Add PDFs, HTML, Markdown, or text files from disk
2 - openalex_topic: Discover a legal corpus from an OpenAlex topic
3 - doi_list: Resolve a list of known DOI records
4 - demo_only: Keep the bundled synthetic corpus for evaluation
Choose from [1/2/3/4] (1): 2
OpenAlex topic
openalex_topic (your topic): polyamide membrane fouling
Maximum OpenAlex results
max_results (100): 250
Select pdf_parser: PDF parser
1 - pypdf: Lightweight text extraction with no machine-learning stack
2 - docling: Structure-aware parsing with stronger table extraction
Choose from [1/2] (1): 2
Select reranker: Result reranker
1 - none: Return the fused ranking as-is
2 - llm: Ask the configured model to reorder retrieved passages
3 - local_cross_encoder: Run a local cross-encoder model
Choose from [1/2/3] (1):
Select include_terraform: Keep production Terraform?
1 - Yes: Keep the optional Cloud Run and Cloud SQL deployment module
2 - No: Remove production infrastructure files and their CI job
Choose from [1/2] (1): 2
Select include_cloud_database: Include the Cloud SQL development helper? Include the opt-in Cloud SQL development helper and Terraform module.
1 - Yes: Keep the opt-in shared development database helper
2 - No: Use Docker, conda-forge, or another PostgreSQL server
Choose from [1/2] (2):
Select include_demo_corpus: Keep the demo corpus?
1 - Yes: Keep five synthetic documents for a known-good first run
2 - No: Remove the demo and examples from the generated project
Choose from [1/2] (1): 2
Select open_source_license: Open-source license
1 - BSD-3-Clause: Permissive license with non-endorsement protection
2 - MIT: Short permissive license
3 - Apache-2.0: Permissive license with an explicit patent grant
4 - No license file: Do not grant redistribution rights yet
Choose from [1/2/3/4] (1):
Select initialize_git: Initialize a Git repository?
1 - Yes: Create a repository and make the generated baseline commit
2 - No: Leave version-control setup to you
Choose from [1/2] (1):
Select draft_domain_files: Draft the remaining domain files next?
1 - Yes: Put the corpus-grounded drafting commands in next steps
2 - No: Point next steps at the hand-written route
Choose from [1/2] (1):
Checking the credential with one small model request...
gemini-3.6-flash check simulated for this recording; no model request sent.
Drafting an ontology for "Membrane chemistry and performance for water treatment"...
Entity types Membrane, Material, Contaminant, Process, Property, Application, Organization, Standard
Relation types MADE_OF, REMOVES, HAS_PROPERTY, USED_IN, REQUIRES, COMPARED_WITH
Query classes performance, fabrication, fouling, application
Accept this ontology? [y/n/redraft] (y):
Fetching sci-rag-kit for membrane-materials-kb...
Writing membrane-materials-kb/
removed docs/planning/, infra/terraform/, scripts/cloud_postgres.py, infra/terraform/dev-database/, data/demo/, examples/, scripts/graph_replay.py, tests/unit/test_graph_replay_contract.py, tests/integration/test_graph_replay.py, tests/unit/test_graph_replay_makefile.py, tests/unit/test_graph_replay_scaffold.py, docs/adr/0011-committed-benchmark-graph-replay.md
domain/domain.yaml 8 entity types, 6 relation types, 4 query classes
domain/eval_seed_questions.jsonl guided blank
.env google_ai_studio, gemini-3.6-flash, gemini-embedding-001
pyproject.toml name, description, extras: docling
Makefile commands prefixed with `pixi run`, database defaults to conda-forge, no Docker needed
docs/ kit onboarding, player, and cast removed
pyproject.toml workspace, environments, tasks
Dockerfile pixi base image
.devcontainer/ ghcr.io/prefix-dev/devcontainer-features/pixi:0
rendered 9 files for pixi
pixi.lock created on first `pixi install`
data/campaigns/ openalex topic "polyamide membrane fouling"
LICENSE BSD-3-Clause
README.md rewritten opening
git initialized, 1 commit
Done. Membrane Materials KB is set up. Next:
cd membrane-materials-kb
make setup # install, start Postgres, create the tables
pixi run sci-rag doctor
make corpus # discover papers and write data/corpus.jsonl
pixi run sci-rag build --manifest data/corpus.jsonl
pixi run sci-rag answer "a question in your field"
Then let a model draft the rest of your domain files:
pixi run sci-rag draft manifest --folder data/raw
pixi run sci-rag draft ontology --from-corpus
pixi run sci-rag draft questions --count 10
Each command proposes a file for review. Add --print-prompt to copy its prompt into an assistant you already use.
The walkthrough: docs/bring-your-own-domain.md