Changelog

Notable changes to sci-rag-kit. The format follows Keep a Changelog; versions follow Semantic Versioning once past 1.0.

[Unreleased]

Added

• sci-rag draft questions: an assisted first pass at domain/eval_seed_questions.jsonl, grounded in the documents you already have. It prefers the ingested corpus and falls back to reading data/raw/ through the same parsers, so it works before make setup. Every quoted evidence phrase is verified in Python against a passage belonging to a document the question names; rows that fail are dropped and reported by id and reason, with one repair round. A run proposes eval_seed_questions.jsonl.proposed; --apply appends to the seed file and never displaces a question a human wrote.
• Two lanes on every drafter. --print-prompt writes the fully rendered, corpus-grounded prompt to stdout for pasting into any assistant, and --from-file reads the reply back through identical validation, so the drafters work with no API key and no provider account.
• domain/prompts/seed_questions.md: the prompt behind that draft, with $DOMAIN_NAME, $ENTITY_TYPES, $QUERY_CLASSES, $PASSAGES, $COUNT, and $REJECTED slots.
• The drafted tag on seed questions, and the honesty plumbing behind it. sci-rag eval retrieval and sci-rag eval answers now warn in the generated Markdown when any question behind the numbers is still model-drafted, and both report payloads carry "ground_truth": {"drafted": N, "reviewed": M}. Removing the tag is the expert sign-off; nothing in the kit removes it for you.
• docs/llm-assisted-setup.md: the three lanes, the copy-paste path, the review discipline, and the honesty rules for drafted ground truth.
• sci-rag draft manifest: reads title, authors, year, DOI, journal, and a source bucket off each document's opening pages, through the same parsers ingestion uses, and proposes data/corpus.jsonl.proposed. Source buckets are chosen across the whole batch rather than one per file. license_class is never guessed: every drafted row is written unknown, the command reports how many documents need a rights decision, and a license sentence is kept in license_source as evidence only, and only when it appears verbatim in the document.
• sci-rag draft ontology: --from-corpus redrafts the ontology from real passages, --refine asks only for additions and removals with a reason for each, and --cold is the wizard's description-only draft on its own. The tuned retrieval: and compression: blocks are carried over untouched, and a refinement that would leave no entity type is rejected. Writes domain/domain.yaml.proposed with a summary diff; --apply writes in place.
• domain/prompts/manifest_metadata.md and domain/prompts/ontology_from_corpus.md.
• sci-rag draft prompts: rewords entity_extraction.md or answer.md for your field while keeping the job identical. The rewrite is re-rendered against dummy values and rejected if a required $SLOT went missing, if one was invented, or if the template will not render. judge_grounding.md, judge_correctness.md, snippet_compression.md, and ontology_draft.md are refused by name, each with a reason. New prompt domain/prompts/prompt_localization.md.
• Domain-coherence rows in sci-rag doctor: ontology size, unique names and SCREAMING_SNAKE relations; seed questions that cite nothing and seed sets with no unanswerable probe; how many questions are still tagged drafted; whether every reference title resolves to an ingested document; and manifest paths that no longer exist plus a count of rows with unknown rights.
• One new sci-rag init question, draft_domain_files, asked only when the project has credentials. It decides whether the next-steps block leads with the drafters or with the hand-written route.

Changed

• docs/bring-your-own-domain.md steps 2, 3, 4 and 6 are now Generate it / Write it yourself tab pairs. The generated route goes first; the full schema for every file stays underneath, unchanged.
• The next-steps blocks in scripts/init_domain.py and sci-rag init name sci-rag draft ... instead of telling you to go hand-write four files.
• AGENTS.md tells a coding agent pointed at a generated project to run sci-rag draft ... --print-prompt rather than authoring domain YAML from scratch.
• The rendered-geometry guard ignores zero-height wrappers. Two inactive tab panels in a row sit at the same point, which it read as a zero-pixel gap between blocks a reader can never see at the same time.

[0.3.0] - 2026-08-28

The "Campaigns" release: the kit gets the parts that make it better for science than a general-purpose RAG framework, plus an interactive project factory so starting one is two commands instead of a seven-step tutorial.

Two features ship switched off, on purpose. Contextual compression cut median prompt tokens by 70% but every judged dimension moved down, so its paired gate did not hold (#90). Entity resolution finds nothing to merge on the demo corpus, so its ablation cannot be measured here. Both are in docs/benchmarks.md with the numbers.

Added

• sci-rag init: an interactive setup wizard that specializes a checkout for your own domain. Asks about the project, credentials and models, ontology, corpus source, PDF parser, reranker, license, and stack, then writes domain/domain.yaml, .env, pyproject.toml, the Makefile, README.md, and the corpus scaffold. --defaults and --answers-file make generation reproducible; --dry-run previews it. The ontology can be drafted by the configured model and is validated against DomainConfig before anything is written, so a malformed draft is rejected rather than saved as unusable YAML.
• domain/prompts/ontology_draft.md: the prompt behind that draft.
• Multi-provider generation. SCI_RAG_LLM_MODEL and friends now accept a provider:model spec selecting google, anthropic (Claude on Vertex AI or the direct API), or openai-compatible (Vertex Model Garden partner models such as Grok, Llama, Mistral, and DeepSeek; also OpenAI and self-hosted vLLM/Ollama). New anthropic and openai extras; the SDKs are imported lazily so a Google-only install carries neither. See ADR 0006 for why the adapters are hand-written and why embeddings stay Google-only.
• SCI_RAG_JUDGE_MODEL, so the evaluation judge can run on a different provider than the generator. Answer eval reports now record which model answered and which graded, making cross-provider judging auditable.

• sci-rag doctor reports the resolved spec per role and checks credentials for each generation provider in use.

• Environment manager choice in the wizard: pixi, conda, and venv+pip alongside uv. One RunnerProfile renders all five uv-wired surfaces (task commands, CI, Dockerfile, dev container, docs), writes whatever manifest that manager needs ([tool.pixi] tables or pixi.toml, environment.yml, requirements.txt plus requirements-dev.txt), and is the only place a manager-specific string lives.

• generated-projects.yml: a CI matrix that generates a project per manager and runs that project's own lint, typecheck, and offline demo. uv runs on every pull request; all four run nightly, on release tags, and on any pull request that changes what a generated project looks like.

• sci-rag-new: the project factory as its own entry point. Runs from a parent directory with nothing cloned, fetches the template at the tag matching its own installed version, then applies the wizard's answers. --template-path generates from a local checkout with no network, and --ref overrides the tag. The LLM ontology draft is offered here and in sci-rag init, with accept, reject, and redraft.

• .github/workflows/release.yml: tag-driven publishing to TestPyPI and then PyPI over Trusted Publishing, gated on the ci workflow having passed for the tagged commit and on the tag matching the packaged version. The one-time maintainer setup is in docs/VERSIONING.md.

• The documentation homepage leads with pipx install sci-rag-kit and sci-rag-new, and shows the session in a player. The transcript below it is generated by driving the real wizard (make cast), so it cannot drift from the questions; make docs fails when it is stale. The asciinema player is vendored under docs/assets/vendor/, not loaded from a CDN, which keeps the docs build hermetic and the offline link check honest.

• docs/adr/0007-interactive-project-generator.md: why the generator is a post-fetch applier rather than a template renderer, and how that implements the escape hatch ADR 0004 left open instead of overturning it.

• A Docker-free database for pixi and conda projects. Those two managers read conda-forge, which ships the PostgreSQL server and pgvector as ordinary packages, so their generated manifests declare postgresql >=16,<19 and pgvector, and their make setup starts that server instead of a container. scripts/local_postgres.py drives initdb and pg_ctl against a project-local .pgdata/, producing a database the unmodified .env.example already points at. uv and venv+pip keep Docker: PyPI ships no server, and a manager that cannot take the path does not advertise it.

• .github/workflows/docker-free-postgres.yml: the integration and server suites against a conda-forge server on linux-64 and osx-arm64, failing if the suite skips rather than runs. With ci.yml on PostgreSQL 16 and this on 18, both ends of the supported range are tested.
• docs/adr/0008-supported-postgresql-versions.md: why the project supports a range instead of moving everyone to one major.

Changed

• Supported PostgreSQL versions are now stated and tested: 16 through 18. Nothing moves. Compose and the CI service stay on the pgvector/pgvector:pg16 image, and no existing database needs a migration. The schema uses no pgvector feature newer than 0.5 and no version-specific SQL, so the 16 in this repository was three container image tags rather than a requirement.

• Generated projects no longer carry the kit's own onboarding. The pipx instructions, the recorded session, the vendored player, and the renderer that keeps it current are all removed, along with their entries in mkdocs.yml and the Makefile, so a generated project's documentation still builds.

• docs/bring-your-own-domain.md opens with what the wizard asked and where each answer landed, then keeps the by-hand walkthrough for people who want to know what it chose for them.
• docs/tour.md, docs/deploy-gcp.md, README.md, and docs/benchmarks.md say which pieces the wizard can decline, instead of describing directories a generated project may not have.
• .env now reaches the process environment. pydantic-settings reads it into Settings but never exported it, so Typer's envvar= lookups and the OPENALEX_API_KEY read in sci-rag campaign discover could not see values placed there. A real environment variable still wins over the file.

• scripts/init_domain.py is now a thin shim over sci_rag.scaffold, so it and the wizard cannot disagree about what a seed-question reset looks like. Its command line, dry-run behavior, and output are unchanged.

• A bare model id still resolves to SCI_RAG_LLM_PROVIDER (default google), so existing configurations keep working unchanged.

• GoogleLLM moved from sci_rag.llm.client to sci_rag.llm.google, matching the layout of sci_rag.embed. sci_rag.llm re-exports it as before.
• Retry policy is shared across providers in retry_async(), and status-code detection no longer matches a code embedded in a longer number.

Fixed

• sci-rag doctor no longer reports FAIL for a project that runs offline on purpose. A project with the local-hash embedder, no credentials anywhere, and the shipped generation defaults now sees warnings that name the unavailable features, and exits 0. Reaching for a model still fails: a Google embedder without credentials, or a deliberately named generation model such as anthropic:claude-opus-5 with no key behind it, reports FAIL as before. The generated-project matrix gates on doctor again.
• sci-rag doctor prints its table when the configured embedder cannot be built. It looked up the embedder to report embedding-version drift, so a Google embedder without credentials aborted the whole command with a traceback, hiding the credentials row that explained the cause.
• sci-rag doctor --probe no longer warns on a healthy setup. Its generation probe capped output at 10 tokens, which reasoning models spend on thought before writing anything, so a working provider looked like it returned nothing.
• The Vertex Model Garden endpoint is now derived correctly for the global location, which is served by an unprefixed host. Grok is offered only globally, so the previous URL made it unreachable.
• The OpenAI-compatible adapter closes its response stream, returning the connection to the pool when a consumer stops early.
• sci-rag init no longer rewrites the commented examples in .env.example with the answers it collected. Only the first assignment of a key is the setting; substituting the later illustrative ones turned worked examples into confidently wrong advice and emitted the same key several times.

[0.3.0a1] - 2026-08-28

A packaging pre-release. No behavior of its own: it exists to establish Trusted Publishing to TestPyPI and PyPI and to reserve the sci-rag-kit name on both, using a version number that can be burned without cost if the publishing path turns out to be wrong.

The changes it carries are the ones listed under Unreleased above. The real 0.3.0, with the Wave 2 benchmark refresh, is #49.

[0.2.0] - 2026-08-27

The "Credibility" release: the gaps between what the methodology document promised and what the code did are closed, and the launch package (roadmap, governance, comparison, benchmarks) is in place.

Added

• Post-fusion reranker stage: Reranker protocol with an LLM adapter (default, zero new dependencies) and a local cross-encoder adapter behind the new rerank extra; per-request include_rerank overrides on the retriever and /v1/query; honest rerank stage traces with fused-order fallback on any failure. Off by default pending ablation evidence.
• Eval statistics: bootstrap 95% confidence intervals on every reported mean (question-level resampling, stdlib only, deterministic), small-sample warnings below 10 questions, paired bootstrap comparisons, and nDCG@10 alongside hit@k/MRR.
• sci-rag eval diff: per-question rank moves (improved, regressed, appeared, disappeared) and paired metric deltas with significance between any two eval runs; answers-mode dimension diffs.
• sci-rag eval calibrate: Cohen's kappa per judge dimension against human labels, agreement matrices, calibration sections appended to answers reports; ships a non-expert seed label set for the demo corpus (domain/eval_calibration_labels.jsonl).
• sci-rag embed reindex: find and re-embed rows stamped by retired embedder versions, batch-committed and idempotent; refuses cross-dimension reindexes; community summaries now carry embedding version stamps (migration 0002).
• sci-rag corpus delete + sci-rag graph gc: transactional document deletion that scrubs graph evidence arrays, drops evidenced relationships and affected communities, and a garbage-collection sweep for evidence-less entities and dangling pointers; regression test proves deleted content unreachable through every retrieval layer.
• sci-rag corpus snapshot: named, immutable corpus fingerprints (counts, per-document content hashes, embedding versions, git commit, corpus digest); eval runs record the snapshot name via --snapshot.
• Adaptive routing: --profile auto resolves per query through transparent heuristics (multi-hop, overview, lookup cues) with reasons, an optional ambiguity-only LLM fallback, a router trace, and --explain-routing; default profile unchanged pending the published ablation.
• docs/benchmarks.md: measured demo-corpus results (all ablation configs including rerank and routing, judged answers, judge calibration) with confidence intervals, corpus snapshot, commit and model ids, reproducible via make benchmark.
• docs/operations.md: backup/restore runbook (pg_dump, Cloud SQL, restore drill, Parquet export note).
• Launch package: docs/ROADMAP.md (waves 2-3, UW SSEC collaboration seams, BioCirV flagship, launch-gated decisions), docs/VERSIONING.md (0.x rules, 1.0 criteria), docs/GOVERNANCE.md, docs/choosing-sci-rag-kit.md (honest comparison), ADOPTERS.md, and a runnable examples/bring_your_own_domain.ipynb (verified offline).
• Doctor: embedding-version staleness and graph-hygiene checks.

Changed

• Retrieval eval reports state n and a 95% CI everywhere a mean appears; retrieval tables gained an nDCG@10 column.
• /v1/query accepts profile: "auto" and include_rerank.

[0.1.0a0] - 2026-08-26

First working release of the template.

Added

• Ingestion: Docling/pypdf/Markdown parsing, structure-aware chunking (section breadcrumbs, intact tables, 800/150 token defaults), content-hash deduplication, per-document license classes, JSONL corpus manifests.
• Storage: single-Postgres schema (Alembic) with pgvector HNSW and full-text GIN indexes; embedding version stamping.
• Embeddings: Google gemini-embedding-001 at 1536 dimensions (Matryoshka, re-normalized) via AI Studio key or Vertex AI; a deterministic offline hash embedder for tests and dry runs.
• Knowledge graph: ontology-constrained LLM extraction with evidence provenance, incremental stamping, deterministic label-propagation communities with LLM summaries.
• Retrieval: five layers (vector, keyword, graph traversal, community summaries, HyDE) with weighted RRF fusion, interactive/deep profiles, per-stage timeouts and traces, fail-closed license/source/exclusion scoping inside every layer.
• Answering: numbered inline citations, refusal when nothing is in scope, streaming events.
• Evaluation: seed questions, hit@k/MRR with seven ablation configs, a two-pass blind judge (grounding never sees the reference answer), fingerprint-stamped JSON and Markdown reports, a CI smoke eval.
• Serving: FastAPI /v1 (SSE answers, RFC 9457 errors, request ids, scoped API keys with rate limits, BYO LLM key) and an MCP server with seven tools and two resources, over streamable HTTP and stdio.
• Operations: sci-rag doctor, docker-compose Postgres, Dockerfile, validated Terraform for Cloud SQL + Cloud Run, GitHub Actions CI.
• Template ergonomics: domain/ specialization surface, scripts/init_domain.py, offline synthetic demo corpus with seed questions and committed real-model eval reports, full documentation suite with ADRs.