Corpus campaigns¶

A campaign turns a research topic or a seed DOI file into a reproducible list of scientific works. Discovery is deliberately separate from ingestion: you can inspect the candidates and resume network work before anything lands in the corpus.

Discover from a topic¶

Identify every API request with a monitored contact address:

uv run sci-rag campaign discover \
  --topic "rice straw valorization" \
  --name rice-straw \
  --mailto you@example.org \
  --max-results 100

Topic discovery searches OpenAlex and follows cursor pagination. Casual use works without a key. For a larger API budget, set OPENALEX_API_KEY in the environment. The kit never prints that key and never writes it to campaign state.

Discover from DOI seeds¶

Put one bare DOI, DOI URL, or doi: value on each line. The reader skips blank lines and lines beginning with #.

10.7717/peerj.4375
https://doi.org/10.1038/s41586-020-2649-2

Then run:

uv run sci-rag campaign discover \
  --doi-file seed-dois.txt \
  --name seed-review \
  --mailto you@example.org

DOI-file discovery normalizes and deduplicates the values, then retrieves bibliographic metadata from Crossref. The report counts invalid lines and malformed upstream records rather than letting them disappear.

Resume behavior¶

The default state path is data/campaigns/<name>/state.jsonl. The run appends each completed DOI step and flushes it to disk. Repeating the command skips DOI records already present in state. If a process dies during the final write, the next run ignores only the truncated tail and safely continues appending.

OpenAlex and Crossref calls are rate limited and identify the contact address in both the query and User-Agent. The client retries HTTP 429 and server errors with bounded backoff. Exhausted retries fail visibly; they never become an empty success report.

Discovery metadata is not proof that a document may be redistributed. Resolve rights without downloading first:

uv run sci-rag campaign build \
  --topic "rice straw valorization" \
  --name rice-straw \
  --mailto you@example.org \
  --max-results 20 \
  --dry-run

The dry run queries Unpaywall for each DOI, prints direct-PDF counts and the license-class distribution, and writes only resumable state. It does not create pdfs/ or corpus.jsonl.

Fail-closed rights mapping¶

Availability and redistribution rights are different signals. Unpaywall marking a work green or gold is not enough on its own. A work earns an open license class only when its selected location also carries an explicit, recognized license:

Explicit location license	Corpus class
CC0 or public-domain mark	`public`
CC BY family	`open_commercial`
CC BY-NC family	`open_noncommercial`
Missing, `implied-oa`, publisher-specific, or unrecognized	`unknown`

The mapping never infers a license from oa_status, a reachable URL, or a PDF response. unknown is the intentional safe result when rights are unclear.

Download and ingest¶

After reviewing the dry-run distribution, repeat the command without --dry-run. The builder fetches only Unpaywall's direct url_for_pdf for a record marked open access. It never visits a landing page to scrape through a paywall.

Each response must declare application/pdf, stay within --max-pdf-mb, and begin with a PDF signature. The builder writes files through a temporary path and renames them only after validation. After an interruption it reuses files it has already verified rather than downloading them again.

Successful downloads produce data/campaigns/<name>/corpus.jsonl, in the same format sci-rag ingest reads:

uv run sci-rag ingest --manifest data/campaigns/rice-straw/corpus.jsonl

Every manifest row retains the normalized DOI, bibliographic metadata, fail-closed license_class, and the exact Unpaywall license signal in license_source.

Screen a discovered campaign¶

Write the review protocol as plain text. State the inclusion and exclusion criteria precisely enough that another reviewer could apply them without outside context:

Include field studies of rice-straw conversion with a measured material yield.
Exclude reviews, simulations without experimental validation, and studies of
other feedstocks.

Screen the abstracts already retained in campaign state:

uv run sci-rag campaign screen \
  --name rice-straw \
  --criteria-file screening-criteria.txt \
  --confidence-threshold 0.8

The model receives abstracts in bounded batches and must return one strict include or exclude decision, confidence, and reason per work. The command does not trust that output blindly:

confidence below the threshold becomes review;
a missing abstract becomes review without a model call;
malformed JSON, missing rows, duplicate indexes, and provider failures make the affected batch review;
no failure path silently excludes a work.

Decisions append to state.jsonl. screening-report.json records the current protocol, its SHA-256 digest, the confidence floor, every per-work reason, failure counts, and the current PRISMA-aligned totals. PRISMA is the Preferred Reporting Items for Systematic Reviews and Meta-Analyses. It is the reporting standard systematic reviews are held to, so aligning with it means the counts are the ones a reviewer expects. Repeating the same protocol resumes without calling the model again. Changing the criteria or confidence floor starts a new set of decisions while preserving the old append-only history.

The screening report begins at the deduplicated campaign-state boundary. identified, screened, and the sum of included, excluded, and awaiting_review therefore reconcile against the unique discovered works. campaign discover already removed and reported the upstream duplicates, so duplicates_removed is zero at this boundary. Exclusions also include a reason breakdown rather than only an aggregate count.

Review uncertain rows¶

Walk the queue interactively:

uv run sci-rag campaign review --name rice-straw

For each row, choose include, exclude, or skip, then record a reason. Human decisions append after the model suggestion instead of overwriting it, and the report regenerates from the latest decision under that protocol. Skipping leaves the row in awaiting_review, so the totals continue to reconcile without pretending the campaign is complete.