Paper Analysis¶

paper_analysis is OSA's canonical analysis-only workflow for extracting technical claims from a paper and verifying their implementation against an OSA-supported repository. It never invokes scheduler workflows, README generation, forks, pull requests, repository mutation, Streamlit, leaderboard logic, or PDF presentation.

Pipeline¶

repository clone ──> source context ──> filtered, batched claim verification ──> canonical JSON + text
paper PDF ───────> paper_claims ───────^
sections JSON ─> paper_claims ───────^
claims JSON ─────────────────────────^
                       └─ optional formal repository-quality score

PDF input uses the typed paper_claims operation and defaults to OpenRouter Mistral OCR. Install osa_tool[paper-claims] for this path. Select Cloudflare AI or Marker with --pdf-parser or [paper_analysis.paper_claims].pdf_parser; cloud parsers require api = "openai" and an OpenRouter base URL in the paper-claims model profile. The host API uses Codex and ignores base_url, so cloud parser runs warn and stop.
OpenRouter parser calls also invoke the configured paper-claims model for each PDF chunk. Cloudflare AI parsing is free; normal model usage charges may still apply.
Marker is an optional local parser. Install osa_tool[paper-claims,paper-claims-marker] to use it.
Parsed section input uses generic PaperSection JSON via --sections-json, skipping PDF splitting, Marker, and Markdown parsing. Install osa_tool[paper-claims-lite] for parsed-section claim extraction without the PDF stack.
Existing claims.json, claims_legacy.json, and bare claim arrays can start directly at verification.
Only high- and medium-verifiability claims are verified by default. Low-confidence results are hidden and excluded from the implementation rate unless the corresponding CLI policy override is used.
Verification uses repository source snippets, Jupyter notebook content, CSV/TSV statistics, and strict batches of at most 50 claim indices.
The default run does not construct or call the formal quality scorer. Add --include-repository-quality or set include_repository_quality = true in configuration to produce the existing formal 0--100 report.

Each completed stage writes its own report.json. The root paper_analysis.json and paper_analysis.txt are written only after every requested stage succeeds. Earlier completed stage reports remain available after a later failure.

Configuration¶

config.toml contains [paper_analysis], [paper_analysis.paper_claims], and [paper_analysis.verification]. include_repository_quality defaults to false; command-line quality flags override it for one run.

Models inherit from [llm] unless their stage profile supplies an override:

[llm.for_paper_claims] for PDF claim extraction;
[llm.for_paper_verification] for implementation verification;
[llm.for_repository_quality] only when optional quality scoring is enabled.

Use --model-paper-claims, --model-paper-verification, or --model-repository-quality for per-run overrides. --paper-claims-prompts-dir can point at a directory of TOML prompt overrides using the same [prompts] keys as the built-in prompt files; only keys present in that directory are replaced.

Configured defaults are namespaced by clone name outside the repository. A relative paper_analysis default becomes <clone-parent>/paper_analysis/<clone-name>/. If this would collide with the clone, OSA uses the sibling <clone-parent>/<clone-name>.osa-artifacts/<clone-name>/. Explicit output paths inside the analyzed repository are rejected.

CLI¶

osa-tool --paper-analysis \
  --repository https://github.com/example/project \
  --paper ./paper.pdf \
  --pdf-parser mistral-ocr \
  --paper-output-dir ./analysis

The option reference below is generated from the same metadata used by osa-tool --help.

Paper-analysis options¶

Parameter	CLI options	Type	Required	Description	Choices
`claims_json`	`--claims-json`	path	—	Typed, legacy, or bare claim JSON to verify.	—
`force_parser_refresh`	`--force-parser-refresh` `--force-marker-refresh`	flag	—	Ignore cached PDF parser output for this run; claim extraction still runs normally.	—
`hide_low_confidence`	`--hide-low-confidence`	flag	—	Hide low-confidence verification results from the reported rate.	—
`include_low_confidence`	`--include-low-confidence`	flag	—	Keep low-confidence verification results in the report and rate.	—
`include_low_verifiability`	`--include-low-verifiability`	flag	—	Also verify low or missing-verifiability claims.	—
`include_repository_quality`	`--include-repository-quality`	flag	—	Calculate and export the formal 0-100 repository-quality score.	—
`only_high_medium_verifiability`	`--only-high-medium-verifiability`	flag	—	Verify only high- and medium-verifiability claims.	—
`paper`	`--paper`	path	—	PDF paper to extract through typed paper_claims.	—
`paper_claims_prompts_dir`	`--paper-claims-prompts-dir`	path	—	Directory of TOML prompt overrides for paper-claim extraction.	—
`paper_output_dir`	`--paper-output-dir`	path	—	Directory for paper-analysis artifacts. Configured defaults use a collision-safe clone-name namespace outside the repository; paths inside the analyzed repository are rejected.	—
`pdf_parser`	`--pdf-parser`	str	—	PDF parser for --paper. Defaults to the [paper_analysis.paper_claims] configuration.	`mistral-ocr`, `cloudflare-ai`, `marker`
`sections_json`	`--sections-json`	path	—	Parsed PaperSection JSON to extract and verify.	—
`skip_repository_quality`	`--skip-repository-quality`	flag	—	Skip formal repository-quality scoring for this run.	—

Parser behavior, configuration, and caching are described in the paper claims pipeline guide.

Use already parsed generic sections and optional paper-claim prompt overrides:

osa-tool --paper-analysis \
  --repository https://github.com/example/project \
  --sections-json ./sections.json \
  --paper-claims-prompts-dir ./prompt-overrides \
  --paper-output-dir ./analysis

sections.json may be either a bare list of PaperSection objects or an object shaped as {"sections": [...]}.

Include the formal repository score in the same artifact tree:

osa-tool --paper-analysis \
  --repository https://github.com/example/project \
  --claims-json ./paper_claims/claims.json \
  --include-repository-quality \
  --paper-output-dir ./analysis

Rich progress is sent to stderr and stdout contains only the final root JSON path. For remote repositories, --delete-dir removes only a clone created by the current invocation; local repositories and pre-existing remote clones are preserved.

Artifacts¶

analysis/
  paper_analysis.json
  paper_analysis.txt
  paper_claims/
    report.json
    claims.json
    document.md            # PDF input only
    sections.json          # PDF or sections JSON input
  claim_verification/
    report.json
  repository_quality/      # only with --include-repository-quality
    report.json
    report.txt

paper_analysis.json uses schema version 1.0. It always contains repository_quality; the value and quality artifact paths are null when scoring is disabled. Model provenance only lists stages that ran. Every stage report contains source metadata and configured/successful model provenance. Claim source metadata uses kind values pdf, sections_json, or claims_json.