---
name: autoresearch-scaffold
description: Use when the user wants to stand up a self-updating, trajectory-aware research ledger for a topic domain — an "autoresearch" workspace that sweeps sources, ingests them into an accreting evidence vault with a moves-both-ways status lifecycle, reasons across the corpus (interactions/connections), gates every change behind an audit, and optionally writes synthesis articles and a personal regimen. Triggers: "autoresearch workspace for X", "self-updating research ledger", "nightly research system for X", "build the Memory-Atlas / research-radar pattern for a new domain", "trajectory-aware knowledge vault".
---

# autoresearch-scaffold

Scaffold an **autoresearch workspace**: an ICM workspace whose `vault/` is a plain-markdown MOC ledger of a
domain's evidence, refreshed by an agent that sweeps → ingests → reconciles status → connects across the
corpus → (optionally) writes articles and a personal decision layer, all hard-gated by an audit script.
Proven twice: **Memory Atlas** (`~/source/memory-aggregator`, AI-memory research) and **Healthspan Atlas**
(`~/source/healthspan-atlas`, health interventions). This skill generalizes that shape to **any topic**.

**This is a specialization ("preset") of `icm-workspace-builder`, not a replacement.** Use that skill for
the generic ICM shell and non-developer interview; this skill overlays the autoresearch machinery that
`icm-workspace-builder` doesn't know about: a persistent accreting vault, an evidence-grade lifecycle, the
cross-corpus connect layer, a scripted audit gate, PDF archival, and the optional nightly `auto` runner.

## When this is the right pattern (filter check)

Autoresearch fits a domain where **all** of these hold — confirm before scaffolding:
- New primary sources arrive **over time** (papers, filings, releases, posts) and you want to track how
  understanding *moves*, not just snapshot it.
- Claims can be **graded and can flip** — a stronger source should be able to *downgrade* an earlier
  belief. (If facts don't contest each other, you want a wiki, not this.)
- The value is in the **accreting trajectory + cross-source connections**, not a one-shot report.

If it's a linear, run-once, no-memory workflow → plain `icm-workspace-builder` (or a skill) is the better
tool. Say so and stop.

## The two beyond-ICM extensions (name them honestly)

Vanilla ICM is human-checkpointed *linear* pipelines. Autoresearch pushes past that in exactly two ways —
call them out to the user so nobody pretends this is stock ICM:
1. **A persistent, accreting vault** — a mutable Layer-3/4 store stages read *and* write, whose statuses
   move both ways. Not a one-way stage→stage handoff.
2. **An (optional) authorized unattended runner** — the nightly `auto` mode replaces human checkpoints with
   the audit gate + caps + an emailed audit trail. Interactive modes keep the human checkpoint (textbook ICM).

## Workflow

Create a TodoWrite list with these phases. **Do not skip the gate or the two verify steps** — a scaffold
that looks right but whose audit gate doesn't actually run is worse than none.

```
Phase 0  Confirm fit + target      → filter check above; get the destination repo path
Phase 1  Interview the domain      → references/interview.md (the axes that specialize the shape)
─ GATE   Present the plan          → the shape you'll emit + the domain-specific choices; get a go
Phase 2  Emit the workspace        → stamp the tree from references/workspace-templates.md + copy scripts/
Phase 3  Verify the gate runs      → `python3 scripts/audit.py` must exit 0 on the empty vault
Phase 4  Seed one real source      → ingest ONE source by hand end-to-end; re-run the gate
Phase 5  Report + stop             → summarize; leave everything UNCOMMITTED for the human
```

### Phase 0 — Confirm fit + target
Run the filter check. Get the absolute destination path (e.g. `~/source/<domain>-atlas`). If the user
hasn't `git init`'d it, offer to. Run `date` (fresh datestamps).

### Phase 1 — Interview the domain
Read `references/interview.md` and walk it. It asks only the questions that *specialize* the shape — the
domain name + one-sentence purpose, the primary sources (the "arXiv equivalent") and their fetch ladder,
the **claim/evidence model** (what a graded claim looks like in this field and what its tier hierarchy is),
whether the domain has **interactions/connections worth a connect layer**, and which optional surfaces are
wanted (**regimen** decision layer? **articles**? **nightly auto**?). Lift the user's own words.

For the generic ICM shell questions (cadence, who runs it, stage naming) you may hand off to
`icm-workspace-builder` and come back — or ask them inline. Don't re-derive ICM theory for the user.

### GATE — Present the plan (before writing files)
Show, concisely: the folder shape you'll emit, the evidence-model frontmatter you'll use for this domain,
the tier hierarchy, whether a connect layer / regimen / articles / nightly-auto are in, and any
domain-specific naming. Call out the two beyond-ICM extensions. Then **stop for a go** (unless the user
already said "just build it").

### Phase 2 — Emit the workspace
Read `references/architecture.md` (the canonical shape + what earned its keep) and
`references/workspace-templates.md` (every file's template with `## File:` headers). Then write the tree:

```
<domain>-atlas/
  CLAUDE.md                      Layer 0: map + routing table + trigger keywords + rails
  GOAL.md                        Layer 3: north-star (ONLY if a regimen/decision layer is wanted)
  shared/                        Layer 3 reference, loaded per stage
    evidence-model.md             the grading rubric + status lifecycle + highest-tier-wins aggregation
    decision-rule.md              the regimen scoring rule           (only if regimen)
    voice-rules.md                article voice do/don'ts            (only if articles)
    safety.md                     domain safety/quality contract     (if the domain needs one)
    premise.md                    why this exists
    pdf-archival.md               the fetch ladder recipe            (if archiving source PDFs)
  stages/                        the pipeline as ICM stage contracts (Layer 2)
    01-sweep/CONTEXT.md
    02-ingest/CONTEXT.md          carries the connect checks (§6.5) from day one
    03-audit/CONTEXT.md           runs scripts/audit.py — the ICM Audit table
    04-synthesis-article/CONTEXT.md   (only if articles)
    05-refresh-regimen/CONTEXT.md     (only if regimen)
    06-commit-and-email/CONTEXT.md    (only if nightly auto)
    07-connect/CONTEXT.md             (only if a connect layer)
  vault/                         Layer 4 + persistent store
    index.md                      field map: groups + entries + Status + "Recently ingested" + Open gaps
    maps/  concepts/  sources/  raw/(git-lfs)
  regimen/current-protocol.md    (only if regimen)
  articles/index.md              (only if articles)
  scripts/audit.py               copied VERBATIM from this skill's scripts/
  scripts/fetch-pdf.py           copied if archiving PDFs (swap the resolver per domain — see file header)
  scripts/<domain>-radar-nightly.sh   the nightly driver          (only if nightly auto — references/scheduling.md)
  deploy/<label>.plist           the launchd schedule             (only if nightly auto)
  .claude/skills/<domain>-radar/SKILL.md   thin Layer-1 router into the stages
```

Rules while emitting:
- **Copy `scripts/audit.py` verbatim.** It's domain-agnostic (MOC vault vocabulary). Adjust only the FORK
  thresholds / required frontmatter keys if the domain truly differs.
- **Skills are thin routers.** The process lives in `stages/*/CONTEXT.md`, never in the skill.
- Only emit the optional surfaces (regimen/articles/auto/connect) the interview asked for. A minimal
  autoresearch workspace is sweep + ingest + audit + a connect layer — that's the irreducible core.
- Set up git-lfs for `vault/raw/*.pdf` if archiving PDFs.
- **If nightly auto is chosen, emit the actual wiring, not just the stage-06 contract** — the driver +
  plist from `references/scheduling.md`. A "self-updating" ledger that has no scheduled job isn't one. The
  scaffolder writes the two files and prints the `launchctl load` command; **it never installs the job** —
  loading a runner that pushes to `main` unattended is the human's authorization to give (Phase 5 names it).
- **Never wikilink a map/note you haven't created** in any emitted or seeded file — the audit fails on dead
  links. Intended-but-uncreated entries go in `vault/index.md` "Open gaps" as plain text, not as `[[links]]`.

### Phase 3 — Verify the gate runs
`python3 <dest>/scripts/audit.py` must exit 0 against the empty (index-only) vault. **A scaffold isn't
scaffolded until its audit gate exists and passes.** Fix the templates, not the gate, if it fails.

### Phase 4 — Seed one real source (prove the loop by hand)
Per the spec-discipline that produced this skill: **derive from a real run, don't trust the template.**
Ingest ONE real source end-to-end via stage `02` — a source digest, ≥1 graded concept note (with `source`
+ `evidence_tier`), a reconciled map, and the connect checks (titles-scan + global-ripple). Re-run the
audit gate. This surfaces template gaps while they're cheap to fix.

### Phase 5 — Report + stop
Summarize what was emitted, the seed source, the gate result, and the two beyond-ICM extensions. **Leave
everything uncommitted** — the commit gate is the human's hands on the keyboard. Only a later authorized
nightly `auto` job commits/pushes/emails.

## Rails the scaffolded workspace must carry (non-negotiable)

Bake these into the emitted `CLAUDE.md` + `shared/` — they're what makes autoresearch trustworthy:
- **Provenance mandatory.** Every concept note carries `source` + `evidence_tier`. Can't cite it → skip and
  log it, never guess. (The audit gate enforces the two fields.)
- **Status moves both ways.** A stronger source can downgrade a claim (`contested`/`fading`). Never bury a
  rebuttal under a confident summary. Highest-tier-wins aggregation, applied deterministically.
- **The connect layer is core, not a bolt-on.** Every ingest runs titles-scan + global-ripple; a monthly
  `connect` sweep hunts cross-corpus edges. In safety-bearing domains an interaction is safety-critical.
- **The audit gate is a hard stop.** It runs before any commit; a failing gate is a bug in the run, never
  something to commit around.
- **Bounded + revertible.** Cap the sweep and articles per run; git history + the email are the audit trail.
- **`auto` self-maintains, not just ingests.** The nightly walk (01→06) also: **auto-writes a synthesis
  article when a map newly reaches `consolidating`** (and refreshes on `contested`/drift, cap 1/run); and
  **auto-forks** a fork-due map (≥10 dated bullets + ≥3 sub-threads) into child maps — *bounded and
  git-revertible*, flagged prominently in the email. A freshly-forked child that later consolidates triggers
  its own write-due article next run (the follow-up chain). Interactive `synthesize`/fork stay human-gated.
- **Interactive = research → normalize → validate → STOP** (uncommitted). Only authorized `auto` commits.
- If the domain gives advice that could harm (health, legal, financial): a **not-advice** safety contract,
  no fabricated specifics (doses, dollar figures, legal conclusions), and safety-flagged items held as
  "proposed — review with a professional."

## References
- `references/architecture.md` — the canonical autoresearch shape, the five-layer mapping, the lifecycle,
  the connect layer (§6.5), and **what earned its keep vs. what to drop** (the learnings from both builds).
- `references/interview.md` — the domain-adaptation interview.
- `references/workspace-templates.md` — every emitted file as a fill-in template.
- `references/scheduling.md` — wiring the nightly `auto` runner (driver + launchd plist), generalized from
  the proven Memory Atlas job. Read this whenever nightly auto is chosen.
- `scripts/audit.py`, `scripts/fetch-pdf.py`, `scripts/pdf-archival.md` — the portable tooling.

## Prerequisites
- `icm-workspace-builder` (companion, for the generic ICM shell) — referenced, not required to run.
- `git` + `git-lfs` (only if archiving source PDFs). `python3` (stdlib only; the scripts have no deps).
- Optional domain fetch skills the workspace's ladder leans on: `web-to-md`, `playwright-browser`.
- **For nightly auto:** a shared launchd wrapper (logging + status channel + heartbeat + timeout + exit-code
  conventions, incl. the `exit 2` no-op) and a mailer CLI for the haul email — see `references/scheduling.md`.
  `claude` on PATH; `jq` for the readable log stream.


============================================================
REFERENCE: references/architecture.md
============================================================

# The autoresearch shape — architecture + learnings

The canonical shape this skill emits, and **what earned its keep vs. what to drop** — distilled from two
working builds: Memory Atlas (`~/source/memory-aggregator`, AI-memory research, `.ts`-typed + a Next.js
site) and Healthspan Atlas (`~/source/healthspan-atlas`, health interventions, pure markdown). Read this
before emitting a workspace.

## 1. The pattern in one paragraph

An agent, on a cadence, **sweeps** primary sources, **ingests** each into a persistent markdown **vault**
of graded claims, **reconciles** how each source moves a topic's *status* (a lifecycle that goes both
ways), reasons **across the corpus** for connections a single ingest can't see, and — optionally — writes
synthesis **articles** and a personal **decision layer**. Every change is hard-gated by a scripted
**audit**. Interactive runs stop for a human; an authorized nightly runner commits + emails.

## 2. Five-layer ICM mapping

| ICM layer | Autoresearch realization |
|---|---|
| L0 map | `CLAUDE.md` — workspace tree, routing table (trigger → stage), rails. Holds no process. |
| L1 route | the routing table + the thin `<domain>-radar` skill |
| L2 contract | `stages/<NN>/CONTEXT.md` — inputs → process → checkpoints → audit → outputs |
| L3 reference | `GOAL.md`, `shared/*.md` (evidence-model, decision-rule, voice-rules, safety, premise, pdf-archival) |
| L4 artifacts + **persistent store** | `vault/`, `regimen/`, `articles/` |

"Configure the factory, not the product": `GOAL.md` + `shared/` config the workspace once; each run
produces new artifacts from that config.

## 3. The MOC vault (the persistent store)

```
vault/
  index.md      field map: entries grouped by domain sub-area, each with Status; "Recently ingested"; "Open gaps"
  maps/         ONE file per topic/theme/intervention — the accreting through-line + "What we believe now" + Status
  concepts/     atomic graded claims — one finding each, evidence-grade frontmatter
  sources/      per-source digest (one per paper/filing/post) + provenance frontmatter
  raw/          archived source PDFs (git-lfs) + manifest.json
```

**Reachability invariant** (audit-enforced): every note reachable from `index.md` in ≤2 hops
(index → map → concept/source). No orphans. This is what keeps an accreting vault navigable instead of a
junk drawer.

**Concept-note frontmatter** — adapt the fields to the domain, but two are mandatory everywhere and the
audit fails without them: `source` and `evidence_tier`. A biomedical/AI-research shape:
```
topic: <topic-slug>            # which map this rolls up to
claim: <what it asserts>
direction: benefit | no-effect | harm | supports | refutes   # domain-appropriate
effect_size: small | moderate | large | none                 # magnitude, not just significance
evidence_tier: <domain tier>   # e.g. meta-analysis-rct > rct > cohort > mechanistic > anecdote
stance: confirms | pushes-back | extends | open-question      # vs the topic's current belief
strength: strong | suggestive | single-study
source: sources/<source-slug>  # MANDATORY
```

## 4. The status lifecycle (the load-bearing mechanic)

A map's Status, set by **highest-tier-wins aggregation** (not vote-counting): the *strongest* evidence
sets it, so one well-powered top-tier null outweighs several weak positive signals. Effect size + replication
gate the top — a high tier with a trivial effect does not consolidate.

| Status | Meaning |
|---|---|
| `emerging` | early / thin / single group |
| `consolidating` | ≥2 independent groups, high-tier, real effect, replicated, no live rebuttal |
| `consensus` | assumed by newer work. **Higher bar + reversible** — many fields' consensus flips. |
| `contested` | a credible higher-tier result pushes back; state the tension, never bury it |
| `fading` | superseded / debunked / gone quiet |

**Moves both ways is the point.** A good new source that nullifies an older signal **downgrades** the map.
This is the single most important behavior to bake into `shared/evidence-model.md` and the ingest stage.

**Fork-due**: a map with ≥10 dated through-line bullets AND ≥3 distinct `**(n)**` sub-threads is a split
candidate; the audit prints it as an advisory.
- **Interactive:** surfaced, human carves it.
- **`auto` mode (opt-in auto-fork):** when the *deterministic* criteria fire, the nightly run carves the
  clearest sub-thread(s) into child maps and leaves the parent as a **hub** that links them (no content lost;
  through-line preserved). Guardrails make this safe to automate: the trigger is mechanical (bullet + sub-thread
  counts), the split is conservative (only clearly-separable sub-threads), it's **git-revertible**, and every
  fork is **flagged prominently in the nightly email** for human eyes. **The follow-up chain falls out for
  free:** a freshly-forked child is a normal map, so once it reaches `consolidating` it trips stage 04's
  write-due and gets its own synthesis article next run. Keep auto-fork behind the same audit gate as everything
  else — a fork that orphans a note or breaks a link fails the gate and reverts.

## 5. The connect layer (§6.5 — cross-corpus reasoning)

Theme-local ingestion (one source → one map) systematically misses edges only visible when the whole vault
is read together. Three tiers matched to cost:
1. **Titles-scan before linking** (every ingest, cheap) — scan the full concepts index before writing any
   cross-link, so a new source links against *all* topics, not just the ones the agent recalled.
2. **Global ripple** (every ingest) — after filing a source in its home map, walk every *other* map: does
   this confirm / refute / **interact-with** its claims? A hit → cross-link; an interaction hit →
   `safety_flags: [interaction]` on the concept + a note in **both** maps.
3. **`connect` mode** (monthly + on demand, stage `07`) — load the whole claim layer in one context; hunt
   missing links, un-named tensions, interaction edges no map records. Apply high-confidence links;
   **flag, never auto-adopt** candidate topics. Degrade to map-reduce when the vault outgrows one context.

In safety-bearing domains (health, legal, financial) the connect layer is **safety-critical**, not tidiness:
it's where interactions/contraindications/conflicts live, and a missed edge is worse than an over-cautious
flag — so flag on any *credible* signal but keep interaction strength **out of** any benefit/ranking score.

## 6. The pipeline (stages)

- `01-sweep` — find recent sources not yet ingested; propose. Interactive: **stop** for greenlight.
- `02-ingest` — one source → digest + graded concepts + reconciled map, **with the connect checks**. Core.
- `03-audit` — run `scripts/audit.py`. The ICM Audit table as a script. Hard stop.
- `04-synthesis-article` *(optional)* — one article per `consolidating` topic; write / contested-refresh /
  drift-refresh; ≤1 per run.
- `05-refresh-regimen` *(optional)* — the personal decision layer; re-score against `GOAL.md`.
- `06-commit-and-email` *(optional)* — the authorized `auto` tail: commit + push + email the haul.
- `07-connect` *(optional but recommended)* — the monthly full-vault sweep.

Irreducible core = `01` + `02` + `03` + a connect layer. Everything else is opt-in per domain.

**The `auto` walk is self-maintaining, not just an ingest loop.** A nightly `auto` run walks 01→06 and, in one
pass: sweeps → ingests (≤5) → audits (hard gate) → **auto-writes/refreshes one synthesis article** (stage 04
write-due on new-consolidation; refresh on contested/drift) → **auto-forks** any fork-due map (§4) → commits +
pushes + emails. If nothing changed it exits no-op (no commit, no email). This whole loop is **proven live in
Memory Atlas** (the `research-radar-nightly` launchd job); the scaffolder now emits its wiring — see
`references/scheduling.md`. It is the difference between a "self-updating ledger" and a folder you have to
remember to feed.

## 7. Optional: the personal decision layer (regimen)

If the domain has a "so what should *I* do" question (a health regimen, a portfolio, a tool stack), add a
`regimen/current-protocol.md` generated by an **auditable decision rule** in `shared/decision-rule.md`
(ordinal scores, explicit thresholds — a floor for honesty and inspectability, not precision theater).

Key design decision proven in both builds: **personal fit is NOT stored in the vault.** The concept note
records the *neutral* fact (e.g. `study_population`); fit-to-the-user is computed at regimen time against
`GOAL.md`. This keeps the vault shareable and means the user's profile can change without staling a note.
**The regimen is itself a connect operation** — score everything, then reconcile the winners for
interactions/redundancy; it is *constructed* by traversing the vault, not *retrieved* from one entry.

## 8. What earned its keep — and what to drop (the learnings)

- **The scripted audit gate earns its keep.** Both builds rely on it; it catches dead links, orphans,
  missing provenance, and (for articles) unbalanced markdown / dead internal links / bare URLs. Ship it as
  stage `03` *and* the nightly hard stop. Non-negotiable.
- **Keep artifacts in markdown unless a rendered site truly needs typed data.** Memory Atlas kept articles
  as `.ts` and hit a recurring bug: raw inline-code backticks inside template literals broke `pnpm validate`.
  Healthspan Atlas kept articles as `.md` and the class of bug **does not exist**. Default to markdown; add a
  typed layer only when a public site earns it (a later phase, not the scaffold).
- **PDF archival: ship a deterministic script (the 60%), and key on the right signal.** Don't hand the agent
  "go download PDFs" and hope. For biomedical, resolve via Europe PMC's `fullTextUrlList` (Free/Open +
  `documentStyle: pdf`) — **NOT** `isOpenAccess`, which is `N` for free NIH author manuscripts. Then a
  documented fallback ladder (`web-to-md` → `playwright-browser`). "curl 404'd" is never a valid stopping
  point. See `scripts/pdf-archival.md`.
- **Decouple personal fit from the vault** (§7). This was a deliberate revision in both specs; it's what
  makes the vault reusable and the scaffolder domain-general.
- **Name "resolved interaction" as a first-class state.** A flag that a *foundation resolves* (e.g. a timing
  rule that fixes an evening-stimulant-vs-sleep conflict) is common. `shared/safety.md` should distinguish
  *unresolved* flag (caps the item at "proposed — review with a professional") from *resolved-by-X*
  (co-recommend, but only with the resolution stated inline) — so a nightly `auto` run doesn't over-cap.
- **Sorting and sectioning are two steps in the regimen.** The decision rule scores an item; the section
  model then routes it (e.g. a nutrition item that scores into "Protocol" numerically still belongs under
  "Foundations"). Make both steps explicit in the stage contract.
- **Skills stay thin; process lives in stages.** Every skill is a trigger + "read CLAUDE.md, walk these
  stages." Resist putting process in the skill — it's the ICM discipline that lets the workflow be edited as
  plain markdown.
- **Cadence honesty.** Slower-moving domains (health) no-op most nights; that's fine. Cap the sweep and dedup
  hard so nightly stays purposeful. Fast domains (AI papers) may want a tighter cap and a shorter connect cadence.
- **Emit the scheduled job, not just its contract.** The first two builds only *described* stage 06; the actual
  nightly wiring lived by hand in Memory Atlas and never in the scaffolder — so scaffolded domains looked
  "self-updating" but weren't. The scaffold now emits the driver + launchd plist (`references/scheduling.md`),
  and the human does the one outward step (`launchctl load`). *(NAMS-run learning, 2026-07.)*
- **Guidance/standards docs have no PMID/DOI.** OECD/EPA/FDA/ISO PDFs can't be resolved by the biomedical
  `fetch-pdf.py`; they skip rung 1 and go to the `web-to-md` rung, and they're public (archive them, don't mark
  them paywalled). Noted in `scripts/pdf-archival.md`. *(NAMS-run learning.)*
- **Never wikilink a note you haven't created.** The connect layer will *want* to link entries that don't exist
  yet (a battery's component assays, a fork's future child); doing so fails the audit's dead-link check. Log
  them as plain-text "Open gaps" instead — the audit is the forcing function that keeps the vault honest. *(NAMS-run learning.)*
- **Derive, don't speculate.** Both builds were hand-run before any automation; this skill was extracted
  only after. When scaffolding a new domain, Phase 4 (seed one real source by hand) is the same discipline in
  miniature — it surfaces the template gaps the interview couldn't predict.

## 9. Governance rails (bake into every scaffold)

Provenance mandatory · status moves both ways · connect layer core · audit gate is a hard stop · bounded +
revertible (caps + git + email) · interactive stops uncommitted, only authorized `auto` commits · a
not-advice safety contract for any domain that could harm.


============================================================
REFERENCE: references/interview.md
============================================================

# The domain-adaptation interview

Ask only what *specializes the shape* — the autoresearch machinery (vault, lifecycle, connect layer, audit
gate) is fixed; the interview decides how it's tuned for this domain and which optional surfaces exist.
One question at a time; reflect each answer; lift the user's own words into the emitted files. Hand the
generic ICM shell questions (who runs it, cadence, stage naming) to `icm-workspace-builder` if you like.

### Q1 — Domain + purpose (one sentence)
> What field is this tracking, and in one sentence what should someone have at the end that they don't now?

Sets the workspace name (`<domain>-atlas` by default), `CLAUDE.md` header, and `shared/premise.md`.

### Q2 — Primary sources (the "arXiv equivalent") + fetch ladder
> Where does the authoritative primary material come from? (Papers on PubMed/arXiv? SEC filings? Standards
> bodies? Vendor changelogs?) And which are *leads* vs. *sources* — what points at evidence vs. what grounds
> a claim?

Sets `stages/01-sweep`, `shared/pdf-archival.md` (if archiving), and the source-note frontmatter. Note the
tiering: primary literature grounds claims; secondary/opinion (blogs, pundits) are *leads, not sources*.

### Q3 — The claim/evidence model + tier hierarchy
> When a source makes a claim in this field, how do you judge how much to trust it? What's the ranking from
> strongest to weakest kind of evidence?

This is the most domain-specific answer. It sets `shared/evidence-model.md`:
- the concept-note frontmatter fields (always incl. `source` + `evidence_tier`),
- the **tier hierarchy** (biomedical: meta-analysis-rct > rct > cohort > mechanistic > anecdote; AI research:
  reproduced-benchmark > paper-with-code > paper > blog-claim; law: binding-precedent > persuasive > commentary; etc.),
- confirm the **highest-tier-wins** aggregation and the **moves-both-ways** lifecycle apply (they almost always do).

### Q4 — Connections / interactions
> Do entries in this field interact, combine, or contradict in ways you'd miss looking at one at a time?
> (Drug interactions, tool incompatibilities, conflicting rulings, stack synergies?)

If yes (usually) → emit the connect layer: titles-scan + global-ripple in `02-ingest`, and a `07-connect`
stage. If the interactions can *harm* → the connect layer is safety-critical; emit `shared/safety.md` and
the "resolved vs unresolved flag" distinction.

### Q5 — Optional surfaces
> Three add-ons — want any? (a) **Synthesis articles** — one per topic that reaches "consolidating",
> written in a house voice. (b) **A personal decision layer** — a "so what should *I* do" ranked
> regimen/protocol scored against your goals. (c) **A nightly unattended runner** that commits + emails.

- Articles → `stages/04`, `shared/voice-rules.md`, `articles/index.md`.
- Decision layer → `stages/05`, `GOAL.md` (interview the user's goals/constraints), `shared/decision-rule.md`,
  `regimen/current-protocol.md`. Also triggers `shared/safety.md` if the advice could harm.
- Nightly auto → `stages/06`, and document the authorized-unattended-runner extension + the audit-gate-as-checkpoint.

### Q6 — Safety / advice exposure
> Could acting on this system's output hurt someone (health, money, legal exposure)?

If yes → mandatory `shared/safety.md`: a not-advice contract, no fabricated specifics (doses, dollar
figures, legal conclusions), safety-flagged items held as "proposed — review with a professional", and the
resolved/unresolved flag rule.

### If a regimen is wanted — the GOAL.md sub-interview
Ask the user's **philosophy** (what they optimize for), **profile** (the neutral facts fit is computed
against), **hard constraints** (gates), and **decision weighting** (how to trade the scoring factors). Keep
personal facts in `GOAL.md`, never in the vault. Convert relative dates to absolute.


============================================================
REFERENCE: references/scheduling.md
============================================================

# scheduling.md — wiring the nightly `auto` runner (the critical component)

Emit this when the interview picks **nightly auto**. The stage-06 *contract* is not enough — a research
ledger that doesn't run itself isn't self-updating. This is the actual wiring, generalized from the proven
Memory Atlas job (`research-radar-nightly.sh` + its launchd plist + the shared `launchd_wrapper.sh`).

**Three parts:** launchd plist (the schedule) → shared `launchd_wrapper.sh` (logging, Slack, heartbeat,
timeout) → the **domain driver** (headless Claude in `auto` mode). The scaffolder emits the driver + plist
into the repo; **the human loads the plist** (`launchctl`) — that go-live step is outward-facing and stays a
human action, never something the scaffold does.

## What the scaffolder emits (macOS/launchd)
- `scripts/<domain>-radar-nightly.sh` — the domain driver (template below).
- `deploy/<label>.plist` — the launchd schedule (template below); `<label>` = `com.<user>.<domain>-radar-nightly`.
Linux swap: a systemd timer + service unit; the driver script is identical.

## Dependencies (prerequisites, referenced not vendored)
- A shared launchd wrapper providing logging + a status channel + heartbeat + timeout + exit-code conventions
  (David's `~/source/scripts-and-agents/scripts/launchd_wrapper.sh`). **Exit-code contract the driver must honor:**
  `0` success · `2` **no-op → wrapper skips the notification** (use when nothing new was found) · `3` completed-with-warnings · other = failure.
- A mailer CLI the driver calls for the haul email (David's `.../email_sender/send_email.py --to … --subject … --markdown-file …`).
- `claude` on PATH; `jq` for the readable log stream.

## The domain driver — template (`scripts/<domain>-radar-nightly.sh`)

```bash
#!/bin/bash
# <domain>-radar-nightly — autonomous nightly sweep+ingest for <TITLE> Atlas.
# Called by launchd through launchd_wrapper.sh. Headless Claude in `auto` mode:
# sweep -> ingest (cap 5) -> audit (HARD GATE) -> article (auto write/refresh, cap 1) ->
# auto-fork (bounded) -> commit + push -> email. Authorized to run unattended and push to main.
set -euo pipefail

# Absolute PATH so headless Claude never Globs/Greps for binaries (trips macOS TCC + hangs).
export PATH="{{ABSOLUTE_PATH_LIST}}:$PATH"
REPO={{ABSOLUTE_REPO_PATH}}
cd "$REPO"

# Never run on top of uncommitted human work or a stale tree.
if [ -n "$(git status --porcelain)" ]; then
  echo "[<domain>-radar-nightly] working tree dirty — aborting." >&2; exit 1
fi
git fetch origin main && git pull --ff-only origin main

# --strict-mcp-config + empty server set: this job needs zero MCP servers (built-in WebSearch/WebFetch +
# Bash/Read/Write/Edit/git). Without it, headless claude spawns every configured server — a hang risk.
# stream-json (+ --verbose) streams events; tee the raw NDJSON, pipe a readable one-liner per event.
RAW_LOG={{ABSOLUTE_LOG_DIR}}/<domain>-radar-nightly.raw.ndjson
JQ_FMT='(fromjson? // empty) as $e
  | if $e.type=="assistant" then ($e.message.content[]? |
      if .type=="text" then "  💬 "+(.text|gsub("\n";" ")|.[0:200])
      elif .type=="tool_use" then "  → "+.name+" "+((.input|tostring)|gsub("\n";" ")|.[0:160]) else empty end)
    elif $e.type=="result" then "  ✅ "+(($e.result//"")|gsub("\n";" ")|.[0:200])
    else empty end'
set +e   # so PIPESTATUS[0] captures claude's real exit code (tee/jq would mask it)
claude -p --verbose --output-format stream-json \
  --permission-mode bypassPermissions --strict-mcp-config --mcp-config '{"mcpServers":{}}' \
  -- \
"Run the /<domain>-radar skill in **auto** mode end-to-end. Skill: $REPO/.claude/skills/<domain>-radar/SKILL.md \
(read its auto row + walk stages 01→06). Conventions: $REPO/CLAUDE.md. Run \`date\`. \
Sweep primary sources for material NOT already in $REPO/vault/sources/ (dedup); ingest AT MOST 5 through the \
full ingest flow (digest + graded concepts + reconcile maps + the connect checks), auto-coining a new map \
when warranted and flagging each. Run \`python3 scripts/audit.py\` as a HARD STOP — on failure that isn't \
trivially fixable, 'git checkout -- vault articles', email the failure, exit non-zero (never commit a broken vault). \
ARTICLES (cap 1/run, priority contested-refresh > write > drift-refresh): WRITE when a map newly became \
'consolidating' this run with no article yet; REFRESH in place (stable slug) when its map went 'contested' or \
its through-line drifted ≥3 dated bullets past the article's updatedAt. \
AUTO-FORK (bounded): if a map is fork-due (≥10 dated bullets AND ≥3 distinct sub-threads), carve the clearest \
sub-thread(s) into child maps, leave the parent as a hub linking them, and FLAG each fork prominently in the \
email; a freshly-forked child that itself is already 'consolidating' triggers a follow-up write next run. \
If ≥1 source was ingested OR an article was written/refreshed OR a fork happened: 'git add -A', commit to main \
(message lists each source URL + any new/forked map + any article slug+trigger), 'git push origin main', then \
email via {{MAILER_CMD}} with the haul (per-source summary; new/forked maps flagged; article embedded if written). \
If NOTHING new: do nothing and exit 2 (no-op; the wrapper skips the notification). \
Use WebSearch/WebFetch. Absolute paths only. Do NOT Glob/Grep for binaries or wander Desktop/Documents/Downloads/iCloud/Drive." \
  | tee "$RAW_LOG" | jq -Rr --unbuffered "$JQ_FMT"
rc=${PIPESTATUS[0]}; set -e; exit "$rc"
```

## The launchd plist — template (`deploy/<label>.plist`)

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>Label</key><string>{{LABEL}}</string>
  <key>ProgramArguments</key><array>
    <string>/bin/bash</string>
    <string>{{ABSOLUTE_WRAPPER_PATH}}</string>
    <string><domain>-radar-nightly</string>
    <string>/bin/bash</string>
    <string>{{ABSOLUTE_REPO_PATH}}/scripts/<domain>-radar-nightly.sh</string>
  </array>
  <key>StartCalendarInterval</key><dict><key>Hour</key><integer>7</integer><key>Minute</key><integer>0</integer></dict>
  <key>EnvironmentVariables</key><dict><key>JOB_TIMEOUT_SECONDS</key><string>5400</string></dict>
</dict></plist>
```

## Go-live (the human's outward step — the scaffolder stops before this)
```
cp deploy/<label>.plist ~/Library/LaunchAgents/<label>.plist
launchctl load ~/Library/LaunchAgents/<label>.plist    # (or launchctl bootstrap gui/$UID …)
```
The scaffolder emits the two files and prints these commands; it does **not** run them — installing a job that
pushes to `main` unattended is the human's authorization to give.

## Cadence honesty
Health/tox/most non-AI domains no-op most nights — that's why the `exit 2` no-op path matters (silent nights,
no email spam). Fast domains (AI papers) fire more often; tighten the cap, not the cadence.


============================================================
REFERENCE: references/workspace-templates.md
============================================================

# Workspace templates

Every file the scaffolder emits, as a fill-in template. Replace `{{TOKENS}}`; drop the files for optional
surfaces the interview didn't ask for. Placeholders: `{{DOMAIN}}` (slug, e.g. `healthspan`), `{{TITLE}}`
(e.g. `Healthspan`), `{{PURPOSE}}`, `{{TIERS}}` (the domain tier hierarchy), `{{GROUPS}}` (index sub-areas),
`{{SOURCES}}` (primary source list). Outer fences here are `~~~~` so inner ``` and `---` are safe to copy.

Scale each stage `CONTEXT.md` to ICM norms: routing/contract-only, <80 lines. `scripts/audit.py` and
`scripts/fetch-pdf.py` are copied **verbatim** from this skill's `scripts/`, not templated.

---

## File: CLAUDE.md

~~~~markdown
# {{TITLE}} Atlas — workspace map (Layer 0)

This repo is an **ICM workspace** (Interpreted Context Methodology): the folder structure is the
architecture, markdown is the interface. It's a self-updating research ledger for **{{PURPOSE}}**.
Design/premise: `shared/premise.md`.

## The five-layer context model — load only what the step needs

| Layer | File(s) | "…" |
|---|---|---|
| 0 map | `CLAUDE.md` (this file) | Where am I? |
| 1 route | this table + `stages/` | Where do I go? |
| 2 contract | `stages/<NN>/CONTEXT.md` | What do I do here? |
| 3 reference | `GOAL.md`, `shared/*.md` | What rules apply? |
| 4 artifacts | `vault/`, `regimen/`, `articles/` | What am I working with? |

## Routing table (trigger → stage)

| Trigger | Does | Stage(s) |
|---|---|---|
| `sweep` | find recent sources not yet ingested; propose (interactive: stop) | `01-sweep` |
| `ingest <url\|id>` | one source → digest + graded concepts + reconcile map (+ connect checks) | `02-ingest` → `03-audit` |
| `connect` | monthly full-vault cross-entry sweep | `07-connect` |
| `status` | print vault state | (read `vault/index.md`) |
{{OPTIONAL_ROUTES}}
| `auto` | unattended nightly: sweep → ingest (≤5) → audit → … → commit+push+email | `01`→`06` |

## Rails (non-negotiable)
- Provenance mandatory: every concept note carries `source` + `evidence_tier`. Can't cite it → skip and note.
- Status moves **both ways**: a stronger source can downgrade a claim.
- The connect layer is core (titles-scan + global-ripple every ingest; monthly `connect`).
- Bounded: ≤5 sources/run, ≤1 synthesis article/run.
- The audit gate `python3 scripts/audit.py` is a hard stop before any commit.
- Interactive runs stop uncommitted; only the authorized `auto` job commits/pushes/emails.
{{SAFETY_LINE}}
~~~~

`{{OPTIONAL_ROUTES}}` adds `synthesize` (→04), `regimen` (→05) as the interview dictates. `{{SAFETY_LINE}}`
is the not-advice one-liner for advice-bearing domains, else omitted.

---

## File: shared/premise.md

~~~~markdown
# premise.md — why this exists (Layer 3)

{{PURPOSE}}, tracked one source at a time as the evidence *moves*. The value is the accreting, trajectory-aware
ledger + the cross-corpus connections — not a one-shot report. Guide, not guru.
~~~~

---

## File: shared/evidence-model.md

~~~~markdown
# evidence-model.md — grading, aggregation, lifecycle (Layer 3)

How a source becomes a graded claim, and how a set of claims sets a topic's Status. Applied by `02-ingest`,
read by `03-audit` (required fields) and `05` (if a regimen exists).

## Concept-note frontmatter (one atomic finding each)
- `topic:` which map this rolls up to
- `claim:` what it asserts
- `direction:` {{DIRECTIONS}}
- `effect_size:` magnitude (not just significance)
- `evidence_tier:` {{TIERS}}          # MANDATORY
- `stance:` confirms | pushes-back | extends | open-question
- `strength:` strong | suggestive | single-study
- `source:` sources/<slug>            # MANDATORY

`source` and `evidence_tier` are mandatory — the audit fails a note missing either.
{{PERSONAL_FIT_NOTE}}

## Tier hierarchy (strongest → weakest)
{{TIERS}}

## Status aggregation — highest-tier-wins, not vote-counting
A map's Status is set by its *strongest* evidence; one top-tier null outweighs several weak positives. Effect
size + replication gate the top. Apply deterministically.

## Status lifecycle (moves BOTH ways)
emerging → consolidating → consensus (higher bar, reversible); contested / fading are downgrades. A strong
higher-tier `pushes-back` **downgrades** the map — rewrite "What we believe now" to carry the tension.

## Fork-due
≥10 dated through-line bullets AND ≥3 distinct `**(n)**` sub-threads → a split candidate. Interactive: a
human carves child maps. `auto`: bounded auto-fork (carve clear sub-threads into children, parent becomes a
hub, git-revertible, flagged in the email); a forked child that later consolidates gets its own article.
~~~~

`{{PERSONAL_FIT_NOTE}}` (only if a regimen exists): "**Personal fit is NOT in the vault** — record the
neutral fact; fit is computed at regimen time (`05`) vs `GOAL.md`."

---

## File: stages/01-sweep/CONTEXT.md

~~~~markdown
# Stage 01: Sweep

Find recent primary sources not yet in the vault; propose them. **Interactive: STOP** — never ingest without
a greenlight.

## Inputs
- Primary sources: {{SOURCES}}
- `vault/index.md` (what's already ingested; dedup against it)

## Process
1. Run `date`. Search the primary sources for material newer than the last ingest.
2. Dedup against `vault/sources/`. Rank by relevance + evidence tier (primary over secondary/opinion).
3. Present a candidate list (title, source, why it matters). Cap at ≤5.

## Audit / Outputs
Candidate list only. No vault writes. Human picks what to ingest.
~~~~

---

## File: stages/02-ingest/CONTEXT.md

~~~~markdown
# Stage 02: Ingest

One source → a source digest + graded concept note(s) + a reconciled map. **Not done until the connect
checks have run** (safety-critical where the domain has interactions).

## Inputs
- The source (url/id) · `shared/evidence-model.md` · `shared/pdf-archival.md` (if archiving) · the full
  `vault/concepts/` + `vault/maps/` (for the connect checks)

## Process
1. Fetch + read the source (climb the fetch ladder; archive the PDF per `shared/pdf-archival.md`).
2. Write `vault/sources/<slug>.md` (digest + provenance frontmatter). Grade each finding into
   `vault/concepts/<slug>.md` with the mandatory `source` + `evidence_tier`.
3. Reconcile the topic map (`vault/maps/<topic>.md`): update "What we believe now", append a dated
   through-line bullet, recompute Status (highest-tier-wins; **moves both ways**).
4. **Connect checks (§6.5):** (a) titles-scan the full concepts index before writing any cross-link;
   (b) global-ripple — walk every *other* map: confirm / refute / **interact-with**? A hit → cross-link;
   an interaction → `safety_flags: [interaction]` on the concept + a note in **both** maps.
5. Update `vault/index.md` ("Recently ingested"; add the map if new). **Never wikilink a map/note you
   haven't created** — the audit fails on dead links. An entry you *intend* to add later (a battery's
   component, a fork's future child) goes in "Open gaps" as plain text, not as a `[[link]]`.
6. **Fork-due check:** if the reconciled map now has ≥10 dated bullets AND ≥3 distinct `**(n)**` sub-threads,
   it's fork-due. Interactive: flag it for the human. `auto`: carve the clearest sub-thread(s) into child
   maps, leave the parent as a hub linking them, and flag the fork in the email (bounded + git-revertible).

## Audit / Outputs
`stages/03-audit` must pass. Leave uncommitted; hand off.
~~~~

---

## File: stages/03-audit/CONTEXT.md

~~~~markdown
# Stage 03: Audit (the ICM Audit gate)

The hard stop before any commit. Unambiguous pass/fail.

## Process
1. Run `python3 scripts/audit.py`.
2. Non-zero exit → **fix the reported issues**; never commit around them. A failing gate is a bug in the run.
3. In `auto`: an unfixable failure → `git checkout -- vault articles`, email the failure, exit non-zero.

## What it checks (gating): reachability (≤2 hops from index), dead wikilinks, required concept frontmatter
(`source` + `evidence_tier`), index drift, article markup (balanced `**`/backticks, closed links, resolving
internal links, no bare URLs). Advisories: fork-due, raw-pdf-drift.
~~~~

---

## File: stages/07-connect/CONTEXT.md  (recommended)

~~~~markdown
# Stage 07: Connect (monthly full-vault cross-entry sweep, §6.5)

Load the whole claim layer in one context; hunt missing links, un-named tensions, and interaction edges no
single ingest saw. Apply high-confidence links + map clauses; **flag, never auto-adopt** candidate topics or
interaction notes. Degrade to per-topic map-reduce when the vault outgrows one context read. Run `03-audit`
after. In safety-bearing domains this is the interaction/contraindication safety net.
~~~~

---

## File: stages/04-synthesis-article/CONTEXT.md  (only if articles)

~~~~markdown
# Stage 04: Synthesis article

One article per `consolidating` topic, refreshed in place as the through-line moves. ≤1 written-or-refreshed
per run. Triggers (priority contested > write > drift): write-due (newly consolidating, no article yet);
contested-refresh (a map with an article went contested — the live piece may be wrong); drift-refresh (≥3
new bullets since `updatedAt`). Follow `shared/voice-rules.md`. Register in `articles/index.md`. Run `03`.
Frontmatter: slug, title, type: synthesis, sourceTopic, createdAt, updatedAt.
~~~~

---

## File: stages/05-refresh-regimen/CONTEXT.md  (only if regimen)

~~~~markdown
# Stage 05: Refresh regimen

Re-score entries against `GOAL.md`; rewrite `regimen/current-protocol.md`. **A global synthesis, not a
per-item lookup** — score everything, THEN reconcile winners for interactions/redundancy.
1. For every map, compute Benefit + Burden (`shared/decision-rule.md`), computing personal fit vs `GOAL.md`.
2. Sort into the section model (scoring and sectioning are TWO steps — a high score doesn't override the
   section a lever belongs in).
3. **Across-protocol interaction check**: every pair of proposed items. Unresolved interaction → both marked
   "proposed — review with a professional"; resolved-by-X → co-recommend with the resolution stated inline.
4. Hard overrides (`GOAL.md`/`shared/safety.md`): prescriptions/regulated actions → "discuss with a pro"
   subsection only; hard gates block section entry; never invent a specific (dose/figure). Run `03`.
~~~~

---

## File: vault/index.md

~~~~markdown
# State of {{TITLE}} — field map

The map of what the evidence says, one source at a time. Each entry links to its map; open it and read the
through-line top-to-bottom. New notes are added by `{{DOMAIN}}-radar` via `02-ingest`.

Grouped by sub-area. Status: emerging / consolidating / consensus / contested / fading (see
`shared/evidence-model.md`).

{{GROUPS}}

## Recently ingested
- _(none yet)_

## Open gaps (living log)
- _(none yet)_
~~~~

`{{GROUPS}}` is a `## Sub-area` heading per domain grouping, each listing `- [[maps/<slug>]] — **Status** · one-liner`.

---

## File: vault/raw/README.md

~~~~markdown
# raw/ — archived source PDFs (git-lfs)

One `<source-slug>.pdf` per source note, tracked in git-lfs, keyed to the source slug. `manifest.json` maps
slug → {source_url, sha256, bytes, retrieved}. Licensing: OA / author-manuscript / preprint bytes only;
paywalled sources get a manifest entry + `pdf_unavailable` reason, never the bytes. See
`shared/pdf-archival.md` for the fetch ladder.
~~~~

---

## File: articles/index.md  (only if articles)

~~~~markdown
# Articles — manifest

Source of truth for articles. Type: **synthesis** (trajectory-driven, one per consolidating topic).

| slug | title | type | sourceTopic | createdAt | updatedAt | published |
|------|-------|------|-------------|-----------|-----------|-----------|
| _(none yet)_ | | | | | | |
~~~~

---

## File: .claude/skills/{{DOMAIN}}-radar/SKILL.md

~~~~markdown
---
name: {{DOMAIN}}-radar
description: The {{TITLE}} Atlas engine. Use when the user says ingest / sweep / connect / auto (or "run the radar") in the {{DOMAIN}}-atlas repo. A thin router into stages/; the process lives in the stage contracts, not here.
---

# {{DOMAIN}}-radar — engine entry point (Layer 1 router)

This skill only **routes**; it holds no process. Read `CLAUDE.md` (Layer 0) first, run `date`, then walk the
stages the mode names, following each `stages/<NN>/CONTEXT.md` contract exactly.

**Rails (from `CLAUDE.md`):** provenance mandatory; status moves both ways; every ingest runs the connect
checks (§6.5); bounded (≤5 sources/run, ≤1 article/run); the audit gate (`python3 scripts/audit.py`) must
exit 0 before any commit. Interactive: research → normalize → validate → **STOP** (uncommitted). Only `auto`
commits/pushes/emails.

## Modes → stages
| Mode | Walk | Then |
|---|---|---|
| `sweep` | `01-sweep` | Candidate list. Interactive: STOP. |
| `ingest <id>` | `02-ingest` → `03-audit` | One source, with the connect checks. Leave uncommitted. |
| `connect` | `07-connect` → `03-audit` | Monthly full-vault sweep; flag, never auto-adopt. |
{{OPTIONAL_MODES}}
| `auto` | `01`→`06` | Authorized unattended only; gate + caps replace human checkpoints. Self-maintains: ingest (≤5) → **auto-write/refresh 1 article** on new-consolidation → **bounded auto-fork** of fork-due maps → commit + push + email. No-op → exit 2 (no email). |
~~~~

The `auto` row is wired to run unattended by the nightly job — the driver script + launchd plist emitted from
`references/scheduling.md`. The human loads the plist (`launchctl load …`); the scaffolder never installs it.


============================================================
REFERENCE: scripts/pdf-archival.md
============================================================

# pdf-archival.md — how to actually get the source PDF (a shipped reference)

Copy this into a scaffolded workspace's `shared/pdf-archival.md` when the domain archives source PDFs. Read
by `stages/02-ingest`. Archival is non-blocking, **but "curl 404'd" is not a valid reason to give up** —
bot-walls are a solved problem; you climb the ladder. `pdf_unavailable` is only legitimate after the
**browser rung** genuinely fails or the source is confirmed **not open-access**.

## Why rung 1 (bare curl) usually fails
Academic hosts reject the default `curl` user-agent (bot-detection → 404), and OA tarball endpoints are
often FTP (proxy-blocked in sandboxes). Publishers paywall the PDF. So a bare `curl` almost always fails on
scholarly corpora — that's expected, not a dead end.

## The ladder (climb until one works)
Given a source note's identifier(s) (`pmid`/`pmcid`/`doi`, or an arXiv id, or a canonical URL):

1. **Run the script (rung 1, deterministic):** `python3 scripts/fetch-pdf.py <slug>`. The shipped resolver
   targets **biomedical** literature via Europe PMC and keys on the **`fullTextUrlList`** (an entry with
   `availability` Free/Open access **and** `documentStyle` = `pdf`) — **the correct signal, NOT
   `isOpenAccess`**, which is `N` for free author manuscripts. For **other domains, swap `resolve_free_pdf()`**:
   - arXiv → derive `https://arxiv.org/pdf/<id>` directly.
   - general DOI → Unpaywall / OpenAlex `best_oa_location`.
   If the script prints `✅`, done. If it flags the slug (`⛔` no free PDF / `!` render error), climb.
   - **Regulatory / guidance / standards-body docs (OECD, EPA, FDA, ISO, ECVAM…) have no PMID/DOI**, so the
     script can't resolve them — skip rung 1 for these and go straight to rung 2. They are usually *public
     and freely distributable*, so they belong in `raw/` once fetched; a `pdf_unavailable` note on one should
     say "public guidance PDF, fetch via web-to-md" — not treat it as paywalled.
2. **`web-to-md` skill** on the DOI/landing/article/guidance page — routes through backends that render where curl can't.
3. **`playwright-browser` skill** — a real browser; defeats bot-detection and JS-gated PDFs. Drive it to the
   free-PDF URL or OA landing page and save the PDF.
4. **Only now**, if the browser rung fails *or* the source is confirmed non-OA (paywalled, no free deposit):
   record `pdf_unavailable: <specific reason + what was tried + OA status>` and keep the manifest entry.
   Never archive paywalled bytes.

**Gotcha — a free PDF can exist while its host is down.** Some render endpoints (e.g. Europe PMC's
`?pdf=render` / `ptpmcrender.fcgi`) return HTTP 500 intermittently — that's neither non-OA nor a bot-wall.
Diagnose as "free copy exists, host route broken"; retry later, or manual-import.

**Manual import** (paywalled-but-you-have-access, or a broken free host): obtain the PDF yourself, then
`python3 scripts/fetch-pdf.py --manual <slug> <file.pdf>` — verifies the `%PDF` header, hashes, writes the
`pdf_*` frontmatter (`fetched_via: manual import`), and updates the manifest.

## After a successful fetch
Write to `vault/raw/<source-slug>.pdf`, replace `pdf_unavailable` in the note frontmatter with `pdf_path` /
`pdf_sha256` / `pdf_retrieved` / `pdf_bytes`, note the winning rung in `fetched_via`, and update
`vault/raw/manifest.json`. Licensing: OA / author-manuscript / preprint bytes only; paywalled → manifest +
reason, never the bytes.

## The rule
Stopping at `curl` and writing `pdf_unavailable` is a **bug**, not a miss. A genuine `pdf_unavailable` names
the browser-rung failure or the non-OA status — and lists what was tried.


============================================================
SCRIPT: scripts/audit.py
============================================================

#!/usr/bin/env python3
"""Autoresearch audit gate (the ICM Audit table, as a script).

The hard stop before any commit. Two parts, both gating:
  (a) Vault structural lint — index exists, reachability (<=2 hops from vault/index.md),
      dead wikilinks, required concept frontmatter (source + evidence_tier), index drift.
  (b) Article/prose lint — balanced ** and backticks, closed ](...) links, resolving internal
      links, no bare URLs in prose (articles/*.md, excluding the manifest index.md).

Advisories (printed, NON-gating): fork-due, raw-pdf-drift.

This is domain-agnostic: the MOC vault vocabulary is
  maps/     — one file per theme/topic/intervention (the accreting through-line)
  concepts/ — atomic, graded claims (one finding each; carry `source` + `evidence_tier`)
  sources/  — per-source digests
Adapt FORK thresholds / required frontmatter keys to the domain if needed, but the shape holds.

Zero dependencies — python3 stdlib only. Exit 0 = pass; exit 1 = gate failure.
"""
import json
import re
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent.parent
VAULT = ROOT / "vault"
ARTICLES = ROOT / "articles"

FORK_MIN_BULLETS = 10
FORK_MIN_SUBTHREADS = 3

WIKILINK = re.compile(r"\[\[([^\]]+?)\]\]")
MD_LINK = re.compile(r"\]\(([^)]*)\)")
MALFORMED_LINK = re.compile(r"\]\(")           # every "](" must be followed by a closing ")"
MALFORMED_LINK_OK = re.compile(r"\]\([^)]*\)")
BARE_URL = re.compile(r"https?://[^\s)`'\"]+")
FENCE = re.compile(r"^\s*(```|~~~)")
CODE_SPAN = re.compile(r"`[^`]*`")
INTERNAL_LINK = re.compile(r"\]\((?:\.\./)?(vault/)?(maps|articles|concepts|sources)/([a-z0-9-]+)(?:\.md)?\)")


def strip_code_blocks(text: str) -> str:
    """Drop fenced code blocks so template/example links aren't treated as real."""
    out, in_fence = [], False
    for line in text.splitlines():
        if FENCE.match(line):
            in_fence = not in_fence
            continue
        if not in_fence:
            out.append(line)
    return "\n".join(out)


def frontmatter(text: str) -> str:
    if not text.startswith("---"):
        return ""
    end = text.find("\n---", 3)
    return text[:end] if end != -1 else ""


def note_ids(md_files):
    """Every resolvable id: path-relative-to-vault (no .md) AND bare basename."""
    ids = {"index", "README"}
    for f in md_files:
        rel = f.relative_to(VAULT).as_posix()[:-3]  # strip .md
        ids.add(rel)
        ids.add(f.stem)
    return ids


def links_in(text: str):
    for m in WIKILINK.finditer(strip_code_blocks(text)):
        yield m.group(1).split("|")[0].split("#")[0].strip().removesuffix(".md")


def main() -> int:
    failures, advisories = [], []

    if not (VAULT / "index.md").exists():
        print("FAIL [missing-index] vault/index.md does not exist")
        return 1

    maps = sorted((VAULT / "maps").glob("*.md")) if (VAULT / "maps").exists() else []
    concepts = sorted((VAULT / "concepts").glob("*.md")) if (VAULT / "concepts").exists() else []
    sources = sorted((VAULT / "sources").glob("*.md")) if (VAULT / "sources").exists() else []
    vault_md = [VAULT / "index.md"] + maps + concepts + sources
    ids = note_ids(vault_md)

    # --- (a) structural -------------------------------------------------------
    # dead wikilinks
    for f in vault_md:
        text = f.read_text(encoding="utf-8")
        for target in links_in(text):
            if target and target not in ids:
                failures.append(f"[dead-link] {f.relative_to(ROOT)} -> [[{target}]]")

    # reachability: index -(hop1)-> maps -(hop2)-> concepts/sources
    index_text = (VAULT / "index.md").read_text(encoding="utf-8")
    hop1 = set(links_in(index_text))
    hop2 = set()
    for f in maps:
        # a map is reachable only if index links it
        rel, stem = f.relative_to(VAULT).as_posix()[:-3], f.stem
        if rel not in hop1 and stem not in hop1:
            failures.append(f"[orphan] {f.relative_to(ROOT)} not linked from vault/index.md")
        for t in links_in(f.read_text(encoding="utf-8")):
            hop2.add(t)
    reachable = hop1 | hop2
    for f in concepts + sources:
        rel, stem = f.relative_to(VAULT).as_posix()[:-3], f.stem
        if rel not in reachable and stem not in reachable:
            failures.append(f"[orphan] {f.relative_to(ROOT)} not reachable from index in 2 hops")

    # index drift: every map listed in index
    for f in maps:
        rel, stem = f.relative_to(VAULT).as_posix()[:-3], f.stem
        if rel not in hop1 and stem not in hop1:
            failures.append(f"[index-drift] map {f.stem} missing from vault/index.md")

    # required concept frontmatter
    for f in concepts:
        fm = frontmatter(f.read_text(encoding="utf-8"))
        if not re.search(r"^source:\s*\S", fm, re.M):
            failures.append(f"[missing-source] {f.relative_to(ROOT)}")
        if not re.search(r"^evidence_tier:\s*\S", fm, re.M):
            failures.append(f"[missing-evidence-tier] {f.relative_to(ROOT)}")

    # --- (b) article/prose ----------------------------------------------------
    art_md = [f for f in (sorted(ARTICLES.glob("*.md")) if ARTICLES.exists() else []) if f.name != "index.md"]
    art_ids = {f.stem for f in art_md}
    for f in art_md:
        body = strip_code_blocks(f.read_text(encoding="utf-8"))
        where = f.relative_to(ROOT)
        if body.count("**") % 2 != 0:
            failures.append(f"[unbalanced-bold] {where} — odd count of **")
        if body.count("`") % 2 != 0:
            failures.append(f"[unbalanced-code] {where} — odd count of backticks")
        if len(MALFORMED_LINK.findall(body)) != len(MALFORMED_LINK_OK.findall(body)):
            failures.append(f"[malformed-link] {where} — a '](' without closing ')'")
        # internal links resolve
        for m in INTERNAL_LINK.finditer(body):
            kind, slug = m.group(2), m.group(3)
            pool = art_ids if kind == "articles" else ids
            if slug not in pool:
                failures.append(f"[dead-internal-link] {where} -> {kind}/{slug}")
        # bare URLs (after stripping code spans and link targets)
        narrative = MD_LINK.sub("]()", CODE_SPAN.sub("", body))
        for u in BARE_URL.findall(narrative):
            failures.append(f"[bare-url] {where} — wrap as [text](url): {u}")

    # --- advisories (non-gating) ---------------------------------------------
    for f in maps:
        text = f.read_text(encoding="utf-8")
        bullets = len(re.findall(r"^\s*-\s+\d{4}-\d{2}\b", text, re.M))
        subthreads = len(re.findall(r"\*\*\(\d+\)\*\*", text))
        if bullets >= FORK_MIN_BULLETS and subthreads >= FORK_MIN_SUBTHREADS:
            advisories.append(f"[fork-due] {f.stem} ({bullets} bullets, {subthreads} sub-threads) — split candidate")

    manifest = {}
    mf = VAULT / "raw" / "manifest.json"
    if mf.exists():
        try:
            manifest = json.loads(mf.read_text() or "{}")
        except json.JSONDecodeError:
            failures.append("[bad-manifest] vault/raw/manifest.json is not valid JSON")
    for f in sources:
        fm = frontmatter(f.read_text(encoding="utf-8"))
        has_url = re.search(r"^source_url:\s*\S", fm, re.M)
        pdf_unavailable = re.search(r"^pdf_unavailable:", fm, re.M)
        if has_url and not pdf_unavailable and not (VAULT / "raw" / f"{f.stem}.pdf").exists():
            advisories.append(f"[raw-pdf-drift] {f.stem} — no raw/{f.stem}.pdf (or pdf_unavailable note)")

    # --- report ---------------------------------------------------------------
    if failures:
        print(f"✖ {len(failures)} gate failure(s):")
        for x in failures:
            print(f"  - {x}")
    else:
        print(f"✓ vault ok — {len(maps)} maps, {len(concepts)} concepts, {len(sources)} sources, {len(art_md)} articles")
    if advisories:
        print(f"↳ {len(advisories)} advisory(ies) (non-gating):")
        for x in advisories:
            print(f"  - {x}")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())


============================================================
SCRIPT: scripts/fetch-pdf.py
============================================================

#!/usr/bin/env python3
"""fetch-pdf.py — rung-1 PDF archival (the deterministic 60%).

Resolves a source note's open/free PDF and archives it to vault/raw/<slug>.pdf, then rewrites the
note's frontmatter with pdf_path/pdf_sha256/pdf_retrieved/pdf_bytes/fetched_via. Zero deps (stdlib).

This reference implementation targets **biomedical** literature via the Europe PMC REST API. The
correct free-PDF signal is the **fullTextUrlList** (an entry with availability Free/Open access +
documentStyle pdf) — NOT the `isOpenAccess` flag, which is `N` for NIH author manuscripts that
nonetheless have a free PDF. If no free PDF exists, it reports NON-OA so the caller climbs the manual
rungs in shared/pdf-archival.md (web-to-md -> playwright-browser).

Other domains: swap `resolve_free_pdf()` for the right resolver (e.g. arXiv: derive the PDF URL from
the arXiv id; OpenAlex/Unpaywall for a general DOI->OA-PDF lookup). The archive/verify/frontmatter
plumbing below is domain-agnostic.

Usage:
  python3 scripts/fetch-pdf.py                    # retry every source with pdf_unavailable
  python3 scripts/fetch-pdf.py <slug> [<slug>]    # specific source slugs
  python3 scripts/fetch-pdf.py --manual <slug> <path-to.pdf>   # import a PDF you obtained yourself

Exit: 0 if every targeted slug is archived (or already archived); 2 if any is NON-OA / needs escalation.
"""
import hashlib
import json
import re
import sys
import urllib.parse
import urllib.request
from pathlib import Path

ROOT = Path(__file__).resolve().parent.parent
SOURCES = ROOT / "vault" / "sources"
RAW = ROOT / "vault" / "raw"
MANIFEST = RAW / "manifest.json"
UA = "autoresearch-scaffold/1.0 (research archival)"
EPMC = "https://www.ebi.ac.uk/europepmc/webservices/rest/search"


def fm_block(text):
    if not text.startswith("---"):
        return "", text
    end = text.find("\n---", 3)
    if end == -1:
        return "", text
    return text[4:end], text[end + 4:]


def fm_get(fm, key):
    m = re.search(rf"^{re.escape(key)}:\s*(.+)$", fm, re.M)
    return m.group(1).strip().strip('"').strip("'") if m else None


def get(url, timeout=30):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return r.read()


def resolve_free_pdf(pmid, doi):
    """Biomedical resolver. Return (pdf_url, source_url, pmcid) via Europe PMC fullTextUrlList."""
    if pmid:
        query = f"EXT_ID:{pmid} AND SRC:MED"
    elif doi:
        query = f'DOI:"{doi}"'
    else:
        return None, None, None
    url = f"{EPMC}?query={urllib.parse.quote(query)}&resultType=core&format=json"
    data = json.loads(get(url).decode("utf-8"))
    results = data.get("resultList", {}).get("result", [])
    if not results:
        return None, None, None
    r = results[0]
    pmcid = r.get("pmcid")
    pdf_url = None
    src_url = None
    for u in r.get("fullTextUrlList", {}).get("fullTextUrl", []):
        avail = (u.get("availability") or "").lower()
        style = (u.get("documentStyle") or "").lower()
        if avail in ("free", "open access") and style == "pdf":
            pdf_url = u.get("url")
        if avail in ("free", "open access") and style == "html":
            src_url = u.get("url")
    return pdf_url, src_url or (f"https://europepmc.org/articles/{pmcid}" if pmcid else None), pmcid


def write_archived(note_path, fm, body, slug, sha, nbytes, via):
    """Rewrite frontmatter: drop pdf_unavailable, upsert pdf_* + fetched_via."""
    lines = [ln for ln in fm.splitlines()
             if not re.match(r"^(pdf_unavailable|pdf_path|pdf_sha256|pdf_retrieved|pdf_bytes|fetched_via):", ln)]
    from datetime import date
    lines += [
        f'fetched_via: "{via}"',
        f"pdf_path: raw/{slug}.pdf",
        f"pdf_sha256: {sha}",
        f"pdf_retrieved: {date.today().isoformat()}",
        f"pdf_bytes: {nbytes}",
    ]
    note_path.write_text(f"---\n" + "\n".join(lines) + "\n---" + body, encoding="utf-8")


def load_manifest():
    if MANIFEST.exists():
        try:
            return json.loads(MANIFEST.read_text() or "{}")
        except json.JSONDecodeError:
            return {}
    return {}


def manual_import(slug, path):
    """Import a locally-obtained PDF for <slug> (paywalled-but-you-have-access, or a broken OA host)."""
    note = SOURCES / f"{slug}.md"
    src = Path(path)
    if not note.exists():
        print(f"  ! no source note vault/sources/{slug}.md"); return 3
    blob = src.read_bytes()
    if not blob.startswith(b"%PDF"):
        print(f"  ! {path} is not a PDF (no %PDF header)"); return 3
    (RAW / f"{slug}.pdf").write_bytes(blob)
    sha = hashlib.sha256(blob).hexdigest()
    fm, body = fm_block(note.read_text(encoding="utf-8"))
    write_archived(note, fm, body, slug, sha, len(blob), "manual import (scripts/fetch-pdf.py --manual)")
    manifest = load_manifest()
    manifest[slug] = {"source_url": fm_get(fm, "source_url"), "sha256": sha, "bytes": len(blob),
                      "retrieved": __import__("datetime").date.today().isoformat()}
    MANIFEST.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    print(f"  ✅ {slug} — imported {len(blob):,} B from {path}")
    return 0


def main(argv):
    if argv and argv[0] == "--manual":
        if len(argv) != 3:
            print("usage: fetch-pdf.py --manual <slug> <path-to.pdf>"); return 3
        return manual_import(argv[1], argv[2])
    if argv:
        notes = [SOURCES / f"{s}.md" for s in argv]
    else:
        notes = [p for p in sorted(SOURCES.glob("*.md"))
                 if re.search(r"^pdf_unavailable:", fm_block(p.read_text(encoding="utf-8"))[0], re.M)]
    if not notes:
        print("nothing to do (no pdf_unavailable sources).")
        return 0

    manifest = load_manifest()
    escalate = []
    for note in notes:
        slug = note.stem
        if not note.exists():
            print(f"  ? {slug} — no such source note"); escalate.append(slug); continue
        fm, body = fm_block(note.read_text(encoding="utf-8"))
        pdf = RAW / f"{slug}.pdf"
        if pdf.exists() and fm_get(fm, "pdf_sha256"):
            print(f"  = {slug} — already archived, skipping"); continue
        pmid, doi = fm_get(fm, "pmid"), fm_get(fm, "doi")
        try:
            pdf_url, src_url, pmcid = resolve_free_pdf(pmid, doi)
        except Exception as e:
            print(f"  ! {slug} — resolver lookup failed ({e}); escalate"); escalate.append(slug); continue
        if not pdf_url:
            print(f"  ⛔ {slug} — no free PDF found (confirmed non-OA); escalate to browser rungs")
            escalate.append(slug); continue
        try:
            blob = get(pdf_url)
        except Exception as e:
            print(f"  ! {slug} — download failed ({e}); escalate"); escalate.append(slug); continue
        if not blob.startswith(b"%PDF"):
            print(f"  ! {slug} — not a PDF body (bot-wall/HTML?); escalate to browser rungs")
            escalate.append(slug); continue
        pdf.write_bytes(blob)
        sha = hashlib.sha256(blob).hexdigest()
        write_archived(note, fm, body, slug, sha, len(blob), "free PDF via resolver (scripts/fetch-pdf.py, rung 1)")
        manifest[slug] = {"source_url": src_url or fm_get(fm, "source_url"),
                          "sha256": sha, "bytes": len(blob),
                          "retrieved": __import__("datetime").date.today().isoformat()}
        print(f"  ✅ {slug} — archived {len(blob):,} B{('' if not pmcid else ' ('+pmcid+')')}")

    MANIFEST.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    if escalate:
        print(f"\n{len(escalate)} need the browser rungs (shared/pdf-archival.md): {', '.join(escalate)}")
        return 2
    print("\nall targeted sources archived.")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))