oncocs runs reproducible survival analyses on public TCGA cohorts from cBioPortal DataHub, then lets a local LLM describe the results while deterministic code does every computation. An LLM proposes and writes the report prose. A claim verifier binds every number in that prose to the recorded computation, including which model each number belongs to, and a human approval gate signs off. Every LLM call is recorded and can be replayed byte-for-byte.
Two different things are evaluated here, and they have opposite outcomes.
- The software passes. The deterministic survival pipeline (Cox PH, RSF, metrics, assumption checks) verifies and replays bit-for-bit on all three cohorts. The offline test suite passes (96 tests). Every number in the results tables below comes from that code.
- Gemma's written reports mostly do not pass. After the numbers are computed, a local Gemma 3 model is asked only to describe them. Across the same-prompt comparison (gemma3:4b, 12b, 27b × LUAD, GBM, BRCA), the claim verifier rejected 8 of 9 drafts, and the one it passed (27b, GBM) was rejected by human review. No report has been approved. A red cell in the comparison table is the verifier catching the model, not a software failure.
What is established about the writing failure, from the preserved drafts:
- The failure is fabrication under fluency, not arithmetic. When handed real metrics, every model size wrote plausible numbers that do not exist in the results (e.g. 27b: Harrell C 0.688 vs computed 0.647 on LUAD; 0.728 vs 0.708 on BRCA). The 27b model was rejected on LUAD and BRCA in all three attempts even though the correct values were in its prompt.
- Scale did not fix it. 27b's single verifier pass was on GBM, where every model abstained and there were no metrics to fabricate, and its prose still misread why the checks failed.
- Smaller models added structural errors (undisclosed abstention, metrics attributed to the wrong model, invented cohort size, invented citations). These largely disappeared at 27b while numeric fabrication did not.
What is not established is the mechanism inside the model (why a transformer given the exact value 0.647 in context emits 0.688). That is a question about the model's internals, not about this pipeline, and it is the subject of the companion bioprocess-decision-runtime work, which reconstructs Gemma inference layer by layer against a pinned checkpoint. Note the gap: that work is bound to a specific Hugging Face checkpoint and runtime, while the runs here use Ollama's quantized gemma3 GGUF builds. Any mechanistic explanation must first be shown to apply to the model that produced these drafts. Until then, this repository claims only what it can show: the fabrication happens, it is reproducible, and it is caught.
The stack is LangGraph StateGraph agents, lifelines Cox PH and scikit-survival Random Survival Forest, BM25 retrieval over NCI PDQ, a FastAPI review API, and a pluggable LLM backend (Ollama gemma3 by default, any OpenAI-compatible endpoint, any LangChain chat model, recorded transcripts for replay).
Retrieval is citation-gated, not decorative. Passages carry deterministic
PDQ:<doc>#<para> tags. A draft may only cite tags the retriever
returned, quoted text must match the cited passage verbatim, and numbers
in the Context section must come from the cited passage rather than the
results table. The verifier rejects drafts that cite unlisted tags,
paraphrase quotes, or pull results-table numbers into retrieved context.
2-minute tour: What the verifier caught is the headline: documented, preserved LLM failure modes (fabricated metrics, misattributed model blocks, undisclosed abstention) caught by the verification layer. Results across three cohorts has the headline survival-model numbers, and Evidence record describes what every run persists.
flowchart LR
download --> split --> run --> results[results.json<br/>evidence record]
results --> verify
results --> cohort_agent --> context_agent[context_agent<br/>BM25 over PDQ corpus]
context_agent --> analysis_agent --> modeling_node[modeling_node<br/>hash recheck]
modeling_node --> report_agent --> claim_verifier
claim_verifier -->|fail, <3 attempts| report_agent
claim_verifier -->|pass| human_gate[human approve / reject]
claim_verifier -->|3 failures| rejected[rejected run<br/>drafts preserved]
human_gate --> report_md[report.md]
| Cohort | n | Events train/test | Checks | Best Harrell C (test) |
|---|---|---|---|---|
| LUAD | 501 | 126 / 55 | all 5 passed | 0.647 - cox/clinical |
| GBM | 580 | 335 / 143 | 2 passed; all 4 models abstained | - |
| BRCA | 1071 | 106 / 45 | all 5 passed | 0.708 - cox/clinical |
- GBM has no AJCC stage in this study (
stage: null, recorded asomitted_covariates), 49% of patients lack age/sex, and 33% are unsequenced. Every covariate exceeded the 20% missingness filter, so the run records acovariatescheck failure and all four models abstain, a legitimate result. - BRCA adds
STAGE IIIC/STAGE Xsource values.STAGE X("stage cannot be assessed") stays unmapped and is counted as missing (19 patients, 1.8%).
Results were regenerated at f61a25f after the config-hash scope fix. Metrics are byte-identical to the earlier run dirs, which are preserved with their agent runs.
LUAD in detail (all checks passed, references: stage I, sex Female):
| Model / features | Harrell C | Uno C | AUC 12m | AUC 24m | AUC 36m | IBS 6–36m |
|---|---|---|---|---|---|---|
| Cox / clinical | 0.647 | 0.640 | 0.649 | 0.691 | 0.694 | 0.153 |
| RSF / clinical | 0.639 | 0.639 | 0.604 | 0.675 | 0.695 | 0.153 |
| Cox / clinical + expression | 0.643 | 0.632 | 0.711 | 0.666 | 0.628 | 0.157 |
| RSF / clinical + expression | 0.632 | 0.615 | 0.671 | 0.672 | 0.609 | 0.153 |
Top clinical Cox hazard ratios: stage IV 3.27 (1.66–6.44, p=0.0006),
stage III 2.62 (1.75–3.93, p<0.0001), mut STK11 1.64 (1.06–2.53, p=0.027),
stage II 1.54 (1.06–2.24, p=0.025). Adding the top-50 variance expression genes
did not improve over clinical features. excluded_genes /
excluded_gene_patterns exist because sex-linked genes dominated the variance
ranking while sex is already a covariate.
agent/7b433c558b79(LUAD, gemma3:4b, human-rejected): the report'srsf/clinical_expressionblock listedcox/clinical_expression's metrics and omittedrsf/clinical. Every number was real, so the numeric-only verifier passed it. A human reviewer rejected it. This run motivated scoped verification: numbers after a model-key mention must match that model's subtree (misattributed), and model-only numbers in unscoped text fail (unscoped_model_numbers).agent/777c93997c7a(LUAD, gemma3:12b): rejected for repeatedly listingcox/clinical_expressionmetrics (0.157, 0.711, 0.666) undercox/clinical.agent/ce75cc3fd75a(GBM, gemma3:4b): rejected because all four models abstained but the draft never disclosed the abstention.agent/4afbc74b7bc5(BRCA, gemma3:12b), verifier false positive, fixed: rejected for writing the IBS window label "6-36 months" in unscoped text. The6matched onlymodels.*values. The verifier now derives integers embedded in metric key names (e.g.integrated_brier_6_36m,auc_24m) and treats them as labels. The follow-up run052fa2c286dapassed in 3 attempts.agent/ea53fcd852c2andagent/5d5a01a0e471(GBM, BRCA, gemma3:4b): rejected for invented citation tags ([PDQ:doc#0]was not among the retrieved passages) and, on GBM, failure to disclose the all-model abstention.agent/4918eaf618e7(LUAD, gemma3:4b): rejected for describing a concordance index as a calibration metric, then cited model-only numbers with no model attribution.agent/a911996a2e14(LUAD, gemma3:12b): rejected for fabricating a cohort size of 602 for a 501-patient cohort on every attempt, in addition to persistent metric misattribution.agent/a706c3f76993(GBM, gemma3:4b): rejected after the first attempt was flagged for not disclosing the all-model abstention, attempts 2 and 3 invented metric values (harrell_c = 0.45, etc.) for models that had abstained.agent/318d61643606andagent/39189caa1502(LUAD, BRCA, gemma3:27b): rejected because both drafts produced tidy metric tables whose values (harrell_c 0.688 / 0.728, uno_c 0.162 / 0.118) do not exist in the results. Every attempt was flaggedunverified_numbers.
agent/ba8da12d0f19(GBM, gemma3:27b): verifier passed (1 attempt). Every number was verified and the abstention was disclosed, but the Limitations section read theproportional_hazardsandconvergencecheck failures as evidence the PH assumption may not hold and fitting was unstable. The recorded details show both were downstream of the covariates check. Human-rejected.
results/agent_model_comparison.json is regenerated by
oncocs agent summarize --replay. The table compares only runs recorded under
the current prompts (all replay PASS, meaning replay proves the recorded prompts and
responses reproduce the same report and verdict under the current code):
| Cohort | gemma3:4b | gemma3:12b | gemma3:27b |
|---|---|---|---|
| LUAD | rejected after 3 for unscoped model numbers, C-index mislabeled as calibration, unverified "significant" (98d9952a9150) |
rejected after 3 for a fabricated cohort size (602 for a 501-patient cohort) and persistent misattribution (a911996a2e14) |
rejected after 3 for invented metric values in every attempt (318d61643606) |
| GBM | rejected after 3 for undisclosed abstention, then fabricated metric values for the abstained models (a706c3f76993) |
rejected after 3 because abstention was never disclosed (c654c99fa00f) |
verifier passed (1 attempt); human-rejected for misreading check details (ba8da12d0f19) |
| BRCA | rejected after 3 for "robust" and unverified "significant" (dd95bfcd2b8b) |
rejected after 3 for persistent misattribution (14dee8f5d071) |
rejected after 3 for invented metric values in every attempt (39189caa1502) |
Under identical prompts, only gemma3:27b produced a verified draft, on GBM (the all-abstained cohort) in a single attempt. The LUAD and BRCA 27b runs were rejected for inventing metric values not present in the results table (e.g. harrell_c 0.688 / 0.728 vs the computed 0.647 / 0.708). Across 20 recorded runs, the verifier passed 1 of 9 same-prompt drafts (gemma3:27b, GBM) and human review rejected it. No report has been approved.
FROZEN runs predate a prompt change and are not re-verified:
- LUAD
76a00fdba97d(4b): passed the numeric verifier. It shares the misattribution defect but predates thehuman_reviewfield, so its recorded status staysdraft_pending_approval. - LUAD
7b433c558b79(4b): same semantic defect, first recorded instance. - LUAD
4918eaf618e7(4b): rejected for mislabeling the C-index as calibration, then unscoped model numbers. - LUAD
777c93997c7a(12b): rejected for persistent misattribution. - GBM
ce75cc3fd75a(4b): rejected because the abstention was not disclosed. - GBM
ea53fcd852c2(4b): rejected for invented citation tags and undisclosed abstention. - GBM
0fb3e28f52d6(12b): passed verifier in 2 attempts, unapproved. - BRCA
b81f24db3970(4b): passed verifier, unapproved. - BRCA
5d5a01a0e471(4b): rejected for an invented citation tag and a C-index mislabeled as calibration. - BRCA
4afbc74b7bc5(12b): rejected for the "6-36" label false positive. - BRCA
052fa2c286da(12b): passed verifier in 3 attempts, unapproved.
The endpoint backend sends generation to an OpenAI-compatible service
(oncocs/llm/endpoint.py) instead of local Ollama. Four LUAD runs
surfaced failure modes the same-prompt gemma runs never hit.
agent/a8a460e3431b(Qwen2.5-7B): every draft was cut mid-sentence at the service's default 256-token generation bound, andmissing_sectionsflagged the absent tail sections on all 3 attempts. Truncation, not writing, was the defect. Fixed by sendingmax_tokensper request.agent/535bfb3be89f(Qwen2.5-7B, bound raised): drafts tripled in length but spent the budget enumerating per-covariate hazard ratios. Invented p-values were flaggedunverified_numbers(p=0.5502where the recorded value is 0.5602) and CI bounds landed in the wrong model blocks (misattributed×12). Still truncated before the required sections.agent/1c01ac53d02f(Qwen2.5-3B, grounding prompt): complete drafts, all sections, zero numeric violations. Rejected on 4 misquotes (paraphrased passage text in quotation marks) and onerobust. Verbatim quoting exceeded small-model fidelity, so the prompt now requires paraphrase-only citations.agent/1abfb93c0599(Qwen2.5-3B): verifier passed on attempt 1, the first verified draft through the endpoint backend and replays byte-for-byte against its recorded transcript (PASS). Pending human review.
A verifier tightening then showed that pass was thin: 1abfb93c0599
named every focus model but reported no values. A focus_without_values
check now requires each in-scope focus model to carry at least one of
its own recorded metrics, and the old run replays FROZEN. Six more
endpoint runs under the tightened verifier mapped three further
small-model failure modes, each fixed in the prompt or the plan
hand-off rather than in the verifier.
- The 3B pasted the retry findings verbatim into its next draft. The
retry prompt now wraps the rejected draft in
<rejected_draft>tags and forbids copying checker phrasing. - The analysis agent wrote sentence claims carrying invented metrics
("Harrell C-index of 0.75"), and the report model quoted them. The
analysis prompt now restricts
claims_to_maketo metric names, andgraph.pymasks digits in that field before it reaches the report prompt (_plan_view). - GBM abstained all four models after failed checks, yet the model wrote invented metrics into the new per-model bullets while also writing "abstained" elsewhere. The bullet instruction now defines the abstention form explicitly, and the prompt's example carries no numbers (an earlier numeric example was copied verbatim into a draft).
Three endpoint runs passed against the same 3B service:
106b1edbf7b5 (LUAD, attempt 3), 1171d5d935a4 (BRCA, attempt 1),
254fd283defd (GBM, attempt 2). Those runs now replay FROZEN under
the tightened prompt and verifier. Live reruns surfaced further
small-model failure modes.
16448fdcde62(BRCA) passed while listingrsf/clinical_expressionas abstained beside its real values. Afalse_abstentioncheck now rejects valued models described as abstained, and that run replays FROZEN.- Retries regenerated token-identical drafts. Attempt 1 runs at temperature 0 and the inference service seeds sampling deterministically, so identical findings produced identical outputs. Retries now sample at nonzero temperature with an attempt-specific marker.
- LUAD's all-PASS cohort produced a fabricated abstention list in "Checks and abstentions": the section header itself invited one. The checks instruction now names abstention only when the results contain a FAIL.
robustsurvived a flat word ban asrobustness, so the verifier now matches the word family. Zero-missingness entries were dropped from the rendered table because "absence of missing data" reliably drew evaluative claims about data completeness.- Passage listings were rendered in quotation marks, which taught the model to dump verbatim quotes into Limitations. Passages now render unquoted, and the checks table renders as copyable sentences.
The final sweep reran all three cohorts under identical code at two model scales, and every run passed on attempt 1.
- Qwen2.5-3B:
77fe3a4ac23e(LUAD),dc07cf8dab4b(BRCA),c85e0b06131c(GBM, all four models abstained with reasons). - Qwen2.5-14B:
a080438819f2(LUAD),eea6fa8ba841(BRCA),401e3254cc64(GBM, where the analysis plan scoped two focus models and the draft disclosed both abstentions).
Each of those six replays PASS against its recorded transcript under
the current code. results/agent_model_comparison.json carries 68
runs: 6 replay PASS and 62 replay FROZEN, where FROZEN means a run's
recorded prompts no longer match current code and its verdict cannot
be re-claimed as current behavior.
pip install -e .[dev]
python -m oncocs download --cohort luad # archive or per-file fallback
python -m oncocs split --cohort luad --seed 20240601
python -m oncocs run --cohort luad --seed 20240601
python -m oncocs verify results/luad/<run_id>/results.json
python -m oncocs rag fetch # NCI PDQ corpus (~92 KB)
python -m oncocs agent run --cohort luad --results results/luad/<run_id>/results.json \
--backend ollama --model gemma3:4b --seed 20240601
python -m oncocs agent replay results/luad/<run_id>/agent/<agent_id>/agent_run.json
python -m oncocs approve <agent_run.json> --by "<name>" # or: reject --reason "..."
python -m oncocs agent summarize # results/agent_model_comparison.json
python -m oncocs rag compare --queries "q1;q2" --modes bm25,tfidf
# -> results/rag_mode_comparison.json. Retrieval mode is a per-cohort
# config key (rag_mode: bm25|tfidf|embed); "embed" needs vector-db-mcp[embed].
python -m oncocs serve --port 8000 # review API on 127.0.0.1Cohorts are config-driven (cohorts/<id>.yaml). Adding a cohort is a yaml file,
not code. GBM and BRCA were added yaml-only. The only oncocs/ changes between
the phase-3 start and end were generic fixes any cohort could trigger (78 lines
across 3 files: LFS-pointer detection in the download fallback, replay
comparison for rejected reports, and clean abstention when no covariates
survive missingness).
Acquisition order (S3 archive → per-file DataHub → raw GitHub fallback →
cBioPortal API for mutations), SHA-256 manifesting, LFS-pointer detection,
harmonization, missingness filtering and split freezing are documented in
docs/DATA_PIPELINE.md. oncocs qc --cohort X audits
already-downloaded raw data and writes results/<cohort>/<run>/qc.json:
$ python -m oncocs qc --cohort luad # abridged
QC luad: wrote results/luad/549df572c5ab/qc.json
schema clinical_patient: OK
schema clinical_sample: OK
schema mutations: OK
schema expression: OK
missingness clinical_patient: 34 columns with missing values
duplicates: {'clinical_patient_patient_id': 0, 'clinical_sample_sample_id': 0}
unsequenced primary samples: 0
split integrity: True {'hash_match': True, 'overlap_count': 0, ...}
Loaders validate every frame against pandera schemas (oncocs/schemas.py).
A malformed frame raises SchemaError instead of being silently coerced.
docker build -t oncocs .
docker run -v "$PWD/results:/app/results" -v "$PWD/data:/app/data" \
-v "$PWD/splits:/app/splits" -v "$PWD/rag:/app/rag" -p 8000:8000 oncocsThe image builds the React UI (web/, served at /ui) and installs the
deploy extra. The LLM backend is not in the image. The model lives
behind ONCOCS_LLM_BASE_URL, any OpenAI-compatible endpoint (a Colab
vLLM/Ollama tunnel, llama.cpp, or a hosted API). docker compose up
brings up the API with a Postgres mirror for run records. See
docs/DEPLOY.md.
Every run writes results/<cohort>/<run_id>/results.json containing git
commit/dirty state, data manifest hash, split hash, config hash, package
versions, missingness/drop counts, omitted covariates, check outcomes, and
per-model metrics (or abstention reasons). oncocs verify recomputes the
hashes and reruns the pipeline. Agent runs write
results/<cohort>/<run_id>/agent/<agent_run_id>/{agent_run.json,report.md}, which hold
the full prompt transcript (replayable via RecordedBackend), every draft with
its verification, the analysis plan, status and human-review metadata.
Upstream data drifts, and verify says so. cBioPortal retired the public
tarballs, and the mutations member now arrives through a live API whose
output shifts when upstream gene symbols are migrated. Re-downloading
today produces a mutations file that hashes differently than the one the
committed results consumed, so verify fails data_manifest_sha256 for
the recorded runs. That is the gate working: the manifest documents the
original inputs byte-for-byte, and a changed upstream is reported as
changed rather than silently absorbed.
GET /cohorts, GET /runs, GET /runs/{cohort}/{run_id},
GET /runs/{cohort}/{run_id}/agent, GET /agent/{cohort}/{run_id}/{agent_id}/report
and .../agent_run, plus POST .../approve / POST .../reject
({by, note|reason}; conflicts return 409).
POST /runs/{cohort}/{run_id}/agent triggers an agent run in the
background ({backend, seed}: endpoint reads ONCOCS_LLM_* env,
scripted-demo needs no model) and returns a job_id for
GET /jobs/{job_id}. When DATABASE_URL is set, agent-run records and
jobs mirror into Postgres and the read endpoints serve from there. The
results/ filesystem remains canonical. The pipeline itself is not
triggered through the API. Runs re-draft reports over saved
results.json files.
- Research/education only. Not validated for clinical, diagnostic, prognostic, or treatment decisions.
- Public retrospective TCGA data. Results reflect the dataset, not any clinical claim.
- The verifier checks number provenance and model attribution, forbidden phrasing, section structure, abstention disclosure and citation integrity. It does not check the scientific correctness of prose. That is what the human gate is for.
- Local small-parameter models are used for drafting. Failure modes observed include misattributed model blocks and missing abstention disclosure.
- Data: cBioPortal DataHub public TCGA PanCancer Atlas 2018 studies (LUAD, GBM, BRCA). Downloaded per study. Manifest records per-file SHA-256.
- Context corpus: NCI PDQ summaries (U.S. public domain). Suggested citation: National Cancer Institute. PDQ(R) Cancer Information Summary. Bethesda, MD: National Cancer Institute. Retrieved from cancer.gov.
- Code: MIT (see LICENSE).
cohorts/<id>.yaml cohort config (study, columns, covariates, rag_docs)
data/<cohort>/{manifest.json,raw/} raw files + sha256 manifest (raw is gitignored)
splits/<cohort>.json frozen stratified split
results/<cohort>/<run_id>/results.json evidence record
results/<cohort>/<run_id>/agent/<id>/ agent_run.json + report.md
rag/corpus/ PDQ text + manifest (license, citation, sha256)
oncocs/ package: data, models, checks, evidence,
agents (graph/verifier/replay/approve),
rag (fetch/retrieve), api (FastAPI review)
tests/ offline tests (synthetic fixtures)
- bioprocess-decision-runtime applies the same claims-bound-to-computation discipline one layer down: bit-exact replay certificates instead of a report verifier.
- llm-posttraining studies the abstention-vs-fabrication trade-off that this repo's verifier enforces procedurally.
- vector-db-mcp supplies the optional
rag_mode: embedpath: itsvdb_mcp.embedE5 embedder replaces BM25 scoring when installed. BM25 stays the default.results/rag_mode_comparison.jsonrecords the per-query agreement.