A reproducible computational-toxicology pipeline predicts Tox21 NR-ER (estrogen receptor agonism, an endocrine-disruption endpoint) from chemical structure on public data.
Status: working baseline. End-to-end Snakemake DAG runs download → standardize → features → scaffold split → train → evaluate on real Tox21 data. Four models are compared on identical splits: logistic regression and random forest on Morgan fingerprints plus a GIN graph network and a TensorFlow/Keras MLP, all Platt-calibrated on validation scaffolds. The evaluation rigor is the point.
New Approach Methodologies (NAMs) aim to reduce reliance on animal testing by predicting toxicity from in-vitro assays and chemical structure. This project builds an end-to-end reproducible ML pipeline for a single high-value toxicity endpoint, with the evaluation rigor that regulatory-adjacent use demands. Scaffold-split evaluation comes with applicability-domain analysis and calibrated uncertainty rather than point predictions.
Tox21 NR-ER (nuclear-receptor estrogen agonism). Chosen because endocrine
disruption is a flagship NAMs use case (in-vitro/in-silico estrogenicity is
actively argued to displace animal uterotrophic assays), labels are fully
public, and the ~13% active rate (9.0% in the held-out test partition,
per-split counts committed in results/metrics.json) exercises the
imbalance-aware evaluation.
Alternatives considered in docs/endpoint-selection.md.
| Source | Use | Access |
|---|---|---|
| Tox21 (MoleculeNet mirror) | NR-ER endpoint on 6091 labeled compounds | public CSV download |
Additional sources are cataloged in docs/datasources.md for endpoint
expansion (ToxCast/invitrodb, PubChem, ChEMBL).
pip install -e .[dev] # needs Python >= 3.10; uv recommended
snakemake --cores 4 # runs the full DAG (stubbed)
pytest tests/
snakemake -n # dry-run DAG check
# or: make install / test / dag / rundownload → standardize → features → scaffold split → train → evaluate → report
Serving adds two derived stages, serve_reference and model_card.
See workflow/Snakefile.
src/comp_tox/serve/ is the runtime layer. serve_reference freezes the
training distribution (train fingerprints, bit frequencies, conformal
qhat, AD threshold, eval-set baseline statistics) into
results/serve_reference.npz, so the service never re-reads pipeline
intermediates.
pip install -e ".[serve]"
uvicorn comp_tox.serve.app:app # model/reference paths come from config.servePOST /predicttakes{"smiles": [...]}and returns per-compound calibrated probability with its 90% conformal set. Each record also carries nearest-neighbor Tanimoto distance and an in-domain flag. Unparseable SMILES come back as error records, not crashes.POST /drifttakes the same payload and returns a batch drift report covering nn-distance against the eval baseline and in-domain fraction, plus bit-frequency Jensen-Shannon divergence and predicted-positive-rate shift. Adrift_warningstatus means the measured metrics no longer apply to that batch.GET /health,GET /model_card.- CLI equivalent for a scored batch file:
python -m comp_tox.serve.drift batch.parquet model.joblib serve_reference.npz drift_report.json
The JS-divergence baseline is the eval set's own divergence from train,
so a batch only warns when it exceeds what an on-distribution batch
already scores. A live check reports ok on the held-out test set. A
30-compound subsample flags on JS alone, because 30 molecules cannot
reproduce 2048 bit frequencies, and that noise floor is the signal.
docs/model-card.md is generated from results/metrics*.json and the
bundle by comp_tox.serve.modelcard. The card cannot state numbers the
evaluation did not measure. The committed card describes the NR-AR
artifact, the last trained bundle on disk. Serving NR-ER is a config +
snakemake rerun away (its split/features artifacts were overwritten by
the second-endpoint run).
docs/nam_dossier*.md are generated by comp_tox.dossier as evidence
documents for New Approach Methodology review. Provenance carries the
artifact sha256s, and the scaffold-split protocol precedes the model
table. Conformal coverage and applicability-domain metrics get their own
sections, and a limitations block is emitted from the artifact's own
status fields. The
claims verifier (src/comp_tox/claims.py, vendored byte-identical with
the statgen and scrna_qc copies plus llm-posttraining's) binds every
numeric token in the document to a recorded metric or config leaf.
snakemake -n covers the dossier rule, and --verify re-binds any
committed copy.
Scaffold-split evaluation, where no scaffold is shared between partitions, is the estimate of prospective performance on new chemotypes. 610 test compounds, 55 actives (9.0%). The NR-ER point estimates below are retained from the original run. Its confidence intervals are withdrawn. The previous bootstrap collapsed repeated scaffold draws, and the original checkpoint or predictions are unavailable for recomputation. Corrected code preserves draw multiplicity.
| Model | AUROC | AUPRC | ECE | Conformal @90% | In-domain AUROC | Out-domain AUROC |
|---|---|---|---|---|---|---|
| Logistic regression (fingerprints) | 0.644 | 0.244 | 0.033 | 0.930 | 0.751 | 0.619 |
| Random forest (fingerprints) | 0.726 | 0.274 | 0.044 | 0.931 | 0.701 | 0.733 |
| GIN graph network | 0.666 | 0.249 | 0.040 | 0.934 | 0.507 | 0.704 |
| Keras MLP (fingerprints) | 0.679 | 0.314 | 0.053 | 0.928 | 0.823 | 0.648 |
Two findings deserve plain statements.
- For logistic regression the applicability domain separates performance. The model is materially better in-domain (0.751 vs 0.619), which is why AD flagging is reported instead of one pooled number.
- For random forest and the GNN it inverts (RF 0.701 in-domain against 0.733 out, GNN 0.507 in against 0.704 out). The Tanimoto nearest-neighbor domain does not discriminate their performance. Applicability domains are model-dependent, not a property of the dataset alone. Any AD claim here is conditioned on the model it was measured with.
Same pipeline, same protocol, with endpoint.assay_id switched to NR-AR
(7,118 compounds after standardization, 2.4% actives in test). Metrics
land in results/metrics_NR-AR.json.
NR-AR confidence intervals were corrected using the retained checkpoint after
reproducing every model's published AUROC and AUPRC. The compact held-out
prediction capture, results/metrics_NR-AR_predictions.parquet, is hash-bound
in the metrics file. A regression test replays the corrected intervals without
loading a model or downloading data.
| Model | AUROC | In-domain AUROC | Out-domain AUROC |
|---|---|---|---|
| Logistic regression | 0.728 | 0.920 | 0.702 |
| Random forest | 0.830 | 0.908 | 0.816 |
| GIN graph network | 0.831 | 0.816 | 0.834 |
| Keras MLP | 0.344 | 0.225 | 0.368 |
On NR-AR the AD direction holds for logreg and RF but inverts for the GNN (0.816 in-domain vs 0.834 out). The MLP's in/out numbers are part of its collapse, not evidence either way. NR-AR has 4.8% train prevalence (272 actives vs 659 on NR-ER) and the MLP memorizes. Train AUROC reaches 1.000 within 10 epochs while validation degrades below chance (training curves from the run log, per-epoch history is not committed). Early stopping on val AUROC selects 0.641-valid. The held-out point estimate is below chance (0.344), but the corrected 95% interval, 0.17 to 0.54, includes 0.5. The sklearn heads and GIN degrade gracefully. The lightly-regularized MLP does not. That is a model×endpoint interaction, and a reminder that "ran fine on the other endpoint" is not evidence of robustness.
Overall AUROC is modest, and it is the scaffold-split result. Random-split numbers for this endpoint are typically ~0.8+ and misleading.
Artifacts: results/metrics.json, results/calibration.png
(reliability curve over test scaffolds).
src/comp_tox/models/gnn_jax.pyreimplements the GIN in JAX/Flax and verifies it:parity_checkmaps the trained PyTorch weights into the Flax parameter tree and asserts identical logits (<1e-4) on the same graphs. Cross-framework agreement is the evidence the port is correct, not a reimplementation that merely runs.src/comp_tox/models/gnn_ddp.py+ddp_main.pytrain the same GIN undertorch.distributedDDP on the gloo backend, with real spawned worker processes, partitioned data, gradient all-reduce, rank-0 evaluation. The code path is identical to multi-GPU (nccl). Only the backend and scale differ. Verified on CPU. Multi-GPU scaling is untested and labeled as such.src/comp_tox/models/mlp_tf.pyis a TensorFlow/Keras MLP head on the same fingerprints and splits, wrapped in a picklable sklearn adapter so it flows through the identical Platt-calibrated conformal and applicability-domain path. PyTorch, JAX and TF are all exercised on this one task: same data, same eval, three frameworks.
- Predictions are research predictions. No GxP or safety claims are made or implied.
- Baseline model only. The contribution is the evaluation scaffold, not an accuracy record. Next step: RF/GP/GNN comparison on the same splits.
- Applicability domain covers only ~18% of the test set at the current threshold. Most of chemical space is flagged, by design.
- Assay labels are noisy and class-imbalanced. See
docs/evaluation.md.
config/ endpoint + model + evaluation config
workflow/ Snakemake DAG
src/comp_tox/
data/ download + standardization
features/ fingerprints + descriptors
models/ training
eval/ scaffold split, calibration, conformal, applicability domain
tests/
docs/ endpoint selection, data sources, evaluation notes
- protein-stability-uncertainty uses the same conformal machinery on a regression problem. Both repos report where marginal coverage fails alongside the headline number.