CS421 — Counterfactual Fact Verification

Zero-shot counterfactual fact verification on the FEVER dataset. Claims are classified as SUPPORTS, REFUTES, or NOT ENOUGH INFO. The pipeline cherry-picks claims across structural complexity tiers, generates counterfactual variants, and measures robustness of a local LLM (Phi-3 Mini 4K, 4-bit quantized).

---

Setup

Requirements: Python 3.11, conda

conda activate cs421
pip install -r requirements.txt

Dataset paths — the datasets live outside the repo. Set env vars for your machine:

cp .env.example .env
# Edit .env — set CS421_WIKI_DIR and CS421_TRAIN_FILE to your local paths
source .env

Variable	What it points to
CS421_WIKI_DIR	Directory containing wiki-001.jsonl … wiki-109.jsonl
CS421_TRAIN_FILE	FEVER train.jsonl (~33 MB, ~145k claims)

Scripts fall back to /datasets/cs421/... if env vars are not set.

---

Pipeline

all_supports.json and tagged_supports.json are committed to the repo — steps 01–03 do not need to be re-run on a new machine. Start from step 04.

Step 01 — Extract SUPPORTS claims

python3 scripts/01_extract_supports.py
# Output: data/processed/all_supports.json (80,035 claims)

Step 02 — Analyse dataset

python3 scripts/02_analyze_supports.py
# Output: data/processed/supports_analysis.json

Step 03 — Tag structural complexity tiers

python3 scripts/03_tag_difficulty.py
# Output: data/processed/tagged_supports.json
# Tags every claim as LC / MC / HC based on token count, evidence sets, wiki pages
# ~2 min, no GPU

Step 04 — Analyse model accuracy across tiers

Runs a stratified sample of claims through the LLM in two modes (zero-shot and with evidence) to characterise model behaviour across LC / MC / HC tiers.

GPU required. Dry-run first to verify your paths before committing time:

python3 scripts/04_analyze_tiers.py --model phi3 --dry-run

Common runs:

# Standard — 300 claims per tier (~14 min on RTX 4060)
python3 scripts/04_analyze_tiers.py --model phi3

# More HC claims (sampling bias check)
python3 scripts/04_analyze_tiers.py --model phi3 --n-hc 500

# All HC claims — definitive run (~25 min with batching)
python3 scripts/04_analyze_tiers.py --model phi3 --n-hc 0

# Different seed — independent draw, separate output file
python3 scripts/04_analyze_tiers.py --model phi3 --seed 123

# Mistral 7B instead of Phi-3
python3 scripts/04_analyze_tiers.py --model mistral

# Custom path if wiki-pages is not in the default location
python3 scripts/04_analyze_tiers.py --model phi3 --wiki-dir /path/to/wiki-pages/

Output files go to docs/tier_analysis/ with the format: tier_analysis_{model}_LC{n}_MC{n}_HC{n}_{date}.json Files are never overwritten — re-runs with the same config get a _v2 suffix.

GPU memory: Phi-3 Mini ~2.5 GB VRAM, Mistral 7B ~5 GB VRAM. Both use 4-bit NF4 quantization. Use --batch-size 16 on higher-VRAM GPUs for more speed; drop to --batch-size 4 if you hit OOM.

Step 05 — Cherry-pick claims (not yet implemented)

python3 scripts/05_cherry_pick.py
# Target: 500 claims across LC / MC / HC tiers
# Output: data/processed/cherry_picked_{lc,mc,hc}.json

Steps 06–09 (not yet implemented)

python3 scripts/06_retrieve_evidence.py
python3 scripts/07_generate_counterfactuals.py
python3 scripts/08_run_inference.py --model phi3
python3 scripts/09_analyze_results.py

---

Structural Complexity Tiers

Claims are tagged using three structural sub-scores (each 1–3), averaged:

Sub-score	1	2	3
Token count	≤ 7	≤ 12	> 12
Evidence sets	≤ 1	≤ 3	> 3
Distinct wiki pages	≤ 1	≤ 2	> 2

avg ≤ 1.3 → LC, avg ≤ 2.0 → MC, avg > 2.0 → HC

Distribution across 80,035 SUPPORTS claims: LC 29.2% / MC 66.6% / HC 4.2%

Note: These tiers measure structural complexity of the evidence chain, not model difficulty. Tier analysis (script 04) showed that HC claims are actually easier for the model zero-shot — they tend to involve well-documented entities with strong training signal. The interesting finding is the ~30-point gap between zero-shot (~66%) and with-evidence (~96%) accuracy across all tiers.

---

Data Layout

data/processed/
  all_supports.json          80,035 SUPPORTS claims (committed)
  supports_analysis.json     category + complexity breakdown (committed)
  tagged_supports.json       all claims with tier/category metadata (committed)
  cherry_picked_{lc,mc,hc}.json   created by 05_cherry_pick.py

docs/
  tier_analysis/             output JSON from 04_analyze_tiers.py runs
  Data Analysis/             reference tables and brainstorming notes
  CF Generation/
  Reports/

scripts/
  01_extract_supports.py
  02_analyze_supports.py
  03_tag_difficulty.py
  04_analyze_tiers.py
  05_cherry_pick.py          (next step)
  06–09_*.py                 (not yet implemented)
  midterm/                   archived v1 pipeline (30-claim proof of concept)