Fine-tune a small instruct model (Qwen2.5-3B-Instruct by default) to turn a doctor-patient dialogue into a concise, structured complaint summary for the physician — then serve it through an OpenAI-compatible API. Everything runs locally on a single consumer GPU (developed on an RTX 3080 Ti, 12 GB): you train, serve, and test without renting cloud GPUs, and patient dialogues never leave the machine at inference time.
Hugging Face Hub your GPU (local) Ollama (local)
MTS-Dialog ── ingest ──▶ data/*.jsonl ── train (QLoRA) ──▶ adapter ── export ──▶ merged
│
ollama create ◀──────────────────┘
│
client.py / your UI ──▶ OpenAI-compatible API @ localhost:11434
- An NVIDIA GPU with ~12 GB VRAM (CUDA). Built/tested on RTX 3080 Ti, driver 591.86.
- uv for environment management.
- Ollama for local serving.
- Python ≥ 3.11.
uv sync # light client deps (no torch)
uv sync --extra train # + training stack (torch CUDA, transformers, trl, peft, bitsandbytes)
copy .env.example .env # defaults already point at local OllamaThe training stack is pinned to the torch 2.6 / cu124 line (matches the
RTX 3080 Ti). transformers>=5 / trl>=1 need torch ≥ 2.7 and are deliberately
excluded. The CUDA torch source is configured in pyproject.toml
([tool.uv.index]).
For the GPU commands, activate the venv once so uv run doesn't uninstall the
training stack between calls:
.venv\Scripts\Activate.ps1 # then: ingest / train / export / monitor / stop / client# 1) Build data -> data/train.jsonl (1236) + data/eval.jsonl (65)
ingest
# 2) Fine-tune locally (QLoRA). Defaults are tuned for 3B / 12 GB.
train
# 2b) In a SECOND terminal: watch live, or stop early (progress is checkpointed)
monitor # live dashboard: step / loss / ETA / VRAM
stop # kill switch: checkpoint + halt; `train` resumes later
# 3) Merge the LoRA adapter into the base model -> out/clinical-qwen2.5-3b/merged
export
# 4) Import into Ollama (quantize on the way in)
ollama create clinical-qwen --quantize q4_K_M -f Modelfile
# 5) Test / interact (see below)
client --eval 5All optional; defaults suit a 3B / 12 GB run.
| Var | Default | Purpose |
|---|---|---|
MODEL_ID |
Qwen/Qwen2.5-3B-Instruct |
base model (Apache-2.0 alternatives below) |
OUTPUT_DIR |
out/clinical-qwen2.5-3b |
checkpoints / adapter / status |
NUM_EPOCHS |
2 |
epochs |
MAX_SEQ_LEN |
2048 |
covers 99.9 % of examples (longest is 2526 tokens) |
GRAD_ACCUM |
8 |
effective batch (batch 1 × 8) |
LR |
2e-4 |
learning rate |
SAVE_STEPS / EVAL_STEPS |
50 |
checkpoint / eval interval |
MAX_TRAIN_SAMPLES |
0 (all) |
set small for a smoke test |
LOAD_IN_4BIT |
1 |
QLoRA 4-bit (0 = bf16 LoRA) |
RESUME |
1 |
0 forces a fresh run |
train.py is built for long, unattended, resumable runs:
- Checkpoints — saved every
SAVE_STEPStoOUTPUT_DIR(keeps the last 3). Re-runningtrainauto-resumes from the latest checkpoint, so you can stop and continue at no cost. - Kill switch —
stopdrops aSTOPfile inOUTPUT_DIR; on the next step the trainer saves a checkpoint and halts cleanly (no progress lost).Ctrl+Calso saves before exit. - Monitoring —
train.pywritesOUTPUT_DIR/training_status.json(step, loss, eval_loss, %, ETA).monitorrenders it live, including VRAM fromnvidia-smi. TensorBoard logs are written too:uv run --extra train tensorboard --logdir out/.
After export you get a standalone fp16 model in out/clinical-qwen2.5-3b/merged.
Ollama imports it directly and quantizes during import — no llama.cpp needed.
The bundled Modelfile already points at it:
ollama create clinical-qwen --quantize q4_K_M -f Modelfile # import + quantize (~2 GB)
ollama run clinical-qwen "Doctor: ... Patient: ..." # quick check
uv run client # call it via the OpenAI APIFallback if your Ollama version can't import safetensors directly: clone llama.cpp,
python llama.cpp\convert_hf_to_gguf.py out\clinical-qwen2.5-3b\merged --outfile clinical-qwen.gguf --outtype q8_0, then setFROM ./clinical-qwen.ggufin theModelfileandollama create clinical-qwen -f Modelfile.
Once the model is in Ollama:
client # built-in sample dialogue
client "Doctor: ... Patient: ..." # your own dialogue
client -i # interactive: paste a dialogue, blank line to summarize
client --eval 5 # 5 held-out examples: model output vs. reference
ollama run clinical-qwen # free-form chat in OllamaFor a fuller run including the hand-crafted edge cases (negations, dosages, contradictions, …) that probe hallucination:
uv run evaluate 8 # 8 held-out + all edge_cases.jsonl -> out/eval_results.mdSee EVALUATION.md for a scored faithfulness/hallucination report on this model.
The default Qwen2.5-3B-Instruct is a good technical fit (128K context, strong
instruction-following / structured output, multilingual incl. German).
MODEL_ID to an Apache-2.0
size (one line, identical data pipeline):
Qwen/Qwen2.5-1.5B-Instruct— Apache 2.0, smaller/fasterQwen/Qwen2.5-7B-Instruct— Apache 2.0, best quality, still fits 12 GB
Pulled from the Hugging Face Hub: har1/MTS_Dialogue-Clinical_Note
(a mirror of MTS-Dialog). 1301 rows; we map dialogue → section_text.
ingest.py downloads it, deterministically holds out a 5 % eval split (the
dataset ships only a train split), and writes chat-format JSONL. It is public,
English, no token required. Add another source by adding an adapter class in
ingest.py with a .pairs() yielding (dialogue, summary).
- Local (default): the model runs on your GPU; patient dialogues never leave the machine — the strongest possible data protection.
- Cloud options (commented in
.env):together(US, public data only) orazure(EU/CH region). Only thePROVIDERswitch changes —client.pyis OpenAI-compatible, so no code changes.
src/clinical_summarizer/
ingest.py download MTS-Dialog, build train/eval JSONL
train.py QLoRA fine-tune + checkpoints/resume + kill switch + status
monitor.py live monitor (`monitor`) and kill switch (`stop`)
export.py merge LoRA adapter into the base model for Ollama
client.py OpenAI-compatible inference client + test modes
config.py provider/model settings (reads .env)
Modelfile Ollama model definition (points at the merged model)
sky/train.yaml optional cloud-GPU path (SkyPilot)
- Local-first. With a 12 GB consumer GPU on hand, training and serving run locally — far cheaper than per-hour cloud GPUs or always-on managed endpoints, and patient data stays on-device.
- Small model, QLoRA. gpt-oss-20b doesn't fit 12 GB; Qwen2.5-3B does, and 4-bit QLoRA keeps the base ~2 GB. Full runs peaked at ~6–11 GB VRAM at seq 2048 — comfortable on 12 GB.
- Pinned stack. uv first resolved transformers 5 / trl 1, which require
torch ≥ 2.7 (uses
torch.float8_e8m0fnu); cu124 tops out at torch 2.6, so the training extra is pinned to transformers 4.x / trl 0.x. - Windows fix.
pyarrowsegfaults (access violation) when imported after the ML stack;train.pyimportsdatasetsfirst to avoid it. - Data split. MTS-Dialog ships only a
trainsplit, soingest.pycarves a deterministic 5 % eval set itself. - Operability. Checkpoints + auto-resume, a file-based kill switch, and a JSON status heartbeat make long runs safe to stop, watch, and continue.
- Quality not enough at 3B? Bump
MODEL_IDtoQwen2.5-7B-Instruct(Apache 2.0, still fits 12 GB with QLoRA). - More data: add adapter classes in
ingest.py(e.g. ACI-Bench). - All data here is English; a German-language product needs German clinical data later — a separate data problem after the PoC.