Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

llm-training-medical-fine-tuning

Fine-tune a small instruct model (Qwen2.5-3B-Instruct by default) to turn a doctor-patient dialogue into a concise, structured complaint summary for the physician — then serve it through an OpenAI-compatible API. Everything runs locally on a single consumer GPU (developed on an RTX 3080 Ti, 12 GB): you train, serve, and test without renting cloud GPUs, and patient dialogues never leave the machine at inference time.

Hugging Face Hub                 your GPU (local)                 Ollama (local)
MTS-Dialog  ── ingest ──▶ data/*.jsonl ── train (QLoRA) ──▶ adapter ── export ──▶ merged
                                                                                    │
                                                  ollama create  ◀──────────────────┘
                                                        │
                          client.py / your UI ──▶ OpenAI-compatible API @ localhost:11434

Requirements

  • An NVIDIA GPU with ~12 GB VRAM (CUDA). Built/tested on RTX 3080 Ti, driver 591.86.
  • uv for environment management.
  • Ollama for local serving.
  • Python ≥ 3.11.

Install

uv sync                       # light client deps (no torch)
uv sync --extra train         # + training stack (torch CUDA, transformers, trl, peft, bitsandbytes)
copy .env.example .env        # defaults already point at local Ollama

The training stack is pinned to the torch 2.6 / cu124 line (matches the RTX 3080 Ti). transformers>=5 / trl>=1 need torch ≥ 2.7 and are deliberately excluded. The CUDA torch source is configured in pyproject.toml ([tool.uv.index]).

For the GPU commands, activate the venv once so uv run doesn't uninstall the training stack between calls:

.venv\Scripts\Activate.ps1    # then: ingest / train / export / monitor / stop / client

Workflow

# 1) Build data  -> data/train.jsonl (1236) + data/eval.jsonl (65)
ingest

# 2) Fine-tune locally (QLoRA). Defaults are tuned for 3B / 12 GB.
train

# 2b) In a SECOND terminal: watch live, or stop early (progress is checkpointed)
monitor                       # live dashboard: step / loss / ETA / VRAM
stop                          # kill switch: checkpoint + halt; `train` resumes later

# 3) Merge the LoRA adapter into the base model -> out/clinical-qwen2.5-3b/merged
export

# 4) Import into Ollama (quantize on the way in)
ollama create clinical-qwen --quantize q4_K_M -f Modelfile

# 5) Test / interact (see below)
client --eval 5

Configuration (env vars)

All optional; defaults suit a 3B / 12 GB run.

Var Default Purpose
MODEL_ID Qwen/Qwen2.5-3B-Instruct base model (Apache-2.0 alternatives below)
OUTPUT_DIR out/clinical-qwen2.5-3b checkpoints / adapter / status
NUM_EPOCHS 2 epochs
MAX_SEQ_LEN 2048 covers 99.9 % of examples (longest is 2526 tokens)
GRAD_ACCUM 8 effective batch (batch 1 × 8)
LR 2e-4 learning rate
SAVE_STEPS / EVAL_STEPS 50 checkpoint / eval interval
MAX_TRAIN_SAMPLES 0 (all) set small for a smoke test
LOAD_IN_4BIT 1 QLoRA 4-bit (0 = bf16 LoRA)
RESUME 1 0 forces a fresh run

Monitoring, kill switch & checkpoints

train.py is built for long, unattended, resumable runs:

  • Checkpoints — saved every SAVE_STEPS to OUTPUT_DIR (keeps the last 3). Re-running train auto-resumes from the latest checkpoint, so you can stop and continue at no cost.
  • Kill switch — stop drops a STOP file in OUTPUT_DIR; on the next step the trainer saves a checkpoint and halts cleanly (no progress lost). Ctrl+C also saves before exit.
  • Monitoring — train.py writes OUTPUT_DIR/training_status.json (step, loss, eval_loss, %, ETA). monitor renders it live, including VRAM from nvidia-smi. TensorBoard logs are written too: uv run --extra train tensorboard --logdir out/.

Serving (Ollama)

After export you get a standalone fp16 model in out/clinical-qwen2.5-3b/merged. Ollama imports it directly and quantizes during import — no llama.cpp needed. The bundled Modelfile already points at it:

ollama create clinical-qwen --quantize q4_K_M -f Modelfile   # import + quantize (~2 GB)
ollama run clinical-qwen "Doctor: ... Patient: ..."          # quick check
uv run client                                                # call it via the OpenAI API

Fallback if your Ollama version can't import safetensors directly: clone llama.cpp, python llama.cpp\convert_hf_to_gguf.py out\clinical-qwen2.5-3b\merged --outfile clinical-qwen.gguf --outtype q8_0, then set FROM ./clinical-qwen.gguf in the Modelfile and ollama create clinical-qwen -f Modelfile.

Testing / interacting

Once the model is in Ollama:

client                        # built-in sample dialogue
client "Doctor: ... Patient: ..."   # your own dialogue
client -i                     # interactive: paste a dialogue, blank line to summarize
client --eval 5               # 5 held-out examples: model output vs. reference
ollama run clinical-qwen      # free-form chat in Ollama

For a fuller run including the hand-crafted edge cases (negations, dosages, contradictions, …) that probe hallucination:

uv run evaluate 8             # 8 held-out + all edge_cases.jsonl -> out/eval_results.md

See EVALUATION.md for a scored faithfulness/hallucination report on this model.

Model & license

The default Qwen2.5-3B-Instruct is a good technical fit (128K context, strong instruction-following / structured output, multilingual incl. German). ⚠️ But the 3B size is under the "Qwen Research License" (non-commercial). Fine for a learning PoC; for a commercial product switch MODEL_ID to an Apache-2.0 size (one line, identical data pipeline):

  • Qwen/Qwen2.5-1.5B-Instruct — Apache 2.0, smaller/faster
  • Qwen/Qwen2.5-7B-Instruct — Apache 2.0, best quality, still fits 12 GB

Training data

Pulled from the Hugging Face Hub: har1/MTS_Dialogue-Clinical_Note (a mirror of MTS-Dialog). 1301 rows; we map dialogue → section_text. ingest.py downloads it, deterministically holds out a 5 % eval split (the dataset ships only a train split), and writes chat-format JSONL. It is public, English, no token required. Add another source by adding an adapter class in ingest.py with a .pairs() yielding (dialogue, summary).

Data protection

  • Local (default): the model runs on your GPU; patient dialogues never leave the machine — the strongest possible data protection.
  • Cloud options (commented in .env): together (US, public data only) or azure (EU/CH region). Only the PROVIDER switch changes — client.py is OpenAI-compatible, so no code changes.

Repository layout

src/clinical_summarizer/
  ingest.py    download MTS-Dialog, build train/eval JSONL
  train.py     QLoRA fine-tune + checkpoints/resume + kill switch + status
  monitor.py   live monitor (`monitor`) and kill switch (`stop`)
  export.py    merge LoRA adapter into the base model for Ollama
  client.py    OpenAI-compatible inference client + test modes
  config.py    provider/model settings (reads .env)
Modelfile      Ollama model definition (points at the merged model)
sky/train.yaml optional cloud-GPU path (SkyPilot)

Design notes (what this project does and why)

  • Local-first. With a 12 GB consumer GPU on hand, training and serving run locally — far cheaper than per-hour cloud GPUs or always-on managed endpoints, and patient data stays on-device.
  • Small model, QLoRA. gpt-oss-20b doesn't fit 12 GB; Qwen2.5-3B does, and 4-bit QLoRA keeps the base ~2 GB. Full runs peaked at ~6–11 GB VRAM at seq 2048 — comfortable on 12 GB.
  • Pinned stack. uv first resolved transformers 5 / trl 1, which require torch ≥ 2.7 (uses torch.float8_e8m0fnu); cu124 tops out at torch 2.6, so the training extra is pinned to transformers 4.x / trl 0.x.
  • Windows fix. pyarrow segfaults (access violation) when imported after the ML stack; train.py imports datasets first to avoid it.
  • Data split. MTS-Dialog ships only a train split, so ingest.py carves a deterministic 5 % eval set itself.
  • Operability. Checkpoints + auto-resume, a file-based kill switch, and a JSON status heartbeat make long runs safe to stop, watch, and continue.

Scaling / next steps

  • Quality not enough at 3B? Bump MODEL_ID to Qwen2.5-7B-Instruct (Apache 2.0, still fits 12 GB with QLoRA).
  • More data: add adapter classes in ingest.py (e.g. ACI-Bench).
  • All data here is English; a German-language product needs German clinical data later — a separate data problem after the PoC.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages