Finetuning Personaplex on Specific Domains with Synthetic Data

Status: WIP. Not tested on a wide range of setups - we welcome any and all contributions for improving developer experience.

Finetune PersonaPlex (a Moshi 7B variant with voice + role conditioning) on a specific domain like pharmaceutical adherence for use in real-time voice conversations, with mid-conversation grounding via a puppeteer LLM that injects context into the model's text stream.

Examples

Two short clips illustrate the input → output of the pipeline.

Synthetic training data — VibeVoice-rendered stereo (left=agent, right=patient) of a Claude-generated pharma support dialogue, the kind of sample that ends up in data/<dataset>/stereo_wav/:

synthetic_data_example.mp4

Trained model output — generation eval of a PersonaPlex pharma finetune at step 1024 (run pharma-v2-22, LoRA r64), with mid-conversation context injection from the puppeteer LLM. The finetuned model learned the structure of a pharmaceutical adherence call and succcessfully applies it in this example.

The base model fails at executing a structured call because it's not good at following instructions although the finetuned model is far from perfect as well. The base model is the client in this example and it is confused especially at the start but the finetuned model (Mia) manages to mostly stay on track, like a real call:

personaplex_eval_512.mp4

What we did with this code

We used this pipeline through March–April 2026 to finetune PersonaPlex on two domains: insurance brokerage (~3,800 synthetic dialogues, ~389 hours of audio) and pharmaceutical patient support (~2,000 calls, ~166 hours), all generated by Claude and voiced with VibeVoice. Across roughly 84 training runs we hit and worked through five recurring failure modes — depformer LoRA corruption, train/inference channel mismatches during context injection, PAD-token collapse, badly applied L2 regularization, and injection dropout at chunk boundaries. The two non-obvious fixes that mattered most were text_padding_weight=0.0 (which eliminated the PAD-collapse degenerate optimum) and L2 regularization on lora_B weights (which prevented catastrophic forgetting at the low learning rates we needed for stable generation). Our best pharma checkpoint reached a 4.84/5 grounding score with the puppeteer context-injection mechanism. A surprising null result on the companion side: pure finetuning was statistically neutral against the base model with a system prompt — sampling parameters mattered more than any training change. Full writeup, configs, and unsampled metric history for all 59 logged runs are in docs/history/.

Released artifacts

Artifact	What it is
`demegire/personaplex-finetune-pharma`	Full pharma run `adhery-v2-21` — merged checkpoint at step 448, LoRA snapshots every 64 steps, training/eval metrics, wandb + TensorBoard logs, generation- and Gemini-judge eval outputs.
`demegire/personaplex-finetuning-pharma-data-sample`	10-dialogue sample (8 train / 2 eval) mirroring this pipeline's exact on-disk format — sidecar JSONL, manifests, stereo+mono WAVs, WhisperX alignments, scripts, speaker prompts. Use as a template when assembling your own dataset.

Overview

The pipeline generates synthetic broker/client dialogues with embedded context injections, renders them to audio, and finetunes PersonaPlex to (a) speak grounded in provided facts, (b) stay silent while receiving injected context, and (c) hedge when information isn't available.

                                              [1] generate_dialogues_sync.py
                                                  Claude API, 8-dim LHS
                                                        │
                                                        ▼
outbound_insurance.jsonl ─────────── text_prompt + dialogue + context_injections
                                                        │
                            ┌───────────────────────────┤
                            ▼                           ▼
               [2] parse_dialogues.py         (injection metadata preserved
                    scripts/{id}.txt           for later frame offset computation)
                            │
                            ▼
               [3] generate_audio.py          VibeVoice 7B TTS (multi-GPU)
                    mono_wav/{id}.wav
                            │
                            ▼
               [4] create_stereo.py           WhisperX align + channel routing
                    --dialogues *.jsonl       propagates text_prompt + context_injections
                    stereo_wav/{id}.wav+.json  both-speaker alignments + turns metadata
                            │
                            ▼
               [5] compute_injection_offsets.py   after_turn → frame_offset
                    stereo_wav/{id}.json          (proactive/reactive placement)
                            │
                            ▼
               [6] create_manifest.py
                    dataset/{train,eval}.jsonl
                            │
                            ▼
               [7] moshi-finetune train.py    FSDP + LoRA (multi-GPU)
                    runs/                     context injection splicing + loss masking

At inference, an external puppeteer LLM injects <context> tokens into the text stream via a hidden channel. The model reads the context through the transformer but the tokens are suppressed from all external output (websocket, transcripts, silence metrics).

Related work. Sakana AI's KAME (Tandem Architecture for Enhancing Knowledge in Real-Time S2S Conversational AI) (arXiv:2510.02327, accepted at ICASSP 2026, posted April 30, 2026) describes the same core idea — a backend LLM asynchronously enriching a fast speech-to-speech model with knowledge during the user's turn — under a different name and with a different architecture (parallel "tandem" model rather than in-stream text-token splicing). Our puppeteer work was developed independently in March–April 2026 ahead of this release; we recommend KAME as the more thoroughly evaluated reference for this class of approach.

Requirements

Python 3.10+ with two venvs:
- .venv/ — VibeVoice, WhisperX, pipeline scripts
- moshi-finetune/.venv/ — Moshi finetuning (separate torch version)
GPUs: 3-5× 24 GB (RTX 4090 tested) for training; a single H100 80 GB also works.
flash-attn built against matching CUDA toolkit
Anthropic API key (for dialogue generation and LLM review). Copy [.env.example](.env.example) to .env and fill in.

For an end-to-end zero-to-trained walkthrough on a fresh GCP VM, see docs/SETUP_GUIDE.md. For inference / serving quickstart, see QUICKSTART.md.

Pipeline steps

1. Generate dialogues

Generate synthetic broker/client dialogues with structured system prompts and context injection markers using Claude.

.venv/bin/python pipeline/generate_dialogues_sync.py \
  --output outbound_insurance.jsonl \
  --num-dialogues 500 \
  --workers 10

Each record contains:

text_prompt: Structured pre-call brief (ROLE / CLIENT / KNOWN / UNKNOWN / GOAL / STYLE)
dialogue: Full transcript with BROKER (Name): / CLIENT (Name): turns
context_injections: [{after_turn: int, text: str}, ...] — directive coaching instructions

8-dimensional Latin Hypercube Sampling ensures diversity across broker persona, client persona, outbound goal, emotional tenor, domain vocabulary, complexity, call outcome (80/20 success/failure), and injection count (0/3/6/9 weighted).

75% of dialogues include context injections (directive coaching mid-conversation), 25% without (teaches hedging on unknowns).

2. Parse dialogues

.venv/bin/python pipeline/parse_dialogues.py data/companion_training/companion_training.jsonl data/companion_training/scripts 6000

Converts JSONL transcripts to VibeVoice Speaker N: format. Strips [Call start/end] markers, normalizes unicode, skips dialogues over 3000 words.

3. Generate mono audio

.venv/bin/python pipeline/generate_audio.py \
  --model-path vibevoice/VibeVoice-7B \
  --dialogues data/companion_training/companion_training.jsonl \
  --output-dir data/companion_training/mono_wav \
  --scripts-dir data/companion_training/scripts \
  --voices-dir data/companion_training/speaker_samples \
  --gpus 0 --resume --batch-size 16

VibeVoice 7B TTS with deterministic per-dialogue voice assignments. --gpus spawns one process per GPU.

4. Create stereo WAVs + alignments

.venv/bin/python pipeline/create_stereo.py \
  --mono-dir data/<dataset>/mono_wav \
  --scripts-dir data/<dataset>/scripts \
  --output-dir data/<dataset>/stereo_wav \
  --dialogues data/<dataset>/<dataset>.jsonl \
  --whisper-model large-v3 \
  --gpus 0 --resume --batch-size 16

WhisperX transcribes mono audio → word-level timestamps via wav2vec2 forced alignment
Greedy sequential matching maps words to ground-truth speaker labels
Broker → left channel, client → right channel (10ms crossfade)
Writes both-speaker alignments (SPEAKER_BROKER/SPEAKER_CLIENT), explicit turns metadata, and propagates text_prompt + context_injections from the JSONL

5. Compute injection frame offsets

.venv/bin/python pipeline/compute_injection_offsets.py --stereo-dir data/<dataset>/stereo_wav

Converts after_turn (turn index) to frame_offset (absolute frame at 12.5 Hz) using preserved turn metadata:

Proactive (~70%): Turn before marker is CLIENT → frame_offset = start of client utterance (context loads during client speech, zero perceived latency)
Reactive (~30%): Turn before marker is BROKER hedge → frame_offset = end of broker utterance (context loads during pause after "let me check on that")

6. Create manifest

.venv/bin/python pipeline/create_manifest.py \
  --stereo-dir data/<dataset>/stereo_wav \
  --train-output data/<dataset>/dataset/train.jsonl \
  --eval-output data/<dataset>/dataset/eval.jsonl \
  --eval-fraction 0.05

7. Finetune

cd moshi-finetune
CUDA_VISIBLE_DEVICES=0 .venv/bin/torchrun --nproc-per-node 1 \
  -m train ../configs/pharma_demo.yaml

Parameter	Value	Notes
Base model	`nvidia/personaplex-7b-v1`	PersonaPlex 7B (dep_q=16)
LoRA rank	128	~5.6% params trainable (~500M / 8.9B), depformer excluded (`skip_depformer`)
duration_sec	60	Sequence length per sample
batch_size	4	Per-GPU; effective = batch_size x num_gpus
max_steps	3000
lr	2e-6	OneCycleLR with 5% warmup
system_prompt	enabled	Voice + text prompt prefix, loss masked
puppeteer	enabled	Context injections spliced into text channel

Context injection during training: The interleaver reads context_injections from each sample's JSON, wraps text in <context>...</context> tags, and inserts frames into the sequence at frame_offset. Agent audio is set to SILENCE_TOKENS (broker stays quiet); user audio is preserved from the displaced conversation frames (client may be speaking — ~70% proactive injections). A context_mask is built over injection frames: text loss is zeroed (model didn't produce these tokens); audio loss stays active (model learns silence for agent audio while tracking user audio).

Context injection at inference: lm.py forces agent audio output to SILENCE_TOKENS when _injecting_context is True, matching the training signal. User audio input flows through unmodified (preserved by the provided mask in the cache update).

8. Merge LoRA + serve

# Merge
.venv/bin/python pipeline/merge_lora.py \
  --checkpoint runs/<run_name>/checkpoints/checkpoint_000128/consolidated \
  --output runs/<run_name>/merged/model.safetensors

# Serve
cd personaplex/moshi
.venv/bin/python -m moshi.server \
  --moshi-weight ../../runs/<run_name>/merged/model.safetensors \
  --static ../../personaplex/client/dist

The server accepts mid-session context injection via JSON websocket messages:

{"type": "context", "text": "Quote the Berkshire at $13,100. Emphasize savings."}

Evaluation

Gen eval (during training)

Runs Gemini Live ↔ Moshi dialogues at checkpoint time, evaluates coherence/naturalness/effectiveness/grounding via Claude, logs to W&B.

python -m finetune.gen_eval \
  --checkpoint runs/.../consolidated \
  --run-dir runs/... --step 200 \
  --config configs/pharma_demo.yaml

Batch eval (standalone)

30 dialogues (10 seeds x 3 scenarios), two in parallel across GPU pairs.

python pipeline/batch_eval.py \
  --checkpoint runs/.../consolidated \
  --output-dir runs/.../batch_eval \
  --gpus 1,2,3,4

# With puppeteer (scheduled context injections):
python pipeline/batch_eval.py \
  --checkpoint runs/.../consolidated \
  --output-dir runs/.../batch_eval_puppeteered \
  --gpus 1,2,3,4 \
  --puppeteer knowledge_base.json

Puppeteer knowledge base format: {scenario_id: [{frame: int, text: str}, ...]}.

Compare runs

Statistical comparison with Wilcoxon tests, Cliff's delta, bootstrap CIs, control charts, and automated grounding rate (factual claim extraction vs provenance).

python pipeline/compare_runs.py \
  --run-a runs/base_eval/results.json \
  --run-b runs/lora_eval/results.json \
  --output runs/comparison_report

Produces comparison.json + dashboard.html with interactive Chart.js visualizations.

Architecture

Moshi 3-stream model: agent text (1ch) + agent audio (8 codebooks) + user audio (8 codebooks) = 17 channels
PersonaPlex dep_q=16 (vs standard Moshi dep_q=8) — depformer handles all 16 audio codebooks
Mimi codec: 12.5 Hz frame rate, 24kHz sample rate, 8 codebooks per audio stream
System prompt prefix: voice prompt → silence → <system> text prompt </system> → silence → conversation
Context injection: <context> tokens spliced into text channel mid-conversation; hidden from all external output via _injecting_context flag. Agent audio forced to silence during injection (both training and inference); user audio preserved from conversation (client may be speaking)
Loss masking: Prompt region masked via prompt_lengths; injection region masked for text loss only via context_masks (audio loss stays active — agent learns silence, user audio predicted normally)
LoRA scope: Applied to backbone transformer only (skip_depformer=True); depformer is frozen to preserve pretrained audio codec generation
Queue priority: context > nudge > greeting (context injection takes precedence over silence-breaking nudges)

File structure

voice-training/
├── configs/
│   ├── pharma_demo.yaml        Training config (insurance broker)
│   └── pharma_demo.yaml                  Training config (pharma patient support, with context injection)
├── pipeline/
│   ├── generate_dialogues_sync.py        [1] Dialogue + prompt + injection generation (Claude)
│   ├── parse_dialogues.py                [2] JSONL → VibeVoice scripts
│   ├── generate_audio.py                 [3] Scripts → mono WAV (VibeVoice TTS)
│   ├── create_stereo.py                  [4] Mono → stereo + alignments (WhisperX)
│   ├── compute_injection_offsets.py      [5] after_turn → frame_offset
│   ├── create_manifest.py               [6] Create train/eval manifests
│   ├── merge_lora.py                     LoRA merge utility
│   ├── batch_eval.py                     Standalone batch evaluation
│   └── compare_runs.py                   Statistical run comparison + grounding rate
├── personaplex/moshi/moshi/
│   ├── models/lm.py                      LMGen with _context_queue + inject_context()
│   ├── bot_to_bot.py                     Bot-to-bot with context injection + --context-injections
│   ├── server.py                         WebSocket server with JSON context channel
│   └── puppeteer.py                      Async puppeteer process (scaffolded)
├── moshi-finetune/
│   ├── train.py                          Training entry point
│   └── finetune/
│       ├── args.py                       Config (SystemPromptArgs, PuppeteerArgs, GenEvalArgs)
│       ├── data/interleaver.py           Tokenizer with prompt + injection splicing
│       ├── loss.py                       Loss with prompt + context masking
│       ├── eval.py                       Evaluation loop
│       └── gen_eval.py                   Generation eval (Gemini Live ↔ Moshi + LLM review)
├── docs/
│   ├── SETUP_GUIDE.md                    Fresh-VM zero-to-trained walkthrough
│   └── history/                          Frozen development artifacts
│       ├── notes/                          Session notes, design docs, experiment reports
│       └── wandb/                          Raw export of 59 historical training runs
├── data/                                 (gitignored) audio data + manifests — regenerate via the pipeline, or grab a 10-sample template from [`demegire/personaplex-finetuning-pharma-data-sample`](https://huggingface.co/datasets/demegire/personaplex-finetuning-pharma-data-sample)
└── runs/                                 (gitignored) training outputs

Documentation map

File	What it is
`README.md`	This file — pipeline overview + architecture
`docs/SETUP_GUIDE.md`	Zero-to-trained walkthrough on a fresh GCP H100 VM
`QUICKSTART.md`	Inference-only quickstart (merge LoRA + serve)
`docs/history/notes/puppeteer.md`	Design doc for the puppeteer / context-injection feature
`docs/history/notes/`	Frozen session notes + experiment reports — debugging log, A/B handoffs, design docs (`puppeteer.md`), and the consolidated `combined_experiment_report.md`
`docs/history/wandb/`	Raw export of every wandb run from the original development period (~8 MB)

License

Our code in this repository is released under the MIT License.

Vendored upstream code keeps its original license — see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for the full list. Notably:

**moshi-finetune/** — Apache-2.0 (Kyutai)
**personaplex/moshi/** — MIT (Kyutai for Moshi runtime, Meta for AudioCraft codec)
**VibeVoice/** — see upstream VibeVoice/LICENSE

Model weights referenced by this repo carry their own licenses, distinct from the code license:

Moshi base weights (kyutai/moshiko-pytorch-bf16) — CC-BY-4.0
PersonaPlex weights (nvidia/personaplex-7b-v1) — NVIDIA Open Model License
VibeVoice weights — see the VibeVoice model card

A finetune you produce by running this code inherits the licenses of the base weights you started from. If you publish a checkpoint derived from PersonaPlex, your release is bound by the NVIDIA Open Model License (and by CC-BY-4.0 attribution, since PersonaPlex itself is built on Moshi). | [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) | Vendored upstream code + license attribution |

Navigation Menu

Repository navigation

About

Repository files navigation

Finetuning Personaplex on Specific Domains with Synthetic Data

Examples

What we did with this code

Released artifacts

Overview

Requirements

Pipeline steps

1. Generate dialogues

2. Parse dialogues

3. Generate mono audio

4. Create stereo WAVs + alignments

5. Compute injection frame offsets

6. Create manifest

7. Finetune

8. Merge LoRA + serve

Evaluation

Gen eval (during training)

Batch eval (standalone)

Compare runs

Architecture

File structure

Documentation map

License

About

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github		.github
VibeVoice		VibeVoice
assets		assets
configs		configs
data		data
docs		docs
moshi-finetune		moshi-finetune
personaplex		personaplex
pipeline		pipeline
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
MODEL_CARD.md		MODEL_CARD.md
QUICKSTART.md		QUICKSTART.md
README.md		README.md
SECURITY.md		SECURITY.md
THIRD_PARTY_NOTICES.md		THIRD_PARTY_NOTICES.md
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

Finetuning Personaplex on Specific Domains with Synthetic Data

Examples

What we did with this code

Released artifacts

Overview

Requirements

Pipeline steps

1. Generate dialogues

2. Parse dialogues

3. Generate mono audio

4. Create stereo WAVs + alignments

5. Compute injection frame offsets

6. Create manifest

7. Finetune

8. Merge LoRA + serve

Evaluation

Gen eval (during training)

Batch eval (standalone)

Compare runs

Architecture

File structure

Documentation map

License

About

Topics

Resources

Security policy

Stars

Watchers

Forks

Contributors

Languages