Slip 👂🟠 — Catch the slip

Reading aloud is how you learn. Knowing exactly which sound you slipped on is how you fix it. 100% in your browser · 0% uploaded · every slip pinned to the moment it happened.

Slip is a browser‑based reading‑aloud pronunciation coach. You bring any passage (type it, paste it, or photograph a page), read it aloud once, and Slip re‑renders the passage with every slipped sound highlighted — then you tap a highlight to scrub your own recording to that exact moment and hear the slip: expected /θ/ → heard /f/. Not the wrong word — the wrong sound.

It runs entirely on your device. After a one‑time model download it works in airplane mode, and the audio of a learner's voice never leaves the machine.

The thesis that beats "isn't this just an LLM?": the model listens; the math judges. A real in‑browser phoneme neural net produces frame‑wise posteriors; a deterministic, from‑scratch CTC forced aligner + Goodness‑of‑Pronunciation (GOP) makes the call.

---

What actually runs (the honest contract)

Everything below genuinely works on a stranger's own input — nothing is mocked or canned.

Piece	What's real
The voice	A live MediaRecorder capture at 16 kHz, decoded + resampled in‑browser.
The text	Whatever you bring — typed, pasted, or OCR'd from a phone photo via tesseract.js (on‑device WASM, per‑word confidence flagged, never hidden).
The acoustic model	facebook/wav2vec2-lv-60-espeak-cv-ft (a ~315 M‑param wav2vec2‑LARGE that emits IPA phonemes, not words), run in‑browser via transformers.js / ONNX Runtime Web, loaded for the raw frame‑wise logits (not a transcript).
The reference	The expected phoneme sequence is generated by espeak‑ng (WASM) — the same phoneme inventory the model was trained on — then matched against the model's own 392‑token vocabulary.
The alignment	A CTC trellis + Viterbi backtrace (the torchaudio forced_align algorithm) reimplemented from scratch in TypeScript — the universal 2L+1 blank‑interleaved form.
The verdict	A per‑phoneme GOP‑CTC score computed deterministically over the aligned posteriors → which sound slipped, expected → heard, the onset (on the ~20 ms frame grid), and a severity.
Persistence	Text, settings and slip maps live in the browser (IndexedDB). "Clear all local data" really wipes them.
Privacy	A live network‑call counter (window.fetch/XHR instrumented) you can corroborate in DevTools. During record → analyse it stays at 0. The only network call ever is the one‑time model download.
Share card	Rendered to a PNG on the device (html-to-image) — a picture, not an upload.
Server seam	An opt‑in fallback for weak devices (services/infer, FastAPI) returning the identical logits so the identical TS math runs either way. Off by default; degrades honestly when not configured.

Honest numbers (not human‑parity — that's the point): model ~318 MB int8, cached once; posteriors at ~50 fps / 20 ms hop; GOP‑CTC ≈ 0.44–0.46 phone‑level correlation vs human raters — useful and directionally right, not examiner‑grade. Slip is a practice coach, not a speech‑pathology assessment: it tells you which sound slipped, not why.

---

Architecture

A single‑page, client‑rendered PWA. The heavy ML libraries are loaded lazily (dynamic imports) so they never bloat the initial bundle or run during SSR.

 bring text ──┐
 (type/paste/ │        ┌──────────── lib/g2p.ts ───────────┐
  photograph) │        │ espeak-ng (phonemizer WASM) →      │
              │        │ greedy-match vs vocab.json →       │  expected phoneme ids
              ▼        │ grapheme anchoring (th ← /θ/)      │  + char anchors
   record (16 kHz) ───┐└────────────────────────────────────┘        │
   lib/audio.ts       │                                               ▼
              ┌────────▼────────────┐   raw logits   ┌─────────────────────────────┐
              │ lib/model.ts         │  [F × 392]     │ lib/dsp.ts  log_softmax     │
              │ transformers.js      │ ─────────────▶ │ lib/align.ts forced align   │
              │ AutoModelForCTC      │                │ lib/gop.ts   GOP-CTC        │
              │ (onnx-community ONNX)│                └──────────────┬──────────────┘
              └──────────────────────┘                              │ highlights
                                                                     ▼
                                            lib/analyze.ts → Highlight[] (severity,
                                            expected→heard, onset, replay segment)
                                                                     │
                                                                     ▼
                                            S4 "Catch the slip" — tap → scrub → replay

lib/backend.ts chooses the browser model (default) or the opt‑in server seam; either way the same lib/dsp + lib/align + lib/gop run, so the verdict is identical.

Project layout

repo/
├── app/
│   ├── layout.tsx          root layout, metadata, viewport, manifest
│   ├── page.tsx            the SPA shell — renders the active screen
│   ├── manifest.ts         PWA manifest (cream theme, maskable icon)
│   ├── globals.css         the whole design system (ported from the Step-2 mockups)
│   └── icon.png            app / favicon
├── components/
│   ├── ui.tsx              Shell, StatusBar, TopBar, Ledger, Mascot, Icon
│   ├── Passage.tsx         the reading surface with grapheme-anchored highlights
│   ├── Waveform.tsx        canvas waveform: bars, coral region, amber playhead (a11y slider)
│   └── screens/            S0 Home … S8 Settings + model-load / listening states
├── lib/
│   ├── model.ts            in-browser phoneme model (raw logits)
│   ├── g2p.ts              espeak-ng G2P + grapheme anchoring
│   ├── dsp.ts align.ts gop.ts   the deterministic verdict (log-softmax → align → GOP)
│   ├── analyze.ts          orchestrates the pipeline → Highlight[]
│   ├── audio.ts            capture, 16 kHz resample, level meter, peaks, segment replay
│   ├── ocr.ts              tesseract.js (per-word confidence)
│   ├── sharecard.ts        on-device PNG export
│   ├── backend.ts          the PHONEME_BACKEND seam (browser | server)
│   ├── network.ts          the live "0 network calls" ledger
│   ├── store.tsx           tiny client store (useSyncExternalStore)
│   └── vocab.json          the model's 392-token vocabulary (bundled)
├── public/
│   ├── art/ fonts/         hand-built icons + self-hosted fonts (offline)
│   ├── demo/               bundled sample recordings (real audio, deliberate slips)
│   ├── tesseract/          self-hosted OCR worker/core/traineddata (offline)
│   ├── icons/ sw.js        PWA icons + offline service worker
└── services/infer/         the optional FastAPI inference seam (see its README)

---

Getting started

Prerequisites: Node.js ≥ 20.9.

npm install
npm run dev          # http://localhost:3000

Open it, tap Read something, allow the mic, and read the passage aloud — or tap a demo line (no microphone needed; it analyses a bundled sample recording for real).

Production build:

npm run build
npm run start

The first analysis downloads the ~318 MB phoneme model from the Hugging Face CDN (shown on the one‑time model‑load screen) and caches it. After that it runs fully offline — reload with the network off and it still works.

Environment variables

All optional. The app derives every URL from the runtime origin; nothing is hardcoded.

Variable	Purpose
NEXT_PUBLIC_SITE_URL	Absolute site URL for Open‑Graph image resolution (Vercel sets VERCEL_URL automatically).
NEXT_PUBLIC_PHONEME_SERVER_URL	Enables the opt‑in server seam (e.g. https://slip-infer.example.run.app). Until set, choosing Server in settings degrades honestly to on‑device.
NEXT_PUBLIC_PHONEME_BACKEND	Set to server to default to the server seam (requires the URL above).

The optional inference server lives in services/infer — see its README to run or deploy it.

Cross‑origin isolation

The app sets Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: credentialless (in next.config.ts) so ONNX Runtime Web can use multi‑threaded WASM (SharedArrayBuffer) while still loading the model cross‑origin from the HF CDN. Browsers without credentialless fall back to single‑threaded WASM — slower, but it still works.

---

Design & accessibility

The visual system is a faithful build of the Step‑2 design package: warm cream paper, one deep slate‑teal ink, one amber accent, and a severity ramp (honey → orange → coral, never stoplight‑red) that appears only on a slipped sound. Fonts are self‑hosted (Fredoka / Lexend / Inter / JetBrains Mono) so the app — and its IPA glyphs — render correctly offline. The waveform is backed by a real role="slider"; highlights are keyboard‑focusable; severity is never colour‑alone.

Honest limitations

• The model listens; the math judges. If the model has an off day, the number is still honest math and the verdict is flagged, not hidden.
• A coach, not a clinician — Slip points at which sound slipped, not why.
• Accuracy is best for the model's trained language(s) and degrades with heavy noise or very thick accents. Rule‑based G2P can mis‑expand rare proper nouns and numerals — shown and editable, not hidden.
• The GOP severity thresholds are honest, calibrated starting points (tunable via the Slip sensitivity slider), not examiner‑grade cutoffs.

Reading aloud is how you learn. Knowing which sound you slipped on is how you fix it. Catch the slip. 👂🟠