The full plan (architecture, data model, protocol, rate limiter, services, diagrams) lives in the "CollabEdit v2 Plan" doc. This file tracks progress against its phases.
Goal: a Google Docs–style editor built on FastAPI, Yjs/pycrdt, PostgreSQL and Redis, whose signature feature is Git-style document branches with merge requests. A live MVP ships at the end of week 6 (Gate 1) and the full v2 at the end of week 14 (Gate 2), on $0/month hosting.
- 0 · Setup and CI (week 1)
- Monorepo layout:
apps/api,apps/web,infra,docs - FastAPI skeleton: settings, async Postgres engine, Redis client, JSON logs with request ids
-
/api/healthz(liveness) and/api/readyz(dependency checks with timeouts) - Alembic wired to app settings
- React + TypeScript + Vite skeleton with Tailwind, TanStack Query, React Router
- Docker Compose for Postgres and Redis; API Dockerfile
- GitHub Actions: lint, types, tests, build, real-service readiness, Docker image
- ADRs 0001–0003
- Verified locally in Docker: migrations, readiness 200, API image builds and runs
- First green CI run on GitHub (run 36440356607)
- Monorepo layout:
- 1 · Accounts and rate limiter (week 2)
- Register, login, refresh, logout, logout-all,
/me(Argon2id, 15-minute JWT) - Rotating refresh tokens in an httpOnly cookie, reuse detection, two-tab grace window
- Token-bucket rate limiter in Redis (Lua), fail open/closed per policy, local fallback
- Documents: create, idempotent
PUTwith client ids, list by scope, rename, trash, restore - Roles enforced server-side; outsiders get 404, too-low roles get 403
- Web: sign-in, registration, silent session restore, dashboard, cross-tab sign-out
- 67 API tests on real Postgres and Redis; 35 web tests with MSW; checked in a real browser
- Register, login, refresh, logout, logout-all,
- 2 · Live editing, offline core (weeks 3–4)
- Sync server on pycrdt: one room per open document, Yjs sync protocol, per-connection send queues, awareness relay that stamps the authenticated name on every cursor
- One-time 30-second WebSocket tickets (Redis
GETDEL); close codes the client acts on - Viewers and commenters can follow live but their edits are refused server-side
- Per-connection message budget, 10 connections per user, slow clients dropped
- Edits saved to an append-only log in Postgres, batched every 500 ms or 50 updates, and flushed on room eviction and on shutdown (pulled forward from phase 3; compaction stays in phase 3)
- Moving a document to trash disconnects everyone editing it
- Web: Tiptap editor, live cursors and presence avatars, per-user undo, own provider with fresh tickets per attempt and exponential backoff, IndexedDB offline copy, sync badge, offline copies wiped on sign-out and when access is lost
- 97 API tests (incl. a 150-case Hypothesis convergence property), 53 web tests; checked in a real browser with two accounts, including killing the API mid-edit
- Found upstream: pycrdt/yrs 0.27 drops some out-of-order updates (Yjs does not); see ADR 0012
- 3 · Storage, versions, search (week 5) ✅ 2026-10-02
- Compaction: the edit log folds into one snapshot at 500 rows and when a session ends,
under an advisory lock; deletes only the rows it read; loads read base and log in one
REPEATABLE READsnapshot. Opening a 10,000-keystroke document: 218 ms → 17 ms (50,000: 6.5 s → 0.28 s); see ADR 0003 - Version history: automatic (session end, every 10 minutes), named, and pre-restore versions; newest 50 automatic kept, up to 100 named; editors only
- Restore is an ordinary edit made by the browser through y-prosemirror: concurrent edits survive and Undo reverts it (ADR 0014)
- Full-text search over titles and text with Postgres (GIN, weighted, prefix matching on stemmed and unstemmed words), highlighted snippets without HTML (ADR 0015)
- Web: search box on the dashboard (debounced, kept in the URL); version history dialog with save, name, read-only preview and restore
- 173 API tests (a 40-schedule compaction property, out-of-order commits, the advisory lock, hostile search input), 79 web tests (restore and Undo on a real editor); checked in a real browser with two tabs
- Found: pycrdt drops waiting updates when re-encoding a document, so snapshots store merged updates (ADR 0012, pitfall 4)
- Found and fixed: a connect race left an orphaned socket that kept rooms open forever; the client now drops stale connects, the server closes sockets silent for 150 s, and the heartbeat judges unanswered pings rather than silence (ADR 0012)
- Compaction: the edit log folds into one snapshot at 500 rows and when a session ends,
under an advisory lock; deletes only the rows it read; loads read base and log in one
- 4 · Sharing and roles (week 6) → Gate 1: MVP live on the web ✅ 2026-10-01
- Members: list, invite by email with a role, change role, remove, leave, transfer ownership
- Share links: role, expiry, turn off; token shown once and stored hashed; accept never lowers access; token sent in the body so it stays out of logs
- Access changes close open editors instantly (4403 removed, 4409 reconnect with new role)
- Web: Share dialog,
/share/:tokenpage (sign-in first when needed), provider handles 4409 - Deploy prep:
render.yaml,vercel.json,docs/DEPLOY.md; Neon URLs adapted for asyncpg; migrations at container start; production refuses a dev secret or insecure cookies; config errors never echo secrets - 121 API tests, 64 web tests; production image rehearsed locally against a fresh database
- Deployed: web https://collaborative-editor-flax.vercel.app, API https://collabedit-api.onrender.com, Neon Postgres 18 and Redis Cloud (all Singapore)
- Production smoke test, 31/31: frontend, auth and cookie rotation, documents, sharing, live editing between two accounts (~110 ms), presence, persistence, revocation
- Found in production: Render's edge drops WebSocket close frames; client heartbeat added
- Scheduled ping keeps the free API awake 08:00–23:59 IST and lets it sleep overnight
- 5 · Multi-instance Redis (week 7) ✅ 2026-10-11, built last of 5–7
- A Redis channel per open document or branch, and a control channel for whole-document events; binary messages with a version byte, so releases can overlap during a deploy
- The receiving instance saves each edit; the others apply and relay it
- Opening a room subscribes, then reads Postgres, then runs the Yjs sync handshake with the other instances, so edits not saved yet are never missed
- Edits and cursors published with a leading-edge throttle (at once, then at most every 50 ms); nothing published while an instance is alone with a document
- Kicks after access changes, trashing, merges (with a cross-instance freeze) and the comment signal reach every instance; retried if Redis refuses them
- Own reconnect handling: a dropped subscription re-syncs from Postgres and its peers;
/api/readyzreportspubsub; the API starts with Redis down - Migrations take a Postgres advisory lock, so instances can start together
- Docker Compose runs two instances behind nginx (round robin, no sticky sessions);
infra/k6/run.shload-tests them, and CI runs it on every push - 248 API tests (17 run two instances: convergence over 5 seeds, a room opened later, a dropped subscription, lost publishes, kicks, merges, presence); real browsers through nginx on two instances: 7/7, including stopping one instance mid-edit (reconnected in 221 ms)
- Measured (ADR 0004): two instances, 100 editors: p50 1 ms, p95 17 ms; four busy documents, 80 editors: p50 4 ms, p95 48 ms, 80% fewer Redis commands than one per edit; about 60 KB per open document
- Found: redis-py re-subscribes silently after a reconnect (messages lost unnoticed); a stale receiver count could mark a room alone while a peer listened; a lost reply to HELLO stranded a newcomer; the first migration lock rolled migrations back
- 6 · Comments (week 8) ✅ 2026-10-03, built before 5
- Threads anchored to a selection with Yjs relative positions, stored beside the text rather than in it, so commenters can comment without edit rights and a thread stays on its words through local, remote and offline edits (ADR 0017)
- Replies, edit (author), delete (author, or an editor or owner), resolve and reopen; a reply reopens a resolved thread; deleting the first comment deletes the thread
- Live: the server sends a one-byte "comments changed" signal over the document's socket and clients refetch; a reconnect refetches whatever was missed
- Email: a new thread notifies the owner, a reply notifies the thread's participants, never the author and only current members; SMTP (Gmail app password) or log-only (DEPLOY.md)
- Comments on branches belong to the branch; merged and closed branches are read-only
- Web: comments panel beside the editor, highlights drawn as decorations, Ctrl+Alt+M, open/resolved lists, notification links open their thread
- 230 API tests (permissions, moderation, live signals, emails, caps, rate limit), 104 web tests (anchors through local and remote edits; a test fails if local edits are not mapped); checked in two real browsers: 11/11, highlight stays on its words for both
- Found: ProseMirror applies a local edit before Yjs sees it, so resolving anchors then reads stale text; local edits map the known ranges instead (ADR 0017). A read-only editor takes no keyboard input, so the shortcut listens on the window
- 7 · Branches and merge requests (weeks 9–10) ✅ 2026-10-02, built before 5 and 6
- A branch is its own stream: its own edit log, compaction snapshot and live room, keyed by branch id; fork copies main, and its starting state is the merge base
- Review: diff3 over paragraphs between the merge base, main and the branch; added, removed, changed and conflicting passages, and a preview of main after the merge
- Merge: refuses conflicts and a branch changed since review; saves main as a "before merging" version; applies the branch through main's live room, so everyone sees it at once
- Update from main moves the merge base up, the way
git merge maindoes, which is how a conflict is settled; close abandons a branch; merged and closed branches are read-only - Commenters branch and propose; owners and editors merge; at most 10 open branches, branch creation rate-limited; losing access or trashing the document disconnects branch editors
- Web: Branches dialog, branch editor with actions, review page with change cards and preview
- 199 API tests (a diff3 partition property, conflicts by edit and by deletion, review after main moved on, stale review, permissions), 89 web tests; checked in a real browser: two tabs, a clean merge, then a conflict settled by updating from main
- Found:
FOR UPDATEon the branch row blocked the foreign-key checks of the branch's own edits (KEY SHARE locks), so a merge waited on itself; the row is lockedFOR NO KEY UPDATE
- Also fixed 2026-10-02: search text backfilled for documents edited before search existed; the GitHub keep-awake schedule proved unreliable (two runs in six hours), so an external cron now does that job (docs/DEPLOY.md)
- Also fixed 2026-10-15: the API took 11 minutes to wake. A migration lock left by a killed
process's lingering database session can block startup; startup now skips the lock when the
schema is current and waits at most 60 s otherwise (
alembic/env.py, tested) - 8 · Google sign-in and email (week 11) ✅ 2026-10-22
- Sign in with Google: authorization code flow with PKCE, state bound to the browser by a
cookie, nonce, ID token checked against Google's keys;
openid email profileonly - Accounts matched by Google's subject id; a Google sign-in on an address someone registered but never verified removes that password and its sessions (pre-hijacking)
- Password reset by email: same answer for unknown addresses, single-use 30-minute links, all sessions end; Google-only accounts can add a password this way
- Email verification at sign-up, resend, banner; comment emails only to verified addresses
- Tokens stored as SHA-256 and sent after
#, out of server logs; dev logs show emails so links can be followed, production never does - Web: Continue with Google, forgot / reset / verify pages, verification banner, and a privacy policy page (Google requires one before the sign-in app can be published)
- 282 API tests (20 against a fake Google with a real RSA key that enforces PKCE), 117 web tests; browser run against a local fake Google: 12/12 (Google sign-in and cancel, sign-up and verify, reset)
- Deployed 2026-10-22 with a published Google OAuth client and Gmail SMTP; checked locally against real Google, then on the live site: Google sign-in, a real verification email, and a password reset
- Found: building an HTTP client reads the whole CA bundle (half a second on Windows), so it is made only when Google is configured
- Found in deployment: Google will not publish a sign-in app without a home page and a privacy
policy link, so the app now has
/privacy
- Sign in with Google: authorization code flow with PKCE, state bound to the browser by a
cookie, nonce, ID token checked against Google's keys;
- 9 · Export, offline app, polish (week 12) ✅ 2026-10-29, not yet deployed
- Stars: personal to each member (
document_members.starred_at), a Starred view, a star on every row - Editor: checklists, tables, images by web address, links (Ctrl+K), word count and a shortcuts dialog; one extension list shared by the live editor and version previews
- App shell offline and installable: a Workbox service worker precaches every built file,
the editor and Word chunks included; never touches
/api; new versions wait for Reload - Opens offline signed in as the last person; document lists, summaries and comments are saved on the device; online-only actions are disabled with the reason
- Offline create, rename, star, comment and reply, queued in an IndexedDB outbox and
replayed in order; every request repeat-safe (browser UUIDv7 +
PUT, commentclient_ids); refusals listed in a banner (ADR 0019) - Text edited offline reaches the server on reconnect even if its editor is closed; documents can be kept available offline ahead of time
- Header badge: offline or sending, with "N changes not synced" and "last online"; signing out warns about unsent changes, then wipes the device
- Export to Markdown (own converter), Word (
docx, loaded on demand) and PDF (print stylesheet), all in the browser and offline (ADR 0020) - Dark mode as a choice (system, light, dark) that survives reloads; layouts checked at phone width
- 289 API tests, 174 web tests; a browser run of the production build against the local API: 31/31 (service worker, offline reload, offline create/edit/comment/star/export, sent exactly once on reconnect, print, phone width, dark mode, sign-out wipe)
- Found: TanStack Query holds every mutation while the browser reports offline, so nothing reached the outbox; mutations now always run and the outbox decides what waits
- Found: the first service worker did not control the page that installed it, so a document opened offline before any reload had no editor code; it now claims the page at once
- Stars: personal to each member (
- 10 · AI assistant (week 13)
- 11 · Hardening and launch (week 14) → Gate 2: v2 release
If a phase slips, cut in this order: AI, then export, then Google sign-in. Never branches or tests.
| ADR | Decision |
|---|---|
| 0001 | Yjs CRDT for merging concurrent edits |
| 0002 | PostgreSQL as the system of record |
| 0003 | Append-only edit log plus snapshots |
| 0004 | Redis pub/sub between API instances, Yjs sync between servers |
| 0005 | Token bucket in a Redis Lua script, fail open or closed per policy |
| 0006 | One-time WebSocket tickets instead of tokens in URLs |
| 0011 | In-memory access tokens, rotating refresh cookie with reuse detection |
| 0012 | Own sync server and client provider on pycrdt/Yjs, and the pycrdt pitfalls |
| 0013 | Membership-only access, hashed share links, instant revocation |
| 0014 | Versions as snapshots; restore as an ordinary edit made by the browser |
| 0015 | Full-text search in Postgres, stemmed and as-typed prefixes |
| 0016 | Branches as separate streams; diff3 over paragraphs gates CRDT merges |
| 0017 | Comments anchored with Yjs relative positions, stored beside the text |
| 0018 | Google sign-in with PKCE; hashed single-use reset and verification links |
| 0019 | Service worker app shell, an outbox of repeat-safe requests, offline edits sent on reconnect |
| 0020 | Markdown, Word and PDF exports built in the browser from the editor's JSON |