Skip to content

Latest commit

 

History

History
224 lines (217 loc) · 17.7 KB

PLAN.md

File metadata and controls

224 lines (217 loc) · 17.7 KB

CollabEdit v2 plan

The full plan (architecture, data model, protocol, rate limiter, services, diagrams) lives in the "CollabEdit v2 Plan" doc. This file tracks progress against its phases.

Goal: a Google Docs–style editor built on FastAPI, Yjs/pycrdt, PostgreSQL and Redis, whose signature feature is Git-style document branches with merge requests. A live MVP ships at the end of week 6 (Gate 1) and the full v2 at the end of week 14 (Gate 2), on $0/month hosting.

Phases

  • 0 · Setup and CI (week 1)
    • Monorepo layout: apps/api, apps/web, infra, docs
    • FastAPI skeleton: settings, async Postgres engine, Redis client, JSON logs with request ids
    • /api/healthz (liveness) and /api/readyz (dependency checks with timeouts)
    • Alembic wired to app settings
    • React + TypeScript + Vite skeleton with Tailwind, TanStack Query, React Router
    • Docker Compose for Postgres and Redis; API Dockerfile
    • GitHub Actions: lint, types, tests, build, real-service readiness, Docker image
    • ADRs 0001–0003
    • Verified locally in Docker: migrations, readiness 200, API image builds and runs
    • First green CI run on GitHub (run 36440356607)
  • 1 · Accounts and rate limiter (week 2)
    • Register, login, refresh, logout, logout-all, /me (Argon2id, 15-minute JWT)
    • Rotating refresh tokens in an httpOnly cookie, reuse detection, two-tab grace window
    • Token-bucket rate limiter in Redis (Lua), fail open/closed per policy, local fallback
    • Documents: create, idempotent PUT with client ids, list by scope, rename, trash, restore
    • Roles enforced server-side; outsiders get 404, too-low roles get 403
    • Web: sign-in, registration, silent session restore, dashboard, cross-tab sign-out
    • 67 API tests on real Postgres and Redis; 35 web tests with MSW; checked in a real browser
  • 2 · Live editing, offline core (weeks 3–4)
    • Sync server on pycrdt: one room per open document, Yjs sync protocol, per-connection send queues, awareness relay that stamps the authenticated name on every cursor
    • One-time 30-second WebSocket tickets (Redis GETDEL); close codes the client acts on
    • Viewers and commenters can follow live but their edits are refused server-side
    • Per-connection message budget, 10 connections per user, slow clients dropped
    • Edits saved to an append-only log in Postgres, batched every 500 ms or 50 updates, and flushed on room eviction and on shutdown (pulled forward from phase 3; compaction stays in phase 3)
    • Moving a document to trash disconnects everyone editing it
    • Web: Tiptap editor, live cursors and presence avatars, per-user undo, own provider with fresh tickets per attempt and exponential backoff, IndexedDB offline copy, sync badge, offline copies wiped on sign-out and when access is lost
    • 97 API tests (incl. a 150-case Hypothesis convergence property), 53 web tests; checked in a real browser with two accounts, including killing the API mid-edit
    • Found upstream: pycrdt/yrs 0.27 drops some out-of-order updates (Yjs does not); see ADR 0012
  • 3 · Storage, versions, search (week 5) ✅ 2026-10-02
    • Compaction: the edit log folds into one snapshot at 500 rows and when a session ends, under an advisory lock; deletes only the rows it read; loads read base and log in one REPEATABLE READ snapshot. Opening a 10,000-keystroke document: 218 ms → 17 ms (50,000: 6.5 s → 0.28 s); see ADR 0003
    • Version history: automatic (session end, every 10 minutes), named, and pre-restore versions; newest 50 automatic kept, up to 100 named; editors only
    • Restore is an ordinary edit made by the browser through y-prosemirror: concurrent edits survive and Undo reverts it (ADR 0014)
    • Full-text search over titles and text with Postgres (GIN, weighted, prefix matching on stemmed and unstemmed words), highlighted snippets without HTML (ADR 0015)
    • Web: search box on the dashboard (debounced, kept in the URL); version history dialog with save, name, read-only preview and restore
    • 173 API tests (a 40-schedule compaction property, out-of-order commits, the advisory lock, hostile search input), 79 web tests (restore and Undo on a real editor); checked in a real browser with two tabs
    • Found: pycrdt drops waiting updates when re-encoding a document, so snapshots store merged updates (ADR 0012, pitfall 4)
    • Found and fixed: a connect race left an orphaned socket that kept rooms open forever; the client now drops stale connects, the server closes sockets silent for 150 s, and the heartbeat judges unanswered pings rather than silence (ADR 0012)
  • 4 · Sharing and roles (week 6) → Gate 1: MVP live on the web ✅ 2026-10-01
    • Members: list, invite by email with a role, change role, remove, leave, transfer ownership
    • Share links: role, expiry, turn off; token shown once and stored hashed; accept never lowers access; token sent in the body so it stays out of logs
    • Access changes close open editors instantly (4403 removed, 4409 reconnect with new role)
    • Web: Share dialog, /share/:token page (sign-in first when needed), provider handles 4409
    • Deploy prep: render.yaml, vercel.json, docs/DEPLOY.md; Neon URLs adapted for asyncpg; migrations at container start; production refuses a dev secret or insecure cookies; config errors never echo secrets
    • 121 API tests, 64 web tests; production image rehearsed locally against a fresh database
    • Deployed: web https://collaborative-editor-flax.vercel.app, API https://collabedit-api.onrender.com, Neon Postgres 18 and Redis Cloud (all Singapore)
    • Production smoke test, 31/31: frontend, auth and cookie rotation, documents, sharing, live editing between two accounts (~110 ms), presence, persistence, revocation
    • Found in production: Render's edge drops WebSocket close frames; client heartbeat added
    • Scheduled ping keeps the free API awake 08:00–23:59 IST and lets it sleep overnight
  • 5 · Multi-instance Redis (week 7) ✅ 2026-10-11, built last of 5–7
    • A Redis channel per open document or branch, and a control channel for whole-document events; binary messages with a version byte, so releases can overlap during a deploy
    • The receiving instance saves each edit; the others apply and relay it
    • Opening a room subscribes, then reads Postgres, then runs the Yjs sync handshake with the other instances, so edits not saved yet are never missed
    • Edits and cursors published with a leading-edge throttle (at once, then at most every 50 ms); nothing published while an instance is alone with a document
    • Kicks after access changes, trashing, merges (with a cross-instance freeze) and the comment signal reach every instance; retried if Redis refuses them
    • Own reconnect handling: a dropped subscription re-syncs from Postgres and its peers; /api/readyz reports pubsub; the API starts with Redis down
    • Migrations take a Postgres advisory lock, so instances can start together
    • Docker Compose runs two instances behind nginx (round robin, no sticky sessions); infra/k6/run.sh load-tests them, and CI runs it on every push
    • 248 API tests (17 run two instances: convergence over 5 seeds, a room opened later, a dropped subscription, lost publishes, kicks, merges, presence); real browsers through nginx on two instances: 7/7, including stopping one instance mid-edit (reconnected in 221 ms)
    • Measured (ADR 0004): two instances, 100 editors: p50 1 ms, p95 17 ms; four busy documents, 80 editors: p50 4 ms, p95 48 ms, 80% fewer Redis commands than one per edit; about 60 KB per open document
    • Found: redis-py re-subscribes silently after a reconnect (messages lost unnoticed); a stale receiver count could mark a room alone while a peer listened; a lost reply to HELLO stranded a newcomer; the first migration lock rolled migrations back
  • 6 · Comments (week 8) ✅ 2026-10-03, built before 5
    • Threads anchored to a selection with Yjs relative positions, stored beside the text rather than in it, so commenters can comment without edit rights and a thread stays on its words through local, remote and offline edits (ADR 0017)
    • Replies, edit (author), delete (author, or an editor or owner), resolve and reopen; a reply reopens a resolved thread; deleting the first comment deletes the thread
    • Live: the server sends a one-byte "comments changed" signal over the document's socket and clients refetch; a reconnect refetches whatever was missed
    • Email: a new thread notifies the owner, a reply notifies the thread's participants, never the author and only current members; SMTP (Gmail app password) or log-only (DEPLOY.md)
    • Comments on branches belong to the branch; merged and closed branches are read-only
    • Web: comments panel beside the editor, highlights drawn as decorations, Ctrl+Alt+M, open/resolved lists, notification links open their thread
    • 230 API tests (permissions, moderation, live signals, emails, caps, rate limit), 104 web tests (anchors through local and remote edits; a test fails if local edits are not mapped); checked in two real browsers: 11/11, highlight stays on its words for both
    • Found: ProseMirror applies a local edit before Yjs sees it, so resolving anchors then reads stale text; local edits map the known ranges instead (ADR 0017). A read-only editor takes no keyboard input, so the shortcut listens on the window
  • 7 · Branches and merge requests (weeks 9–10) ✅ 2026-10-02, built before 5 and 6
    • A branch is its own stream: its own edit log, compaction snapshot and live room, keyed by branch id; fork copies main, and its starting state is the merge base
    • Review: diff3 over paragraphs between the merge base, main and the branch; added, removed, changed and conflicting passages, and a preview of main after the merge
    • Merge: refuses conflicts and a branch changed since review; saves main as a "before merging" version; applies the branch through main's live room, so everyone sees it at once
    • Update from main moves the merge base up, the way git merge main does, which is how a conflict is settled; close abandons a branch; merged and closed branches are read-only
    • Commenters branch and propose; owners and editors merge; at most 10 open branches, branch creation rate-limited; losing access or trashing the document disconnects branch editors
    • Web: Branches dialog, branch editor with actions, review page with change cards and preview
    • 199 API tests (a diff3 partition property, conflicts by edit and by deletion, review after main moved on, stale review, permissions), 89 web tests; checked in a real browser: two tabs, a clean merge, then a conflict settled by updating from main
    • Found: FOR UPDATE on the branch row blocked the foreign-key checks of the branch's own edits (KEY SHARE locks), so a merge waited on itself; the row is locked FOR NO KEY UPDATE
  • Also fixed 2026-10-02: search text backfilled for documents edited before search existed; the GitHub keep-awake schedule proved unreliable (two runs in six hours), so an external cron now does that job (docs/DEPLOY.md)
  • Also fixed 2026-10-15: the API took 11 minutes to wake. A migration lock left by a killed process's lingering database session can block startup; startup now skips the lock when the schema is current and waits at most 60 s otherwise (alembic/env.py, tested)
  • 8 · Google sign-in and email (week 11) ✅ 2026-10-22
    • Sign in with Google: authorization code flow with PKCE, state bound to the browser by a cookie, nonce, ID token checked against Google's keys; openid email profile only
    • Accounts matched by Google's subject id; a Google sign-in on an address someone registered but never verified removes that password and its sessions (pre-hijacking)
    • Password reset by email: same answer for unknown addresses, single-use 30-minute links, all sessions end; Google-only accounts can add a password this way
    • Email verification at sign-up, resend, banner; comment emails only to verified addresses
    • Tokens stored as SHA-256 and sent after #, out of server logs; dev logs show emails so links can be followed, production never does
    • Web: Continue with Google, forgot / reset / verify pages, verification banner, and a privacy policy page (Google requires one before the sign-in app can be published)
    • 282 API tests (20 against a fake Google with a real RSA key that enforces PKCE), 117 web tests; browser run against a local fake Google: 12/12 (Google sign-in and cancel, sign-up and verify, reset)
    • Deployed 2026-10-22 with a published Google OAuth client and Gmail SMTP; checked locally against real Google, then on the live site: Google sign-in, a real verification email, and a password reset
    • Found: building an HTTP client reads the whole CA bundle (half a second on Windows), so it is made only when Google is configured
    • Found in deployment: Google will not publish a sign-in app without a home page and a privacy policy link, so the app now has /privacy
  • 9 · Export, offline app, polish (week 12) ✅ 2026-10-29, not yet deployed
    • Stars: personal to each member (document_members.starred_at), a Starred view, a star on every row
    • Editor: checklists, tables, images by web address, links (Ctrl+K), word count and a shortcuts dialog; one extension list shared by the live editor and version previews
    • App shell offline and installable: a Workbox service worker precaches every built file, the editor and Word chunks included; never touches /api; new versions wait for Reload
    • Opens offline signed in as the last person; document lists, summaries and comments are saved on the device; online-only actions are disabled with the reason
    • Offline create, rename, star, comment and reply, queued in an IndexedDB outbox and replayed in order; every request repeat-safe (browser UUIDv7 + PUT, comment client_ids); refusals listed in a banner (ADR 0019)
    • Text edited offline reaches the server on reconnect even if its editor is closed; documents can be kept available offline ahead of time
    • Header badge: offline or sending, with "N changes not synced" and "last online"; signing out warns about unsent changes, then wipes the device
    • Export to Markdown (own converter), Word (docx, loaded on demand) and PDF (print stylesheet), all in the browser and offline (ADR 0020)
    • Dark mode as a choice (system, light, dark) that survives reloads; layouts checked at phone width
    • 289 API tests, 174 web tests; a browser run of the production build against the local API: 31/31 (service worker, offline reload, offline create/edit/comment/star/export, sent exactly once on reconnect, print, phone width, dark mode, sign-out wipe)
    • Found: TanStack Query holds every mutation while the browser reports offline, so nothing reached the outbox; mutations now always run and the outbox decides what waits
    • Found: the first service worker did not control the page that installed it, so a document opened offline before any reload had no editor code; it now claims the page at once
  • 10 · AI assistant (week 13)
  • 11 · Hardening and launch (week 14) → Gate 2: v2 release

If a phase slips, cut in this order: AI, then export, then Google sign-in. Never branches or tests.

Architecture decisions

ADR Decision
0001 Yjs CRDT for merging concurrent edits
0002 PostgreSQL as the system of record
0003 Append-only edit log plus snapshots
0004 Redis pub/sub between API instances, Yjs sync between servers
0005 Token bucket in a Redis Lua script, fail open or closed per policy
0006 One-time WebSocket tickets instead of tokens in URLs
0011 In-memory access tokens, rotating refresh cookie with reuse detection
0012 Own sync server and client provider on pycrdt/Yjs, and the pycrdt pitfalls
0013 Membership-only access, hashed share links, instant revocation
0014 Versions as snapshots; restore as an ordinary edit made by the browser
0015 Full-text search in Postgres, stemmed and as-typed prefixes
0016 Branches as separate streams; diff3 over paragraphs gates CRDT merges
0017 Comments anchored with Yjs relative positions, stored beside the text
0018 Google sign-in with PKCE; hashed single-use reset and verification links
0019 Service worker app shell, an outbox of repeat-safe requests, offline edits sent on reconnect
0020 Markdown, Word and PDF exports built in the browser from the editor's JSON