Skip to content

Migrate remaining HTTP endpoints to Sebuf (proto-first) #3207

Description

@koala73

Context

Our proto-first stack (Sebuf / protoc-gen-openapiv3, see proto/buf.gen.yaml) auto-generates:

  • Typed TS client (src/generated/client)
  • TS server handler stubs (src/generated/server)
  • OpenAPI v3 docs (docs/api/*Service.openapi.yaml) — published via Mintlify

Today the 30 /api/<domain>/v1/<rpc> domain services are fully proto-driven (173 RPCs, 100% coverage). A sizeable set of endpoints bypasses this path — they're hand-rolled edge functions with no proto, no generated client, and no auto-docs. That means:

  • Breaking changes are only caught at runtime.
  • Clients (dashboard, desktop, partners) duplicate type definitions.
  • Docs have to be written and maintained by hand.
  • Drift between producer and consumer is the Batch market and RSS fetching with progressive updates #1 source of "silent empty panel" bugs.

This issue tracks migrating the remaining user-facing endpoints to Sebuf.

Scope

10 endpoints across 6 services. Split into two tracks.

Track A — Add RPCs to existing domain protos (5)

These endpoints already sit inside a domain we own; the work is adding an rpc to the existing .proto and routing the handler.

Endpoint Target proto Suggested RPC
api/eia/[[...path]].js proto/worldmonitor/economic/v1/ GetEiaSeries — replace catch-all proxy with typed series-id request
api/satellites.js new proto/worldmonitor/orbital/v1/ (or fold into natural) ListSatellitePositions
api/military-flights.js proto/worldmonitor/military/v1/ ListMilitaryAircraftLive
api/ais-snapshot.js proto/worldmonitor/maritime/v1/ GetAisSnapshot
api/sanctions-entity-search.js proto/worldmonitor/sanctions/v1/ SearchSdnEntities

Track B — New services (5 new protos, 13 new RPCs)

Endpoint(s) New service RPCs
api/scenario/v1/{run,status,templates}.ts scenario/v1 RunScenario, GetScenarioStatus, ListScenarioTemplates
api/v2/shipping/{route-intelligence,webhooks}.ts shipping/v2 GetRouteIntelligence, RegisterWebhook, ListWebhooks
api/contact.js, api/register-interest.js leads/v1 SubmitContact, RegisterInterest
api/chat-analyst.ts, api/widget-agent.ts, api/skills/fetch-agentskills.ts analyst/v1 ChatAnalyst (streaming), WidgetComplete, ListAgentSkills
api/enrichment/company.js, api/enrichment/signals.js enrichment/v1 GetCompany, GetDomainSignals

Out of scope (intentionally NOT migrating)

  • Internal — not a user-facing contract: api/bootstrap.js, api/geo.js, api/reverse-geocode.js, api/data/city-coords.ts. These are dashboard-internal helpers we don't want external consumers building against; keeping them hand-rolled avoids implicitly publishing them as API surface.
  • External protocol contracts: api/mcp.ts (MCP JSON-RPC), api/oauth/* (OAuth 2.1), api/discord/oauth/*, api/slack/oauth/*.
  • Non-JSON responses: api/brief/carousel/* (PNG), api/og-story.js (PNG), api/story.js (HTML), api/youtube/embed.js (HTML), api/download.js (binary).
  • Ops/admin: api/health.js, api/seed-health.js, api/version.js, api/cache-purge.js, api/seed-contract-probe.ts, api/invalidate-user-api-key-cache.ts, api/mcp-proxy.js, api/fwdstart.js, api/notify.ts.
  • Thin upstream proxies where we don't own the schema: api/opensky.js, api/polymarket.js, api/gpsjam.js, api/oref-alerts.js, api/rss-proxy.js, api/telegram-feed.js, api/supply-chain/hormuz-tracker.js.
  • Deferred (per repo owner): api/latest-brief.ts, api/brief/share-url.ts, api/brief/public/[hash].ts, api/brief/[userId]/[issueDate].ts, api/notification-channels.ts, api/user-prefs.ts, api/create-checkout.ts, api/customer-portal.ts, api/product-catalog.js, api/referral/me.ts.

Suggested order

  1. Track A first — low-ceremony RPC additions to existing protos; exercises the toolchain on familiar ground.
  2. scenario/v1 — already follows the URL convention; OpenAPI was hand-written in docs/api/ScenarioService.openapi.yaml and can be diffed against the generated output.
  3. shipping/v2 — partner-facing API, type safety matters most here.
  4. analyst/v1 — note ChatAnalyst is streaming; verify Sebuf supports SSE or handle streaming manually while using proto for the request shape.
  5. enrichment/v1, leads/v1 — simple request/response, good warm-up PRs.

Per-endpoint migration checklist

For each endpoint:

  • Write / extend .proto in proto/worldmonitor/<domain>/v1/
  • Run make generate to regen src/generated/{client,server} and docs/api/*.openapi.yaml
  • Add handler to server/worldmonitor/<domain>/v1/<rpc>.ts
  • Wire handler into domain handler.ts
  • Replace the old api/<path>.{ts,js} with a thin gateway (or delete if fully subsumed by the /v1/ route)
  • Update callers (dashboard, desktop, SDK docs) to the generated client
  • Add docs/api/<Service>.openapi.yaml entry to docs/docs.json if new service
  • Smoke test: curl old vs new response shapes match
  • Remove the hand-written MDX entry (from api-platform.mdx / api-proxies.mdx / etc.) once the auto-generated spec is live

Acceptance

  • All 10 endpoints migrated.
  • Hand-written OpenAPI for scenario/v1 (and any others) is replaced by auto-generated output.
  • No breaking changes to existing callers.
  • protoc-gen-openapiv3 output in docs/api/ committed and rendered in Mintlify.

Related

  • Doc audit PR (forthcoming) — adds MDX pages for the non-proto endpoints as an interim.
  • proto/buf.gen.yaml — Sebuf config
  • docs/adding-endpoints.mdx — proto-first endpoint guide

Metadata

Metadata

Labels

area: APIBackend API, sidecar, keysdocumentationImprovements or additions to documentationrefactorCode restructuring, architecture

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions