Skip to content

[Tracking] CI and test performance — memory, runtime, developer experience #57266

Description

@odysseus0

The CI and test suite have recurring performance issues that block development velocity and make contribution harder.

Test worker memory

  • `prepare-pr` uses `unit-fast` config with long-lived workers that accumulate 4GB+ per process over their lifetime. Kills 16GB machines. GC pressure causes crawling or crashes.
  • Workaround: splitting into 2x chunks based on time estimate (dropped from 10s to 5s per chunk estimate). Helped substantially but not a root fix.

CI reliability

  • CI regularly red for multiple days. Multiple maintainers spend hours daily shepherding it green.
  • Large refactors frequently break CI — acknowledged pattern.
  • Barrel imports in extension tests were causing timeout failures (recently fixed).
  • `pnpm check` had become a "giant kitchen sink" (recently cleaned up).
  • `checks-fast-extensions` was chronically red (recently stabilized).

Developer experience

  • Running the full test suite locally requires high-end hardware (Mac Studio class).
  • No guidance on which test configs to use for different development scenarios.
  • Missing AGENTS.md in key directories means coding agents (Codex, Claude Code) cross boundaries and create spaghetti — boundary docs proposed in Docs: add boundary AGENTS guides #56647.

What's working

  • Release automation (server-triggered releases) reduces human-in-loop failure modes.
  • Multiple maintainers actively fixing CI.
  • Test sharding improvements reducing per-chunk time.

Suggested workstreams

  • Test worker memory: profiling and lifecycle optimization for long-lived workers
  • AGENTS.md boundary docs across key directories (Docs: add boundary AGENTS guides #56647)
  • CI reliability dashboard or automated green-tracking
  • Developer experience docs: which configs for which scenarios, minimum hardware requirements
  • Extension test isolation: prevent barrel imports and cross-boundary dependencies from causing timeouts

This is a tracking issue — feel free to pick up any workstream or propose new ones.

Metadata

Metadata

Assignees

No one assigned

    Labels

    maintainerMaintainer-authored PR

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions