Create load generator for Task App Statuses #913

From the plan:

Goal:

• To observe how Coder handles Task usage at scale.

Infrastructure:

• Kubernetes cluster per region, each cluster has:
  • 1/3 of total workspaces.
  • 1/3 of total provisioners.
  • 1 workspace proxy in secondary regions (those w/o the coder deployment).

Implementation:

• Tools:
  • scaletest create-workspaces
  • Custom tool to create fake AI agents in each workspace (i.e. scaletest fake-ai ).
    • These fake AI agents should make tool calls to the Coder MCP server running in the workspace, which then talk to coderd .
• Build concurrency equal to provisioners, to minimize build load.

Run Sequence:

1. Create users.
2. Create workspaces with a Coder MCP server, and a fake AI agent.
3. Have each user client connect to the /api/v2/workspaces/{workspace}/watch-ws endpoint, and listen for status updates.
4. Establish baseline (10 minutes)
5. Have fake AI agents report app status updates every 10 seconds.
6. Run for 10-15 minutes.

Key Results:

• Latency from app status update to update receipt by client.
• DB utilization/load during update fan out.
• If any status updates were missed altogether.

Other Results:

• Coder deployment + database resource utilization (incl. pprofs, Grafana Pyroscope).