feat: add telemetry for task lifecycle events by johnstcn · Pull Request #21922 · coder/coder

johnstcn · 2026-02-04T14:41:20Z

Adds new database queries and telemetry collection functions to gather task lifecycle events (pause/resume cycles, idle time) for analytics.

Task events track pause/resume activity, idle duration before pausing, paused duration, and time from resume to first app status, filtered to recent activity based on the telemetry snapshot interval.

🤖 Created with Mux (Opus 4.6).

…amp control (#21944) Relates to #21922 / coder/internal#1259 * Adds `dbfake.BuilderOption func(*WorkspaceBuildBuilder)` * Adds `BuilderOption` methods for setting various provisioner job related fields on `WorkspaceBuildBuilder`. * Migrates a number of existing tests that previously dependeded on provisioner job timing to use these updated methods in the following packages: * `coderd/jobreaper` * `coderd/notifications/reports` * `enterprise/coderd/schedule` * `enterprise/coderd/prebuilds` * `scripts/workspace-runtime-audit` 🤖 Created using Mux (Opus 4.5) --------- Co-authored-by: Copilot <[email protected]>

Copilot

Pull request overview

Adds richer telemetry around AI task lifecycle events (pause/resume/idle/first status timing) and wires it into the periodic telemetry snapshot reporter.

Changes:

Introduces CollectTasks to gather task telemetry enriched with lifecycle signals derived from workspace builds and app statuses.
Extends the telemetry Task payload with pause/resume timestamps, pause reason, and duration metrics.
Adds DB support for efficiently fetching task lifecycle workspace builds and expands tests covering many lifecycle scenarios.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
coderd/telemetry/telemetry.go	Adds `CollectTasks`, extends `Task` telemetry schema, and uses it in snapshot creation.
coderd/telemetry/telemetry_test.go	Adds comprehensive task telemetry test cases and fixtures.
coderd/database/queries/workspacebuilds.sql	Adds SQL query to fetch pause/resume-related workspace builds by workspace IDs.
coderd/database/queries.sql.go	sqlc-generated Go for `GetTaskLifecycleBuildsByWorkspaceIDs`.
coderd/database/querier.go	Adds new query method to the `sqlcQuerier` interface.
coderd/database/dbmock/dbmock.go	Adds mock support for `GetTaskLifecycleBuildsByWorkspaceIDs`.
coderd/database/dbmetrics/querymetrics.go	Adds query metrics instrumentation for the new DB method.
coderd/database/dbauthz/dbauthz.go	Adds authz wrapper for `GetTaskLifecycleBuildsByWorkspaceIDs`.
coderd/database/dbauthz/dbauthz_test.go	Adds RBAC expectations test for the new DB method.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

coderagents

Code Review

Reviewed telemetry additions for task lifecycle events. The implementation adds comprehensive tracking for task pause/resume operations with metrics like idle duration, paused duration, and time to active. Found 2 important issues related to time calculations and 1 logic bug in the PausedDurationMS computation.

Summary

🟡 IMPORTANT: Negative duration calculations possible in IdleDurationMS (line ~1058)
🟡 IMPORTANT: PausedDurationMS logic has incorrect loop termination (line ~1083)
🔵 NITPICK: TimeToActiveMS could return first non-working status instead of first working status

Detailed Findings

The database queries look well-structured with appropriate indexes. Authorization patterns follow Coder conventions correctly. Test coverage is comprehensive with good edge case handling.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Adds new database queries and telemetry collection functions to gather task lifecycle events (pause/resume cycles, idle time) for analytics. Task events track pause/resume activity, idle duration before pausing, paused duration, and time from resume to first app status, filtered to recent activity based on the telemetry snapshot interval.

johnstcn · 2026-02-13T18:05:47Z

+		assert.Nil(t, taskEvent.LastResumedAt)
+		assert.Nil(t, taskEvent.LastPausedAt)
+		assert.Nil(t, taskEvent.PauseReason)
+		assert.Nil(t, taskEvent.IdleDurationMS)
+		assert.Nil(t, taskEvent.PausedDurationMS)
+		assert.Nil(t, taskEvent.ResumeToStatusMS)


self-review: adding some task app statuses here may not be a bad idea.

mafredri

Nice work!

Some small suggestions, little cleanup, and consolidating query/code conditionals, but otherwise this is looking good.

mafredri · 2026-02-16T13:30:36Z

+        WHERE was.app_id = twa.workspace_app_id
+          AND was.state = 'working'
+        ORDER BY was.created_at DESC
+        LIMIT 1


Nit: This is something of an approximation. Consider a situation where the agent reports working, goes on to do 1h of work, and then reports done. We'd be missing 1h of work from this timeframe.

We could improve this, but I'm not sure it's worth the lift. But a comment would be good IMO.

Comment added above:

-- - Idle duration approximation: If the agent reports "working", does -- work, then reports "done", we miss that working time.

mafredri · 2026-02-16T13:32:39Z

+        lws.created_at          AS last_working_status_at,
+        -- First app status after resume (for resume-to-status duration).
+        -- Only populated for workspaces in an active phase (started more
+        -- recently than stopped).


Was this in the ticket/RFC? Either way, this seems like a useful metric to add. 👍🏻

Nope, it was just a drive-by idea I mentioned briefly in Slack 👍

mafredri · 2026-02-16T13:47:24Z

+        -- Latest start build.
+        start_build.created_at  AS start_build_created_at,
+        -- Last "working" app status (for idle duration).
+        lws.created_at          AS last_working_status_at,


This was lost in the RFC -> ticket conversion, but we also want a "active duration", i.e. how much time the task spent working.

johnstcn · 2026-02-17T16:20:39Z

+        AND EXISTS (
+            SELECT 1 FROM workspace_builds wb
+            WHERE wb.workspace_id = t.workspace_id
+              AND wb.created_at > @created_after


review: now filtering by created_after

mafredri · 2026-02-18T10:17:34Z

+        FROM workspace_builds wb
+        WHERE wb.workspace_id = t.workspace_id
+            AND wb.transition = 'start'
+            AND wb.reason = 'task_resume'


I think we should, rather than filter on reason here, select the reason in the query and add task_resume/other in the telemetry, similar to stop.

If a user manually starts the task workspace, that's still a task resumption, but not the expected way (valuable in telemetry IMO).

mafredri · 2026-02-18T11:01:58Z

+                was.created_at AS interval_start,
+                COALESCE(
+                    LEAD(was.created_at) OVER (ORDER BY was.created_at ASC),
+                    COALESCE(stop_build.created_at, @now::timestamptz)


I think this might need a rethink.

We actually know which build twa.workspace_app_id belongs to, we just don't select the build number from that table.

Also, a stop build (theoretically) never has was content. So these can only come from the start build (i.e. task is running). For a running task, was.created_at will always be newer than stop_build.created_at. So if we ever hit this condition, we'll get interval_start > interval_end.

By the time the task gets paused, the twa app will have changed and we will not get any data attached.

So we get a pause event, but no indication how long the task was active.

One way we could improve the data and make it more explicit is to select only the start_builds app ID from twa, and use that in our joins. That would always give us the relevant data.

mafredri · 2026-02-18T11:23:55Z

+			Bool:  true,
+			Valid: true,
+		},
+	})


We're missing UpsertTaskWorkspaceApp, which might be why the test isn't capturing what I raised on the query.

(For start, we need to sett the build/agent/app, for stop just the build).

johnstcn · 2026-02-23T23:03:04Z

+task_status_timeline AS (
+	-- All app statusese across every historical app for each task,
+	-- plus synthectic obundary rows at each stop/start build transition.
+	-- Boundaries prevent LEAD() from spanning pause gaps and let us
+	-- ignore post-resume statuses.
+	SELECT tai.task_id, was.created_at, was.state::text AS state
+	FROM workspace_app_statuses was
+	JOIN task_app_ids tai ON tai.workspace_app_id = was.app_id
+	UNION ALL
+	SELECT t.id AS task_id, wb.created_at, '_boundary' AS state
+	FROM tasks t
+	JOIN workspace_builds wb ON wb.workspace_id = t.workspace_id
+	WHERE t.deleted_at IS NULL
+		AND t.workspace_id IS NOT NULL
+		AND wb.build_number > 1
+),


review: extracted this into a separate CTE so we can also use this for last_working_status.

johnstcn · 2026-02-23T23:04:22Z

+			// TODO(cian): We define IdleDurationMS as "the time from the last working status to pause".
+			//     We know that the task has reported working since T-6h and got auto-paused at T-50m.
+			//     We can reasonably assume that it has been 'idle' from when it was stopped (T-30m) to
+			//     its next report at T-25m. This is covered by ResumeToStatusMS.
+			//     But do we consider the time since its last report (T-6h) to its being auto-paused
+			//     as truly "idle"?
+			idleDurationMS:   ptr.Ref(310 * time.Minute.Milliseconds()),
+			activeDurationMS: ptr.Ref((5*time.Hour + 10*time.Minute + 25*time.Minute).Milliseconds()),


Review: AFAICT the test as written is "correct" as per the definitions laid out in the OG issue. I had some doubts about the definition but my brain may have encountered E_INSUFFICIENT_TOKENS.

mafredri

Didn't see any major issues here, nice work on the query and test coverage!

I did call out an edge case that I'd recommend addressing just to avoid any funky data (a test-case to reproduce it might be good to see what happens).

Other than that, minor nits that I don't need to re-review.

mafredri · 2026-02-24T15:33:50Z

+	JOIN workspace_builds wb ON wb.workspace_id = t.workspace_id
+	WHERE t.deleted_at IS NULL
+		AND t.workspace_id IS NOT NULL
+		AND wb.build_number > 1


Pseudo: AND was.created_at <= MAX(wb.created_at)?

This would be relevant for the LEAD / OVER.

It may be easier to filter down in the lws lateral join.

johnstcn self-assigned this Feb 4, 2026

johnstcn changed the title ~~Cj/internal 1259~~ feat: add telemetry for task lifecycle events Feb 4, 2026

johnstcn mentioned this pull request Feb 5, 2026

chore(coderd/database/dbfake): add support for provisioner job timestamp control #21944

Merged

johnstcn force-pushed the cj/internal-1259 branch from 4fda142 to f19ae3d Compare February 9, 2026 11:21

johnstcn marked this pull request as ready for review February 9, 2026 18:24

Copilot AI review requested due to automatic review settings February 9, 2026 18:24

Copilot started reviewing on behalf of johnstcn February 9, 2026 18:25

View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

Comment thread coderd/telemetry/telemetry_test.go Resolved

Comment thread coderd/telemetry/telemetry.go Resolved

Comment thread coderd/telemetry/telemetry.go Resolved

Comment thread coderd/telemetry/telemetry.go Resolved

Comment thread coderd/telemetry/telemetry.go Resolved

johnstcn added the code-review label Feb 11, 2026

coderagents Bot reviewed Feb 11, 2026

View reviewed changes

Comment thread coderd/telemetry/telemetry.go Resolved

Comment thread coderd/telemetry/telemetry.go Resolved

Comment thread coderd/telemetry/telemetry.go Resolved

johnstcn marked this pull request as draft February 11, 2026 17:44

johnstcn commented Feb 11, 2026

View reviewed changes

Comment thread coderd/database/queries/workspacebuilds.sql Resolved

johnstcn force-pushed the cj/internal-1259 branch 3 times, most recently from 525c7b4 to c87ab0b Compare February 12, 2026 00:11

coder deleted a comment from chatgpt-codex-connector Bot Feb 12, 2026

johnstcn requested a review from Copilot February 12, 2026 09:19

Copilot started reviewing on behalf of johnstcn February 12, 2026 09:19

View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

johnstcn force-pushed the cj/internal-1259 branch from df7d21d to fbb7384 Compare February 12, 2026 10:30

johnstcn force-pushed the cj/internal-1259 branch from fbb7384 to 9b2eb67 Compare February 12, 2026 14:15

Merge branch 'main' into cj/internal-1259

a8a5e31

johnstcn marked this pull request as ready for review February 12, 2026 16:06

johnstcn requested a review from mafredri February 12, 2026 16:06

johnstcn commented Feb 13, 2026

View reviewed changes

Comment thread coderd/telemetry/telemetry.go Resolved

johnstcn commented Feb 13, 2026

View reviewed changes

mafredri reviewed Feb 16, 2026

View reviewed changes

address PR comments

b3371cb

johnstcn commented Feb 17, 2026

View reviewed changes

mafredri reviewed Feb 18, 2026

View reviewed changes

johnstcn added 9 commits February 18, 2026 17:13

fix spurious interface{}

9af4c29

drive-by for-range fix

1f5eb61

resume reason

1d3afb6

fixup! resume reason

b01ff63

make the tests fail in the right way now that we upsert workspace apps

2fbba40

make tests fail better

fec7feb

fix ActiveDurationMS calculation

febaac7

improve calculation of IdleDurationMS

0fbf97f

fixup! improve calculation of IdleDurationMS

5e477ee

johnstcn commented Feb 23, 2026

View reviewed changes

mafredri approved these changes Feb 24, 2026

View reviewed changes

johnstcn added 5 commits February 24, 2026 16:14

NullBuildReason

47dfeef

comment fixes

e1cf392

RED: test for non-deterministic ordering

16fa000

GREEN: test passing with additional ordering

72adb6c

tabs

094d994

johnstcn merged commit 6336fee into main Feb 24, 2026
27 checks passed

johnstcn deleted the cj/internal-1259 branch February 24, 2026 17:04

github-actions Bot locked and limited conversation to collaborators Feb 24, 2026

Conversation

johnstcn commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderagents Bot left a comment

Choose a reason for hiding this comment

Code Review

Summary

Detailed Findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mafredri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mafredri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johnstcn commented Feb 4, 2026 •

edited

Loading