WORKER-DEPLOYERS

▶ [patch] #8314
Azure double-checks if vm is gone by calling virtualMachines.get after instanceView returns 404.
This is to prevent situations when worker is being removed based on one failed instanceView call.

▶ [patch]
Adds Weight Distribution Playground for Launch Configurations in documentation
to see how initialWeight value change affect distribution

USERS

▶ [patch] #8311
Fixes UI issue for LaunchConfigs not showing location when ARM deployment template was used

Automated Package Updates

DEPLOYERS

▶ [patch] #7472
The queue service's workerRemovedResolver now also listens for workerStopped events from worker-manager, in addition to workerRemoved events. This ensures claimed tasks are resolved as exception/worker-shutdown as early as possible when a worker disappears. Both events are handled idempotently, so receiving both for the same worker is safe.

DEPLOYERS

▶ [minor] #7472
The queue service now listens for workerRemoved events from worker-manager and immediately resolves any tasks claimed by that worker as exception/worker-shutdown, triggering an automatic retry.
Previously, when a worker disappeared (due to VM preemption, crash, or manual termination), its claimed tasks would wait up to 20 minutes for the claim to expire before being retried.
This new workerRemovedResolver background process runs alongside the existing claim-resolver and requires no configuration changes.

USERS

▶ [patch] #8300
Task log profiler optimizations for parsing large task logs, more memory and cpu efficient.

Automated Package Updates

WORKER-DEPLOYERS

▶ [patch] #8083
Generic Worker (windows): waits for the User Profile Service (ProfSvc) to be running before performing profile operations on first boot, and fixes a bug where LoadUserProfile retry logic for "device not ready" errors never actually retried due to an incorrect error type assertion.

DEPLOYERS

▶ [patch] #8292
Claim-resolver and deadline-resolver expose metrics.

GENERAL

▶ [patch]
Upgrades to go1.26.0

Release notes here.

WORKER-DEPLOYERS

▶ [minor] #7147
Worker-manager now decides which workers should be kept during the worker-scanner loop, surfaced via the shouldWorkerTerminate API.
Decision is being made based on several policies:

• if launch config is archived - worker would be marked as "shouldTerminate=true"
• if workers exceed the desired capacity (more workers than pending tasks) - the oldest workers would be marked as "shouldTerminate=true" (newest workers are kept)

▶ [patch] #8289
Generic Worker: fixes credential expiration during high-volume artifact uploads by narrowing the scope of the queue client lock so that credential refresh is no longer blocked by in-flight HTTP calls.

▶ [patch] #8291
Generic Worker: handles SIGTERM during task execution by triggering graceful termination, ensuring preempted tasks are properly resolved as exception/worker-shutdown instead of exception/claim-expired.

USERS

▶ [minor] #8035
The index service no longer adds a bewit (time-limited auth token) to redirect URLs for public artifacts. Artifacts are considered public if the anonymous role has the necessary scopes to get them. The index service caches the scopes associated to the anonymous role and refreshes them from the auth service every 5 minutes. Additionally, for public artifacts, the index service now resolves the final artifact URL server-side by calling the queue's latestArtifact endpoint, reducing the redirect chain from two hops (Index → Queue → storage) to one (Index → storage).

▶ [patch] #8070
Show a warning banner in the UI when live updates are disabled due to missing web:read-pulse scope, instead of silently showing stale data.

DEVELOPERS

▶ [patch] #8292
Add missing queue_exception_tasks metric to the claim-resolver and deadline-resolver.

▶ [patch] #8261
Fix webpack-dev-server crash (ECONNRESET) when proxying WebSocket /subscription endpoint during local UI development.

OTHER

▶ Additional change not described here: #8271.

Automated Package Updates

USERS

▶ [minor] #7374
Add task profiler REST API endpoints:

• GET /api/web-server/v1/task-group/<taskGroupId>/profile — returns Firefox Profiler JSON for a task group timeline
• GET /api/web-server/v1/task/<taskId>/profile — returns Firefox Profiler JSON for a task's log

Profiles are shareable via profiler.firefox.com/from-url/<encoded-url>. External tools like Treeherder can link directly.

UI adds "Open in Profiler" speed-dial actions on task group and task views, and a dedicated /task/groups/:id/profiler route.

No changes

v96.3.0 had issues uploading GitHub artifacts due to a GitHub incident, so this release is just retrying that one.

GENERAL

▶ [patch] #8104
Fixed task duration not updating in real-time when filtering by status. When using react-window for virtualized task lists, filtering caused different tasks to appear at the same index without triggering a re-render due to React.memo. This is fixed by using react-window's itemKey prop to ensure proper component lifecycle when the task at a given index changes.

▶ [patch] #8231
Worker lifecycle log events (worker-running, worker-stopped, worker-removed) now include duration fields (registrationDuration, workerAge, runningDuration) to aid in investigating worker registration and lifetime issues.

DEPLOYERS

▶ [patch]
Fix running the local dev environment and pods from the helm charts in environments not letting non-root users listen on :80

WORKER-DEPLOYERS

▶ [minor] #8232
Worker Runner now includes system boot time when registering a new worker with Worker Manager. Note: you need to have services running on v96.3.0 or higher before rolling out this change to workers, otherwise, they'll fail to register since the Worker Manager API contract has changed.

Worker Manager uses this to report two new metrics: workerProvisionDuration (time from worker requested to system boot) and workerStartupDuration (time from system boot to registration).

Automated Package Updates

GENERAL

▶ [patch]
Bumps uv to v0.10.0 and taskgraph decision task image to v18.1.0.

WORKER-DEPLOYERS

▶ [patch] #8247
Fix azure checkWorker() call during provisioning.

USERS

▶ [patch] bug 2015057
Fix typo in object service artifact upload.

OTHER

▶ Additional change not described here: bug 2015056.