fix(core): propagate continuous task failures to dependent tasks (#33492)

leosvelperez · FrozenPandaz · commit c3343498bec5 · 2025-11-25T18:45:04.000-05:00
## Current Behavior

When a continuous task depends on another continuous task and the
dependent task exits with an error, the parent task continues running
indefinitely. The task execution never terminates, leaving processes
running in the background.

For example, if task `a` (continuous) depends on task `b` (continuous),
and task `b` exits with error code 1, task `a` will continue running
even though its dependency failed.

## Expected Behavior

When a continuous task exits (with any exit code), the failure should be
propagated to dependent tasks:

1. The failed continuous task should be marked as failed
2. Dependent continuous tasks should be marked as skipped
3. All affected continuous tasks should be killed
4. Task execution should terminate with an error

---

## Changes Made

### 1. Restore Error Handling in Continuous Task Exit Handlers

- Re-added the `cleaningUp` flag that was removed in a previous
TUI-related commit
- Modified `onExit` handlers for both regular and shared continuous
tasks to:
  - Check if the task exited during normal cleanup vs. unexpectedly
  - Call `complete()` with 'failure' status for unexpected exits
  - Log error messages for debugging

### 2. Fix `cleanUpUnneededContinuousTasks()` Logic

The previous implementation always added `initializingTaskIds` to the
needed set, even when those tasks were already completed. This prevented
dependency tasks from being killed when the top-level task exited.

Fixed by:

- Only adding tasks from `initializingTaskIds` if they are still
incomplete
- Keeping dependencies of incomplete tasks alive
- This ensures continuous tasks are killed when no longer needed,
whether a dependency fails or a top-level task exits

### 3. Prevent Status Overwrites

Added a check in `onExit` handlers to only set status to `Stopped` if
the task hasn't already been completed. This prevents the async `onExit`
callback from overwriting the correct status (like 'skipped' or
'failure') with 'Stopped'.

### 4. Fix Signal Handling in `PseudoTtyProcess.kill()`

The Rust pseudo-terminal defaults to SIGINT when no signal is provided,
which does not reliably terminate child processes in PTY sessions.

Fixed by:

- Defaulting to SIGTERM in the JavaScript wrapper (rather than changing
the Rust default)
- The JS wrapper is the API boundary that should match Node.js
semantics, where `childProcess.kill(undefined)` defaults to SIGTERM
- The Rust default of SIGINT is appropriate for interactive use
(Ctrl-C), while programmatic cleanup needs SIGTERM
- This ensures child processes are properly killed when
`runningTask.kill()` is called

**File:** `packages/nx/src/tasks-runner/pseudo-terminal.ts`

## Technical Details

The fix leverages the existing task failure propagation mechanism in
`complete()` instead of using `process.exit(1)`, which:

- Allows proper cleanup through normal execution flow
- Respects the `--bail` flag configuration
- Works correctly with both TUI and non-TUI modes
- Maintains consistency with how other task failures are handled
diff --git a/packages/nx/src/tasks-runner/pseudo-terminal.ts b/packages/nx/src/tasks-runner/pseudo-terminal.ts
@@ -191,7 +191,7 @@ export class PseudoTtyProcess implements RunningTask {
   kill(s?: NodeJS.Signals): void {
     if (this.isAlive) {
       try {
-        this.childProcess.kill(s);
+        this.childProcess.kill(s || 'SIGTERM');
       } catch {
         // when the child process completes before we explicitly call kill, this will throw
         // do nothing
diff --git a/packages/nx/src/tasks-runner/task-orchestrator.ts b/packages/nx/src/tasks-runner/task-orchestrator.ts
@@ -85,6 +85,7 @@ export class TaskOrchestrator {
   private groups = [];
 
   private bailed = false;
+  private cleaningUp = false;
 
   private runningContinuousTasks = new Map<string, RunningTask>();
   private runningRunCommandsTasks = new Map<string, RunningTask>();
@@ -732,14 +733,22 @@ export class TaskOrchestrator {
       );
 
       this.runningContinuousTasks.set(task.id, runningTask);
-      runningTask.onExit(() => {
-        if (this.tuiEnabled) {
+      runningTask.onExit((code) => {
+        if (this.tuiEnabled && !this.completedTasks[task.id]) {
           this.options.lifeCycle.setTaskStatus(
             task.id,
             NativeTaskStatus.Stopped
           );
         }
         this.runningContinuousTasks.delete(task.id);
+
+        // we're not cleaning up, so this is an unexpected exit, fail the task
+        if (!this.cleaningUp) {
+          console.error(
+            `Task "${task.id}" is continuous but exited with code ${code}`
+          );
+          this.complete([{ taskId: task.id, status: 'failure' }]);
+        }
       });
 
       // task is already running by another process, we schedule the next tasks
@@ -802,13 +811,22 @@ export class TaskOrchestrator {
     this.runningTasksService.addRunningTask(task.id);
     this.runningContinuousTasks.set(task.id, childProcess);
 
-    childProcess.onExit(() => {
-      if (this.tuiEnabled) {
+    childProcess.onExit((code) => {
+      // Only set status to Stopped if task hasn't been completed yet
+      if (this.tuiEnabled && !this.completedTasks[task.id]) {
         this.options.lifeCycle.setTaskStatus(task.id, NativeTaskStatus.Stopped);
       }
       if (this.runningContinuousTasks.delete(task.id)) {
         this.runningTasksService.removeRunningTask(task.id);
       }
+
+      // we're not cleaning up, so this is an unexpected exit, fail the task
+      if (!this.cleaningUp) {
+        console.error(
+          `Task "${task.id}" is continuous but exited with code ${code}`
+        );
+        this.complete([{ taskId: task.id, status: 'failure' }]);
+      }
     });
     await this.scheduleNextTasksAndReleaseThreads();
     if (this.initializingTaskIds.has(task.id)) {
@@ -1025,6 +1043,7 @@ export class TaskOrchestrator {
   // endregion utils
 
   private async cleanup() {
+    this.cleaningUp = true;
     this.forkedProcessTaskRunner.cleanup();
     await Promise.all([
       ...Array.from(this.runningContinuousTasks).map(async ([taskId, t]) => {
@@ -1054,8 +1073,13 @@ export class TaskOrchestrator {
 
   private cleanUpUnneededContinuousTasks() {
     const incompleteTasks = this.tasksSchedule.getIncompleteTasks();
-    const neededContinuousTasks = new Set(this.initializingTaskIds);
+    const neededContinuousTasks = new Set<string>();
     for (const task of incompleteTasks) {
+      // Keep initiating tasks that are still incomplete
+      if (task.continuous && this.initializingTaskIds.has(task.id)) {
+        neededContinuousTasks.add(task.id);
+      }
+
       const continuousDependencies =
         this.taskGraph.continuousDependencies[task.id];
       for (const continuousDependency of continuousDependencies) {
