Summary

Worker pool improvements to detect and recover from crashed or unresponsive worker processes.

Worker crash detection

• Workers track their active task in a DictProxy (_wpool:{pid}) which survives result queue corruption from crashed peers
• The listener health check (every ~1s) detects dead workers via is_alive() and emits worker_died signals for orphaned task cleanup
• Poisoned result queue detection: if no results arrive for 3s while workers have DictProxy entries, all stuck workers are terminated and restarted
• Per-worker timeout: individual workers running tasks for >30s are terminated (catches cases where one worker crashes silently while others keep the global timer fresh)

Early cancel check

• Workers check the cancel flag before running user_func, skipping already-cancelled tasks immediately. This prevents stale cancelled tasks from blocking the queue and starving current-generation tasks.

Infrastructure

• Workers send a "running" message with PID on task start, enabling _worker_start_time tracking
• _spawn_replacement_worker() / _terminate_stuck_worker() for worker lifecycle management
• worker_died signal connected to TaskManager for orphan finalization
• _pid_to_worker / _worker_task_map tracking for health checks

Files

• rayforge/shared/tasker/pool.py — health checks, stuck worker termination, early cancel, running message
• rayforge/shared/tasker/manager.py — _on_pool_worker_died handler
• tests/tasker/test_pool.py — updated cancel test to accept pre-emptive skip