Conversation
When TriggersFired fails and the subsequent ReleaseAcquiredTrigger also fails (e.g. due to a transient DB error), the exception escapes unhandled and the trigger remains in ACQUIRED state permanently. No runtime recovery mechanism exists — only RecoverJobs on startup handles ACQUIRED triggers. Fix both the immediate cause and the missing safety net: 1. Wrap all ReleaseAcquiredTrigger calls in QuartzSchedulerThread with individual try/catch blocks so one failed release doesn't block other releases or the scheduler thread. 2. Add RecoverStaleAcquiredTriggers in JobStoreSupport that periodically detects and recovers triggers stuck in ACQUIRED state longer than 2x MisfireThreshold (minimum 2 minutes). Integrated into the existing MisfireHandler cycle via DoRecoverMisfires. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Extract duplicate stale threshold calculation into StaleAcquiredTriggerThreshold property (DRY) - Remove incorrect BLOCKED state handling from fired trigger recovery (BLOCKED appears in TRIGGERS table, not FIRED_TRIGGERS table) - Remove redundant per-trigger log, keep only summary log - Add 4 unit tests for RecoverStaleAcquiredTriggers covering: stale ACQUIRED recovery, mixed record filtering, recent trigger preservation, and EXECUTING trigger exclusion Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
There was a problem hiding this comment.
Pull request overview
This PR addresses an ADO job store failure mode where triggers can remain permanently stuck in ACQUIRED state (and never fire) when TriggersFired fails and subsequent ReleaseAcquiredTrigger calls also fail (e.g., transient DB errors). It hardens the scheduler thread against release failures and adds a periodic recovery path during the existing misfire handling cycle.
Changes:
- Wraps
ReleaseAcquiredTriggercalls inQuartzSchedulerThreadwith per-trigger try/catch to prevent scheduler thread disruption when releases fail. - Adds
RecoverStaleAcquiredTriggers(and a pre-check) to detect and recover triggers stuck inACQUIREDbeyond a computed threshold, integrated intoDoRecoverMisfires. - Adds unit tests validating stale-acquired trigger recovery behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/Quartz/Impl/AdoJobStore/JobStoreSupport.cs |
Introduces stale-ACQUIRED detection/recovery logic and integrates it into the misfire recovery cycle. |
src/Quartz/Core/QuartzSchedulerThread.cs |
Prevents ReleaseAcquiredTrigger failures from escaping and stalling the scheduler loop. |
src/Quartz.Tests.Unit/Impl/AdoJobStore/JobStoreSupportTest.cs |
Adds unit coverage for recovering stale acquired trigger records. |
| { | ||
| try | ||
| { | ||
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); |
There was a problem hiding this comment.
RecoverStaleAcquiredTriggers currently only updates TRIGGERS from ACQUIRED→WAITING before deleting the fired trigger record. ReleaseAcquiredTrigger updates from both ACQUIRED and BLOCKED (to cover cases where the trigger state has transitioned) before deleting the FIRED_TRIGGERS row. To avoid leaving a trigger stuck if it is in BLOCKED state, mirror ReleaseAcquiredTrigger by also calling UpdateTriggerStateFromOtherState(..., StateWaiting, StateBlocked) before DeleteFiredTrigger (or otherwise ensure BLOCKED is handled).
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false); |
| /// <summary> | ||
| /// Lightweight check (no lock required) to see if there are any fired trigger records | ||
| /// in ACQUIRED state that have exceeded the stale threshold. | ||
| /// </summary> | ||
| private async Task<bool> HasStaleAcquiredTriggers( | ||
| ConnectionAndTransactionHolder conn, | ||
| CancellationToken cancellationToken = default) | ||
| { | ||
| DateTimeOffset staleCutoff = SystemTime.UtcNow() - StaleAcquiredTriggerThreshold; | ||
|
|
||
| IReadOnlyCollection<FiredTriggerRecord> firedTriggers = await Delegate.SelectInstancesFiredTriggerRecords(conn, InstanceId, cancellationToken).ConfigureAwait(false); | ||
|
|
There was a problem hiding this comment.
HasStaleAcquiredTriggers is described as a “lightweight check”, but it calls SelectInstancesFiredTriggerRecords and iterates all fired trigger records for the instance, which can be sizable under load (lots of EXECUTING records). It also duplicates the same query/scan done again in RecoverStaleAcquiredTriggers after the lock is obtained (two full reads when stale triggers are present). Consider either adjusting the wording, or refactoring to avoid the extra full-table read (e.g., perform the scan only once under lock, or add a cheaper delegate query if feasible).
Mirror ReleaseAcquiredTrigger behavior: also update from BLOCKED→WAITING before deleting the fired trigger record. TriggersFired may have moved the trigger to BLOCKED state (for DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. Also clarify HasStaleAcquiredTriggers doc comment about the intentional double-query pattern (pre-check without lock, recovery under lock). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
| int recoveredCount = 0; | ||
| foreach (FiredTriggerRecord rec in firedTriggers) | ||
| { | ||
| if (rec.FireInstanceState == StateAcquired && rec.FireTimestamp < staleCutoff) |
There was a problem hiding this comment.
RecoverStaleAcquiredTriggers determines staleness using rec.FireTimestamp (acquisition time). Since Quartz acquires triggers up to now + IdleWaitTime (and IdleWaitTime is configurable via quartz.scheduler.idleWaitTime), a trigger can legitimately remain in ACQUIRED state longer than the 2-minute floor while the scheduler thread is simply waiting for its scheduled fire time. This can cause premature recovery (flip back to WAITING + delete fired record) and trigger acquisition thrash. Consider basing the cutoff on rec.ScheduleTimestamp (scheduled fire time) instead of acquisition time, or otherwise incorporating the scheduler's acquisition window (IdleWaitTime/batch window) into the threshold.
| if (rec.FireInstanceState == StateAcquired && rec.FireTimestamp < staleCutoff) | |
| DateTimeOffset acquiredStateStalenessTimestamp = rec.ScheduleTimestamp > rec.FireTimestamp | |
| ? rec.ScheduleTimestamp | |
| : rec.FireTimestamp; | |
| if (rec.FireInstanceState == StateAcquired && acquiredStateStalenessTimestamp < staleCutoff) |
|
|
||
| foreach (FiredTriggerRecord rec in firedTriggers) | ||
| { | ||
| if (rec.FireInstanceState == StateAcquired && rec.FireTimestamp < staleCutoff) |
There was a problem hiding this comment.
HasStaleAcquiredTriggers uses rec.FireTimestamp for the stale cutoff as well, which has the same false-positive risk when IdleWaitTime is configured > 2 minutes (or otherwise exceeds the staleness threshold). Align this pre-check with the recovery logic you intend (e.g., compare against rec.ScheduleTimestamp / scheduled fire time).
| if (rec.FireInstanceState == StateAcquired && rec.FireTimestamp < staleCutoff) | |
| if (rec.FireInstanceState == StateAcquired && rec.ScheduleTimestamp < staleCutoff) |
| try | ||
| { | ||
| // Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED, | ||
| // because TriggersFired may have moved the trigger to BLOCKED state (for | ||
| // DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. | ||
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | ||
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false); | ||
| await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false); | ||
| recoveredCount++; | ||
| } | ||
| catch (Exception e) | ||
| { | ||
| Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e); | ||
| } |
There was a problem hiding this comment.
RecoverStaleAcquiredTriggers catches and logs exceptions per record while running inside the surrounding DoRecoverMisfires transaction. If one of the steps succeeds (e.g., state update) but a later step fails (e.g., DeleteFiredTrigger), the transaction can still be committed, leaving an inconsistent combination like trigger state WAITING with a lingering ACQUIRED fired-trigger row. Consider letting the exception propagate (so the outer logic rolls back and retries later) or otherwise ensuring the recovery of a record is atomic (all-or-nothing) within the transaction.
| try | |
| { | |
| // Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED, | |
| // because TriggersFired may have moved the trigger to BLOCKED state (for | |
| // DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false); | |
| await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false); | |
| recoveredCount++; | |
| } | |
| catch (Exception e) | |
| { | |
| Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e); | |
| } | |
| // Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED, | |
| // because TriggersFired may have moved the trigger to BLOCKED state (for | |
| // DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false); | |
| await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false); | |
| recoveredCount++; |
Use Max(ScheduleTimestamp, FireTimestamp) instead of just FireTimestamp when determining if an ACQUIRED fired trigger record is stale. This prevents premature recovery when IdleWaitTime is configured larger than the stale threshold — triggers legitimately remain in ACQUIRED state until their scheduled fire time arrives. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
| try | ||
| { | ||
| // Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED, | ||
| // because TriggersFired may have moved the trigger to BLOCKED state (for | ||
| // DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. | ||
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | ||
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false); | ||
| await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false); | ||
| recoveredCount++; | ||
| } | ||
| catch (Exception e) | ||
| { | ||
| Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e); |
There was a problem hiding this comment.
FiredTriggerRecord.TriggerKey and FireInstanceId are nullable, but this recovery path dereferences them via rec.TriggerKey! / rec.FireInstanceId!. If a delegate ever returns a partial/invalid record (or data is corrupted), this will throw and prevent recovery of other stale records. Consider adding an explicit null check (and skipping/logging the bad record) instead of relying on null-forgiving here.
| try | |
| { | |
| // Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED, | |
| // because TriggersFired may have moved the trigger to BLOCKED state (for | |
| // DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false); | |
| await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false); | |
| recoveredCount++; | |
| } | |
| catch (Exception e) | |
| { | |
| Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e); | |
| if (rec.TriggerKey is null || rec.FireInstanceId is null) | |
| { | |
| Log.Warn($"Skipping stale acquired trigger recovery for invalid fired trigger record. TriggerKey: '{rec.TriggerKey}', FireInstanceId: '{rec.FireInstanceId}'."); | |
| continue; | |
| } | |
| TriggerKey triggerKey = rec.TriggerKey; | |
| string fireInstanceId = rec.FireInstanceId; | |
| try | |
| { | |
| // Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED, | |
| // because TriggersFired may have moved the trigger to BLOCKED state (for | |
| // DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, triggerKey, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, triggerKey, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false); | |
| await Delegate.DeleteFiredTrigger(conn, fireInstanceId, cancellationToken).ConfigureAwait(false); | |
| recoveredCount++; | |
| } | |
| catch (Exception e) | |
| { | |
| Log.ErrorException($"Error recovering stale acquired trigger '{triggerKey}'.", e); |
| try | ||
| { | ||
| // Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED, | ||
| // because TriggersFired may have moved the trigger to BLOCKED state (for | ||
| // DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. | ||
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | ||
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false); | ||
| await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false); | ||
| recoveredCount++; | ||
| } | ||
| catch (Exception e) | ||
| { | ||
| Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e); | ||
| } |
There was a problem hiding this comment.
Catching and swallowing exceptions inside RecoverStaleAcquiredTriggers can lead to committing partial updates (e.g., trigger moved back to WAITING but fired-trigger row not deleted due to a transient error). That can leave inconsistent data and still require later cleanup. Consider letting exceptions propagate (so the outer DoRecoverMisfires rolls back and the misfire handler retries), or otherwise ensuring the state updates + fired-trigger delete happen atomically per record.
| try | |
| { | |
| // Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED, | |
| // because TriggersFired may have moved the trigger to BLOCKED state (for | |
| // DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false); | |
| await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false); | |
| recoveredCount++; | |
| } | |
| catch (Exception e) | |
| { | |
| Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e); | |
| } | |
| // Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED, | |
| // because TriggersFired may have moved the trigger to BLOCKED state (for | |
| // DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false); | |
| await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false); | |
| await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false); | |
| recoveredCount++; |
| { | ||
| Log.DebugFormat("Found {0} triggers that missed their scheduled fire-time.", misfireCount); | ||
| } | ||
|
|
||
| if (misfireCount > 0) | ||
| { | ||
| transOwner = await LockHandler.ObtainLock(requestorId, conn, LockTriggerAccess, cancellationToken).ConfigureAwait(false); |
There was a problem hiding this comment.
When there are no misfires (misfireCount == 0), HasStaleAcquiredTriggers still loads all fired-trigger records for the instance every misfire-handler cycle. On busy schedulers this can be a sizable result set and adds steady-state DB load even when nothing is stale. Consider throttling this check (e.g., only every N cycles / time window) or otherwise reducing the query frequency/size so idle systems don’t pay this cost each scan.
| A.CallTo(() => driverDelegate.UpdateTriggerStateFromOtherState( | ||
| A<ConnectionAndTransactionHolder>.Ignored, | ||
| triggerKey, | ||
| AdoConstants.StateWaiting, | ||
| AdoConstants.StateAcquired, | ||
| A<CancellationToken>.Ignored)).MustHaveHappenedOnceExactly(); | ||
|
|
||
| A.CallTo(() => driverDelegate.DeleteFiredTrigger( | ||
| A<ConnectionAndTransactionHolder>.Ignored, | ||
| "entry_stale_1", | ||
| A<CancellationToken>.Ignored)).MustHaveHappenedOnceExactly(); |
There was a problem hiding this comment.
This test validates the ACQUIRED→WAITING transition, but the production code also updates from BLOCKED→WAITING to mirror ReleaseAcquiredTrigger. Adding an assertion that UpdateTriggerStateFromOtherState(..., StateWaiting, StateBlocked, ...) was called would better lock in the intended behavior.
| A.CallTo(() => driverDelegate.UpdateTriggerStateFromOtherState( | ||
| A<ConnectionAndTransactionHolder>.Ignored, | ||
| staleTrigger, | ||
| AdoConstants.StateWaiting, | ||
| AdoConstants.StateAcquired, | ||
| A<CancellationToken>.Ignored)).MustHaveHappenedOnceExactly(); | ||
|
|
||
| A.CallTo(() => driverDelegate.DeleteFiredTrigger( | ||
| A<ConnectionAndTransactionHolder>.Ignored, | ||
| "entry_stale", | ||
| A<CancellationToken>.Ignored)).MustHaveHappenedOnceExactly(); |
There was a problem hiding this comment.
Similar to the earlier test, this one asserts the ACQUIRED→WAITING update but doesn’t assert the BLOCKED→WAITING update that the implementation performs. Adding that assertion would ensure the recovery logic continues to mirror ReleaseAcquiredTrigger for DisallowConcurrentExecution scenarios.
Add explicit null check for TriggerKey/FireInstanceId before recovery to handle corrupted data gracefully instead of relying on null-forgiving operators. Add BLOCKED→WAITING assertions to stale recovery tests to verify the full ReleaseAcquiredTrigger mirroring behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
| try | ||
| { | ||
| await qsRsrcs.JobStore.ReleaseAcquiredTrigger(t, CancellationToken.None).ConfigureAwait(false); | ||
| } | ||
| catch (Exception releaseEx) |
There was a problem hiding this comment.
There are now several identical try/catch wrappers around JobStore.ReleaseAcquiredTrigger(...) in this file. Consider extracting a small helper (e.g., SafeReleaseAcquiredTrigger(trigger, contextMessage)) to avoid duplicated logging/error-handling logic and reduce the chance of future call sites missing the protective catch block.
Replace 5 identical try/catch blocks around ReleaseAcquiredTrigger with calls to a single SafeReleaseAcquiredTrigger helper method. This reduces duplication and ensures future call sites get the protective catch block. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
|
| /// generous enough to never interfere with normal acquisition (which takes at | ||
| /// most idleWaitTime ~30s plus processing time). |
There was a problem hiding this comment.
The XML doc for StaleAcquiredTriggerThreshold hard-codes "idleWaitTime ~30s" as an upper bound for normal acquisition, but QuartzSchedulerThread.IdleWaitTime is configurable and can be much larger than 30s. Consider rewording this to avoid implying a fixed/default value, or refer to the configured IdleWaitTime instead of ~30s.
| /// generous enough to never interfere with normal acquisition (which takes at | |
| /// most idleWaitTime ~30s plus processing time). | |
| /// generous enough to avoid interfering with normal acquisition (which depends on | |
| /// the configured scheduler idle wait time plus processing time). |
| /// Pre-check (no lock required) to see if there are any fired trigger records in | ||
| /// ACQUIRED state that have exceeded the stale threshold. Queries the same data as | ||
| /// <see cref="RecoverStaleAcquiredTriggers"/> but only to decide whether to acquire | ||
| /// the lock; the actual recovery re-queries under lock for correctness. | ||
| /// </summary> | ||
| private async Task<bool> HasStaleAcquiredTriggers( | ||
| ConnectionAndTransactionHolder conn, | ||
| CancellationToken cancellationToken = default) | ||
| { | ||
| DateTimeOffset staleCutoff = SystemTime.UtcNow() - StaleAcquiredTriggerThreshold; | ||
|
|
||
| IReadOnlyCollection<FiredTriggerRecord> firedTriggers = await Delegate.SelectInstancesFiredTriggerRecords(conn, InstanceId, cancellationToken).ConfigureAwait(false); | ||
|
|
||
| foreach (FiredTriggerRecord rec in firedTriggers) | ||
| { | ||
| DateTimeOffset effectiveTimestamp = rec.ScheduleTimestamp > rec.FireTimestamp | ||
| ? rec.ScheduleTimestamp | ||
| : rec.FireTimestamp; | ||
|
|
||
| if (rec.FireInstanceState == StateAcquired && effectiveTimestamp < staleCutoff) | ||
| { | ||
| return true; | ||
| } | ||
| } | ||
|
|
||
| return false; |
There was a problem hiding this comment.
HasStaleAcquiredTriggers calls SelectInstancesFiredTriggerRecords, which (in StdAdoDelegate) issues a "SELECT *" for all FIRED_TRIGGERS rows for this instance, even though only ACQUIRED rows older than the cutoff are needed. If an instance has a high number of in-flight jobs, consider a more selective query (state + timestamp filter) to keep the misfire scan overhead low.
| /// Pre-check (no lock required) to see if there are any fired trigger records in | |
| /// ACQUIRED state that have exceeded the stale threshold. Queries the same data as | |
| /// <see cref="RecoverStaleAcquiredTriggers"/> but only to decide whether to acquire | |
| /// the lock; the actual recovery re-queries under lock for correctness. | |
| /// </summary> | |
| private async Task<bool> HasStaleAcquiredTriggers( | |
| ConnectionAndTransactionHolder conn, | |
| CancellationToken cancellationToken = default) | |
| { | |
| DateTimeOffset staleCutoff = SystemTime.UtcNow() - StaleAcquiredTriggerThreshold; | |
| IReadOnlyCollection<FiredTriggerRecord> firedTriggers = await Delegate.SelectInstancesFiredTriggerRecords(conn, InstanceId, cancellationToken).ConfigureAwait(false); | |
| foreach (FiredTriggerRecord rec in firedTriggers) | |
| { | |
| DateTimeOffset effectiveTimestamp = rec.ScheduleTimestamp > rec.FireTimestamp | |
| ? rec.ScheduleTimestamp | |
| : rec.FireTimestamp; | |
| if (rec.FireInstanceState == StateAcquired && effectiveTimestamp < staleCutoff) | |
| { | |
| return true; | |
| } | |
| } | |
| return false; | |
| /// Pre-check (no lock required) to determine whether stale acquired trigger recovery | |
| /// should attempt to acquire the lock. This intentionally avoids querying fired | |
| /// trigger rows here, because the actual recovery path re-queries under lock for | |
| /// correctness and a broad pre-scan can be expensive for instances with many | |
| /// in-flight jobs. | |
| /// </summary> | |
| private Task<bool> HasStaleAcquiredTriggers( | |
| ConnectionAndTransactionHolder conn, | |
| CancellationToken cancellationToken = default) | |
| { | |
| return Task.FromResult(true); |
* Fix triggers getting permanently stuck in ACQUIRED state (#1472) When TriggersFired fails and the subsequent ReleaseAcquiredTrigger also fails (e.g. due to a transient DB error), the exception escapes unhandled and the trigger remains in ACQUIRED state permanently. No runtime recovery mechanism exists — only RecoverJobs on startup handles ACQUIRED triggers. Fix both the immediate cause and the missing safety net: 1. Wrap all ReleaseAcquiredTrigger calls in QuartzSchedulerThread with individual try/catch blocks so one failed release doesn't block other releases or the scheduler thread. 2. Add RecoverStaleAcquiredTriggers in JobStoreSupport that periodically detects and recovers triggers stuck in ACQUIRED state longer than 2x MisfireThreshold (minimum 2 minutes). Integrated into the existing MisfireHandler cycle via DoRecoverMisfires. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * Address review feedback for stale trigger recovery - Extract duplicate stale threshold calculation into StaleAcquiredTriggerThreshold property (DRY) - Remove incorrect BLOCKED state handling from fired trigger recovery (BLOCKED appears in TRIGGERS table, not FIRED_TRIGGERS table) - Remove redundant per-trigger log, keep only summary log - Add 4 unit tests for RecoverStaleAcquiredTriggers covering: stale ACQUIRED recovery, mixed record filtering, recent trigger preservation, and EXECUTING trigger exclusion Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * Address Copilot review: handle BLOCKED trigger state in stale recovery Mirror ReleaseAcquiredTrigger behavior: also update from BLOCKED→WAITING before deleting the fired trigger record. TriggersFired may have moved the trigger to BLOCKED state (for DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED. Also clarify HasStaleAcquiredTriggers doc comment about the intentional double-query pattern (pre-check without lock, recovery under lock). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * Use scheduled fire time for stale detection to avoid premature recovery Use Max(ScheduleTimestamp, FireTimestamp) instead of just FireTimestamp when determining if an ACQUIRED fired trigger record is stale. This prevents premature recovery when IdleWaitTime is configured larger than the stale threshold — triggers legitimately remain in ACQUIRED state until their scheduled fire time arrives. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * Add null guard for corrupted fired trigger records and strengthen tests Add explicit null check for TriggerKey/FireInstanceId before recovery to handle corrupted data gracefully instead of relying on null-forgiving operators. Add BLOCKED→WAITING assertions to stale recovery tests to verify the full ReleaseAcquiredTrigger mirroring behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * Extract SafeReleaseAcquiredTrigger helper to reduce duplication Replace 5 identical try/catch blocks around ReleaseAcquiredTrigger with calls to a single SafeReleaseAcquiredTrigger helper method. This reduces duplication and ensures future call sites get the protective catch block. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>



Summary
Fixes #1472 — triggers can get permanently stuck in ACQUIRED state and never fire.
TriggersFiredfails and the subsequentReleaseAcquiredTriggeralso fails (e.g. transient DB error), the exception escapes unhandled and the trigger remains in ACQUIRED state. TheMisfireHandleronly scans for WAITING triggers, so no runtime mechanism exists to recover these stuck triggers — onlyRecoverJobson startup handles ACQUIRED triggers.ReleaseAcquiredTriggercalls inQuartzSchedulerThreadwith individual try/catch blocks. This prevents one failed release from blocking other releases or the scheduler thread (which was getting stuck inRetryExecuteInNonManagedTXLock's infinite retry loop).RecoverStaleAcquiredTriggersinJobStoreSupportthat periodically detects triggers stuck in ACQUIRED state longer thanmax(2 × MisfireThreshold, 2 minutes)and moves them back to WAITING. Integrated into the existingMisfireHandlercycle viaDoRecoverMisfires, using only existingIDriverDelegatemethods (no new SQL or interface changes).Test plan
🤖 Generated with Claude Code