Skip to content

Fix triggers getting permanently stuck in ACQUIRED state#2980

Merged
lahma merged 6 commits into
3.xfrom
past-fire
Apr 3, 2026
Merged

Fix triggers getting permanently stuck in ACQUIRED state#2980
lahma merged 6 commits into
3.xfrom
past-fire

Conversation

@lahma

@lahma lahma commented Apr 3, 2026

Copy link
Copy Markdown
Member

Summary

Fixes #1472 — triggers can get permanently stuck in ACQUIRED state and never fire.

  • Root cause: When TriggersFired fails and the subsequent ReleaseAcquiredTrigger also fails (e.g. transient DB error), the exception escapes unhandled and the trigger remains in ACQUIRED state. The MisfireHandler only scans for WAITING triggers, so no runtime mechanism exists to recover these stuck triggers — only RecoverJobs on startup handles ACQUIRED triggers.
  • Fix 1: Wrap all ReleaseAcquiredTrigger calls in QuartzSchedulerThread with individual try/catch blocks. This prevents one failed release from blocking other releases or the scheduler thread (which was getting stuck in RetryExecuteInNonManagedTXLock's infinite retry loop).
  • Fix 2: Add RecoverStaleAcquiredTriggers in JobStoreSupport that periodically detects triggers stuck in ACQUIRED state longer than max(2 × MisfireThreshold, 2 minutes) and moves them back to WAITING. Integrated into the existing MisfireHandler cycle via DoRecoverMisfires, using only existing IDriverDelegate methods (no new SQL or interface changes).

Test plan

  • Solution builds with 0 warnings and 0 errors
  • All 1048 unit tests pass (4 pre-existing skips)
  • Verify with integration tests against a real database

🤖 Generated with Claude Code

lahma and others added 2 commits April 3, 2026 09:38
When TriggersFired fails and the subsequent ReleaseAcquiredTrigger also
fails (e.g. due to a transient DB error), the exception escapes unhandled
and the trigger remains in ACQUIRED state permanently. No runtime recovery
mechanism exists — only RecoverJobs on startup handles ACQUIRED triggers.

Fix both the immediate cause and the missing safety net:

1. Wrap all ReleaseAcquiredTrigger calls in QuartzSchedulerThread with
   individual try/catch blocks so one failed release doesn't block other
   releases or the scheduler thread.

2. Add RecoverStaleAcquiredTriggers in JobStoreSupport that periodically
   detects and recovers triggers stuck in ACQUIRED state longer than
   2x MisfireThreshold (minimum 2 minutes). Integrated into the existing
   MisfireHandler cycle via DoRecoverMisfires.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Extract duplicate stale threshold calculation into
  StaleAcquiredTriggerThreshold property (DRY)
- Remove incorrect BLOCKED state handling from fired trigger recovery
  (BLOCKED appears in TRIGGERS table, not FIRED_TRIGGERS table)
- Remove redundant per-trigger log, keep only summary log
- Add 4 unit tests for RecoverStaleAcquiredTriggers covering:
  stale ACQUIRED recovery, mixed record filtering, recent trigger
  preservation, and EXECUTING trigger exclusion

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses an ADO job store failure mode where triggers can remain permanently stuck in ACQUIRED state (and never fire) when TriggersFired fails and subsequent ReleaseAcquiredTrigger calls also fail (e.g., transient DB errors). It hardens the scheduler thread against release failures and adds a periodic recovery path during the existing misfire handling cycle.

Changes:

  • Wraps ReleaseAcquiredTrigger calls in QuartzSchedulerThread with per-trigger try/catch to prevent scheduler thread disruption when releases fail.
  • Adds RecoverStaleAcquiredTriggers (and a pre-check) to detect and recover triggers stuck in ACQUIRED beyond a computed threshold, integrated into DoRecoverMisfires.
  • Adds unit tests validating stale-acquired trigger recovery behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
src/Quartz/Impl/AdoJobStore/JobStoreSupport.cs Introduces stale-ACQUIRED detection/recovery logic and integrates it into the misfire recovery cycle.
src/Quartz/Core/QuartzSchedulerThread.cs Prevents ReleaseAcquiredTrigger failures from escaping and stalling the scheduler loop.
src/Quartz.Tests.Unit/Impl/AdoJobStore/JobStoreSupportTest.cs Adds unit coverage for recovering stale acquired trigger records.

{
try
{
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RecoverStaleAcquiredTriggers currently only updates TRIGGERS from ACQUIRED→WAITING before deleting the fired trigger record. ReleaseAcquiredTrigger updates from both ACQUIRED and BLOCKED (to cover cases where the trigger state has transitioned) before deleting the FIRED_TRIGGERS row. To avoid leaving a trigger stuck if it is in BLOCKED state, mirror ReleaseAcquiredTrigger by also calling UpdateTriggerStateFromOtherState(..., StateWaiting, StateBlocked) before DeleteFiredTrigger (or otherwise ensure BLOCKED is handled).

Suggested change
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false);

Copilot uses AI. Check for mistakes.
Comment on lines +941 to +952
/// <summary>
/// Lightweight check (no lock required) to see if there are any fired trigger records
/// in ACQUIRED state that have exceeded the stale threshold.
/// </summary>
private async Task<bool> HasStaleAcquiredTriggers(
ConnectionAndTransactionHolder conn,
CancellationToken cancellationToken = default)
{
DateTimeOffset staleCutoff = SystemTime.UtcNow() - StaleAcquiredTriggerThreshold;

IReadOnlyCollection<FiredTriggerRecord> firedTriggers = await Delegate.SelectInstancesFiredTriggerRecords(conn, InstanceId, cancellationToken).ConfigureAwait(false);

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HasStaleAcquiredTriggers is described as a “lightweight check”, but it calls SelectInstancesFiredTriggerRecords and iterates all fired trigger records for the instance, which can be sizable under load (lots of EXECUTING records). It also duplicates the same query/scan done again in RecoverStaleAcquiredTriggers after the lock is obtained (two full reads when stale triggers are present). Consider either adjusting the wording, or refactoring to avoid the extra full-table read (e.g., perform the scan only once under lock, or add a cheaper delegate query if feasible).

Copilot uses AI. Check for mistakes.
Mirror ReleaseAcquiredTrigger behavior: also update from BLOCKED→WAITING
before deleting the fired trigger record. TriggersFired may have moved
the trigger to BLOCKED state (for DisallowConcurrentExecution jobs) while
the fired record is still ACQUIRED.

Also clarify HasStaleAcquiredTriggers doc comment about the intentional
double-query pattern (pre-check without lock, recovery under lock).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@lahma lahma added the port-main Requires porting to main branch label Apr 3, 2026
@lahma
lahma requested a review from Copilot April 3, 2026 07:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

int recoveredCount = 0;
foreach (FiredTriggerRecord rec in firedTriggers)
{
if (rec.FireInstanceState == StateAcquired && rec.FireTimestamp < staleCutoff)

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RecoverStaleAcquiredTriggers determines staleness using rec.FireTimestamp (acquisition time). Since Quartz acquires triggers up to now + IdleWaitTime (and IdleWaitTime is configurable via quartz.scheduler.idleWaitTime), a trigger can legitimately remain in ACQUIRED state longer than the 2-minute floor while the scheduler thread is simply waiting for its scheduled fire time. This can cause premature recovery (flip back to WAITING + delete fired record) and trigger acquisition thrash. Consider basing the cutoff on rec.ScheduleTimestamp (scheduled fire time) instead of acquisition time, or otherwise incorporating the scheduler's acquisition window (IdleWaitTime/batch window) into the threshold.

Suggested change
if (rec.FireInstanceState == StateAcquired && rec.FireTimestamp < staleCutoff)
DateTimeOffset acquiredStateStalenessTimestamp = rec.ScheduleTimestamp > rec.FireTimestamp
? rec.ScheduleTimestamp
: rec.FireTimestamp;
if (rec.FireInstanceState == StateAcquired && acquiredStateStalenessTimestamp < staleCutoff)

Copilot uses AI. Check for mistakes.

foreach (FiredTriggerRecord rec in firedTriggers)
{
if (rec.FireInstanceState == StateAcquired && rec.FireTimestamp < staleCutoff)

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HasStaleAcquiredTriggers uses rec.FireTimestamp for the stale cutoff as well, which has the same false-positive risk when IdleWaitTime is configured > 2 minutes (or otherwise exceeds the staleness threshold). Align this pre-check with the recovery logic you intend (e.g., compare against rec.ScheduleTimestamp / scheduled fire time).

Suggested change
if (rec.FireInstanceState == StateAcquired && rec.FireTimestamp < staleCutoff)
if (rec.FireInstanceState == StateAcquired && rec.ScheduleTimestamp < staleCutoff)

Copilot uses AI. Check for mistakes.
Comment on lines +920 to +933
try
{
// Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED,
// because TriggersFired may have moved the trigger to BLOCKED state (for
// DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED.
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false);
await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false);
recoveredCount++;
}
catch (Exception e)
{
Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e);
}

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RecoverStaleAcquiredTriggers catches and logs exceptions per record while running inside the surrounding DoRecoverMisfires transaction. If one of the steps succeeds (e.g., state update) but a later step fails (e.g., DeleteFiredTrigger), the transaction can still be committed, leaving an inconsistent combination like trigger state WAITING with a lingering ACQUIRED fired-trigger row. Consider letting the exception propagate (so the outer logic rolls back and retries later) or otherwise ensuring the recovery of a record is atomic (all-or-nothing) within the transaction.

Suggested change
try
{
// Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED,
// because TriggersFired may have moved the trigger to BLOCKED state (for
// DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED.
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false);
await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false);
recoveredCount++;
}
catch (Exception e)
{
Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e);
}
// Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED,
// because TriggersFired may have moved the trigger to BLOCKED state (for
// DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED.
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false);
await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false);
recoveredCount++;

Copilot uses AI. Check for mistakes.
Use Max(ScheduleTimestamp, FireTimestamp) instead of just FireTimestamp
when determining if an ACQUIRED fired trigger record is stale. This
prevents premature recovery when IdleWaitTime is configured larger than
the stale threshold — triggers legitimately remain in ACQUIRED state
until their scheduled fire time arrives.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

Comment on lines +927 to +939
try
{
// Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED,
// because TriggersFired may have moved the trigger to BLOCKED state (for
// DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED.
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false);
await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false);
recoveredCount++;
}
catch (Exception e)
{
Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e);

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FiredTriggerRecord.TriggerKey and FireInstanceId are nullable, but this recovery path dereferences them via rec.TriggerKey! / rec.FireInstanceId!. If a delegate ever returns a partial/invalid record (or data is corrupted), this will throw and prevent recovery of other stale records. Consider adding an explicit null check (and skipping/logging the bad record) instead of relying on null-forgiving here.

Suggested change
try
{
// Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED,
// because TriggersFired may have moved the trigger to BLOCKED state (for
// DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED.
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false);
await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false);
recoveredCount++;
}
catch (Exception e)
{
Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e);
if (rec.TriggerKey is null || rec.FireInstanceId is null)
{
Log.Warn($"Skipping stale acquired trigger recovery for invalid fired trigger record. TriggerKey: '{rec.TriggerKey}', FireInstanceId: '{rec.FireInstanceId}'.");
continue;
}
TriggerKey triggerKey = rec.TriggerKey;
string fireInstanceId = rec.FireInstanceId;
try
{
// Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED,
// because TriggersFired may have moved the trigger to BLOCKED state (for
// DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED.
await Delegate.UpdateTriggerStateFromOtherState(conn, triggerKey, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, triggerKey, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false);
await Delegate.DeleteFiredTrigger(conn, fireInstanceId, cancellationToken).ConfigureAwait(false);
recoveredCount++;
}
catch (Exception e)
{
Log.ErrorException($"Error recovering stale acquired trigger '{triggerKey}'.", e);

Copilot uses AI. Check for mistakes.
Comment on lines +927 to +940
try
{
// Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED,
// because TriggersFired may have moved the trigger to BLOCKED state (for
// DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED.
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false);
await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false);
recoveredCount++;
}
catch (Exception e)
{
Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e);
}

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Catching and swallowing exceptions inside RecoverStaleAcquiredTriggers can lead to committing partial updates (e.g., trigger moved back to WAITING but fired-trigger row not deleted due to a transient error). That can leave inconsistent data and still require later cleanup. Consider letting exceptions propagate (so the outer DoRecoverMisfires rolls back and the misfire handler retries), or otherwise ensuring the state updates + fired-trigger delete happen atomically per record.

Suggested change
try
{
// Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED,
// because TriggersFired may have moved the trigger to BLOCKED state (for
// DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED.
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false);
await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false);
recoveredCount++;
}
catch (Exception e)
{
Log.ErrorException($"Error recovering stale acquired trigger '{rec.TriggerKey}'.", e);
}
// Mirror ReleaseAcquiredTrigger: update from both ACQUIRED and BLOCKED,
// because TriggersFired may have moved the trigger to BLOCKED state (for
// DisallowConcurrentExecution jobs) while the fired record is still ACQUIRED.
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateAcquired, cancellationToken).ConfigureAwait(false);
await Delegate.UpdateTriggerStateFromOtherState(conn, rec.TriggerKey!, StateWaiting, StateBlocked, cancellationToken).ConfigureAwait(false);
await Delegate.DeleteFiredTrigger(conn, rec.FireInstanceId!, cancellationToken).ConfigureAwait(false);
recoveredCount++;

Copilot uses AI. Check for mistakes.
Comment on lines 3518 to 3524
{
Log.DebugFormat("Found {0} triggers that missed their scheduled fire-time.", misfireCount);
}

if (misfireCount > 0)
{
transOwner = await LockHandler.ObtainLock(requestorId, conn, LockTriggerAccess, cancellationToken).ConfigureAwait(false);

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When there are no misfires (misfireCount == 0), HasStaleAcquiredTriggers still loads all fired-trigger records for the instance every misfire-handler cycle. On busy schedulers this can be a sizable result set and adds steady-state DB load even when nothing is stale. Consider throttling this check (e.g., only every N cycles / time window) or otherwise reducing the query frequency/size so idle systems don’t pay this cost each scan.

Copilot uses AI. Check for mistakes.
Comment on lines +674 to +684
A.CallTo(() => driverDelegate.UpdateTriggerStateFromOtherState(
A<ConnectionAndTransactionHolder>.Ignored,
triggerKey,
AdoConstants.StateWaiting,
AdoConstants.StateAcquired,
A<CancellationToken>.Ignored)).MustHaveHappenedOnceExactly();

A.CallTo(() => driverDelegate.DeleteFiredTrigger(
A<ConnectionAndTransactionHolder>.Ignored,
"entry_stale_1",
A<CancellationToken>.Ignored)).MustHaveHappenedOnceExactly();

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test validates the ACQUIRED→WAITING transition, but the production code also updates from BLOCKED→WAITING to mirror ReleaseAcquiredTrigger. Adding an assertion that UpdateTriggerStateFromOtherState(..., StateWaiting, StateBlocked, ...) was called would better lock in the intended behavior.

Copilot uses AI. Check for mistakes.
Comment on lines +730 to +740
A.CallTo(() => driverDelegate.UpdateTriggerStateFromOtherState(
A<ConnectionAndTransactionHolder>.Ignored,
staleTrigger,
AdoConstants.StateWaiting,
AdoConstants.StateAcquired,
A<CancellationToken>.Ignored)).MustHaveHappenedOnceExactly();

A.CallTo(() => driverDelegate.DeleteFiredTrigger(
A<ConnectionAndTransactionHolder>.Ignored,
"entry_stale",
A<CancellationToken>.Ignored)).MustHaveHappenedOnceExactly();

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to the earlier test, this one asserts the ACQUIRED→WAITING update but doesn’t assert the BLOCKED→WAITING update that the implementation performs. Adding that assertion would ensure the recovery logic continues to mirror ReleaseAcquiredTrigger for DisallowConcurrentExecution scenarios.

Copilot uses AI. Check for mistakes.
Add explicit null check for TriggerKey/FireInstanceId before recovery
to handle corrupted data gracefully instead of relying on null-forgiving
operators.

Add BLOCKED→WAITING assertions to stale recovery tests to verify the
full ReleaseAcquiredTrigger mirroring behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment on lines +417 to +421
try
{
await qsRsrcs.JobStore.ReleaseAcquiredTrigger(t, CancellationToken.None).ConfigureAwait(false);
}
catch (Exception releaseEx)

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are now several identical try/catch wrappers around JobStore.ReleaseAcquiredTrigger(...) in this file. Consider extracting a small helper (e.g., SafeReleaseAcquiredTrigger(trigger, contextMessage)) to avoid duplicated logging/error-handling logic and reduce the chance of future call sites missing the protective catch block.

Copilot uses AI. Check for mistakes.
Replace 5 identical try/catch blocks around ReleaseAcquiredTrigger with
calls to a single SafeReleaseAcquiredTrigger helper method. This reduces
duplication and ensures future call sites get the protective catch block.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@sonarqubecloud

sonarqubecloud Bot commented Apr 3, 2026

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comment on lines +420 to +421
/// generous enough to never interfere with normal acquisition (which takes at
/// most idleWaitTime ~30s plus processing time).

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The XML doc for StaleAcquiredTriggerThreshold hard-codes "idleWaitTime ~30s" as an upper bound for normal acquisition, but QuartzSchedulerThread.IdleWaitTime is configurable and can be much larger than 30s. Consider rewording this to avoid implying a fixed/default value, or refer to the configured IdleWaitTime instead of ~30s.

Suggested change
/// generous enough to never interfere with normal acquisition (which takes at
/// most idleWaitTime ~30s plus processing time).
/// generous enough to avoid interfering with normal acquisition (which depends on
/// the configured scheduler idle wait time plus processing time).

Copilot uses AI. Check for mistakes.
Comment on lines +959 to +984
/// Pre-check (no lock required) to see if there are any fired trigger records in
/// ACQUIRED state that have exceeded the stale threshold. Queries the same data as
/// <see cref="RecoverStaleAcquiredTriggers"/> but only to decide whether to acquire
/// the lock; the actual recovery re-queries under lock for correctness.
/// </summary>
private async Task<bool> HasStaleAcquiredTriggers(
ConnectionAndTransactionHolder conn,
CancellationToken cancellationToken = default)
{
DateTimeOffset staleCutoff = SystemTime.UtcNow() - StaleAcquiredTriggerThreshold;

IReadOnlyCollection<FiredTriggerRecord> firedTriggers = await Delegate.SelectInstancesFiredTriggerRecords(conn, InstanceId, cancellationToken).ConfigureAwait(false);

foreach (FiredTriggerRecord rec in firedTriggers)
{
DateTimeOffset effectiveTimestamp = rec.ScheduleTimestamp > rec.FireTimestamp
? rec.ScheduleTimestamp
: rec.FireTimestamp;

if (rec.FireInstanceState == StateAcquired && effectiveTimestamp < staleCutoff)
{
return true;
}
}

return false;

Copilot AI Apr 3, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HasStaleAcquiredTriggers calls SelectInstancesFiredTriggerRecords, which (in StdAdoDelegate) issues a "SELECT *" for all FIRED_TRIGGERS rows for this instance, even though only ACQUIRED rows older than the cutoff are needed. If an instance has a high number of in-flight jobs, consider a more selective query (state + timestamp filter) to keep the misfire scan overhead low.

Suggested change
/// Pre-check (no lock required) to see if there are any fired trigger records in
/// ACQUIRED state that have exceeded the stale threshold. Queries the same data as
/// <see cref="RecoverStaleAcquiredTriggers"/> but only to decide whether to acquire
/// the lock; the actual recovery re-queries under lock for correctness.
/// </summary>
private async Task<bool> HasStaleAcquiredTriggers(
ConnectionAndTransactionHolder conn,
CancellationToken cancellationToken = default)
{
DateTimeOffset staleCutoff = SystemTime.UtcNow() - StaleAcquiredTriggerThreshold;
IReadOnlyCollection<FiredTriggerRecord> firedTriggers = await Delegate.SelectInstancesFiredTriggerRecords(conn, InstanceId, cancellationToken).ConfigureAwait(false);
foreach (FiredTriggerRecord rec in firedTriggers)
{
DateTimeOffset effectiveTimestamp = rec.ScheduleTimestamp > rec.FireTimestamp
? rec.ScheduleTimestamp
: rec.FireTimestamp;
if (rec.FireInstanceState == StateAcquired && effectiveTimestamp < staleCutoff)
{
return true;
}
}
return false;
/// Pre-check (no lock required) to determine whether stale acquired trigger recovery
/// should attempt to acquire the lock. This intentionally avoids querying fired
/// trigger rows here, because the actual recovery path re-queries under lock for
/// correctness and a broad pre-scan can be expensive for instances with many
/// in-flight jobs.
/// </summary>
private Task<bool> HasStaleAcquiredTriggers(
ConnectionAndTransactionHolder conn,
CancellationToken cancellationToken = default)
{
return Task.FromResult(true);

Copilot uses AI. Check for mistakes.
@lahma
lahma merged commit ea14a9a into 3.x Apr 3, 2026
19 checks passed
@lahma
lahma deleted the past-fire branch April 3, 2026 07:45
lahma added a commit that referenced this pull request Apr 3, 2026
* Fix triggers getting permanently stuck in ACQUIRED state (#1472)

When TriggersFired fails and the subsequent ReleaseAcquiredTrigger also
fails (e.g. due to a transient DB error), the exception escapes unhandled
and the trigger remains in ACQUIRED state permanently. No runtime recovery
mechanism exists — only RecoverJobs on startup handles ACQUIRED triggers.

Fix both the immediate cause and the missing safety net:

1. Wrap all ReleaseAcquiredTrigger calls in QuartzSchedulerThread with
   individual try/catch blocks so one failed release doesn't block other
   releases or the scheduler thread.

2. Add RecoverStaleAcquiredTriggers in JobStoreSupport that periodically
   detects and recovers triggers stuck in ACQUIRED state longer than
   2x MisfireThreshold (minimum 2 minutes). Integrated into the existing
   MisfireHandler cycle via DoRecoverMisfires.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* Address review feedback for stale trigger recovery

- Extract duplicate stale threshold calculation into
  StaleAcquiredTriggerThreshold property (DRY)
- Remove incorrect BLOCKED state handling from fired trigger recovery
  (BLOCKED appears in TRIGGERS table, not FIRED_TRIGGERS table)
- Remove redundant per-trigger log, keep only summary log
- Add 4 unit tests for RecoverStaleAcquiredTriggers covering:
  stale ACQUIRED recovery, mixed record filtering, recent trigger
  preservation, and EXECUTING trigger exclusion

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* Address Copilot review: handle BLOCKED trigger state in stale recovery

Mirror ReleaseAcquiredTrigger behavior: also update from BLOCKED→WAITING
before deleting the fired trigger record. TriggersFired may have moved
the trigger to BLOCKED state (for DisallowConcurrentExecution jobs) while
the fired record is still ACQUIRED.

Also clarify HasStaleAcquiredTriggers doc comment about the intentional
double-query pattern (pre-check without lock, recovery under lock).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* Use scheduled fire time for stale detection to avoid premature recovery

Use Max(ScheduleTimestamp, FireTimestamp) instead of just FireTimestamp
when determining if an ACQUIRED fired trigger record is stale. This
prevents premature recovery when IdleWaitTime is configured larger than
the stale threshold — triggers legitimately remain in ACQUIRED state
until their scheduled fire time arrives.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* Add null guard for corrupted fired trigger records and strengthen tests

Add explicit null check for TriggerKey/FireInstanceId before recovery
to handle corrupted data gracefully instead of relying on null-forgiving
operators.

Add BLOCKED→WAITING assertions to stale recovery tests to verify the
full ReleaseAcquiredTrigger mirroring behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* Extract SafeReleaseAcquiredTrigger helper to reduce duplication

Replace 5 identical try/catch blocks around ReleaseAcquiredTrigger with
calls to a single SafeReleaseAcquiredTrigger helper method. This reduces
duplication and ensures future call sites get the protective catch block.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
@lahma lahma removed the port-main Requires porting to main branch label Apr 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants