Skip to content

Add execution groups for per-node thread limits#3004

Merged
lahma merged 12 commits into
3.xfrom
execution-groups
Apr 4, 2026
Merged

Add execution groups for per-node thread limits#3004
lahma merged 12 commits into
3.xfrom
execution-groups

Conversation

@lahma

@lahma lahma commented Apr 4, 2026

Copy link
Copy Markdown
Member

Summary

Adds execution groups — optional tags on triggers that allow each scheduler node to limit how many threads a category of job can consume concurrently. This prevents resource-intensive jobs from starving lightweight work of available threads.

Closes #1175
Closes #830

How it works

  • Tag triggers with .WithExecutionGroup("batch-jobs") via TriggerBuilder
  • Configure per-node limits through properties, DI, or runtime API
  • The scheduler thread computes available slots per group before each trigger acquisition cycle
  • Job stores filter trigger candidates by execution group during acquisition
  • Running counts are tracked by wrapping job execution delegates — zero IThreadPool changes needed

Configuration

Properties:

quartz.executionLimit.batch-jobs = 2
quartz.executionLimit.high-cpu = 3
quartz.executionLimit._ = 10          # triggers with no execution group
quartz.executionLimit.* = 5           # default for unlisted groups

DI:

services.AddQuartz(q =>
{
    q.UseExecutionLimits(limits =>
    {
        limits.ForGroup("batch-jobs", maxConcurrent: 2);
        limits.ForDefaultGroup(maxConcurrent: 10);
        limits.ForOtherGroups(maxConcurrent: 5);
    });
});

Runtime API:

scheduler.SetExecutionLimits(
    new ExecutionLimits()
        .ForGroup("batch-jobs", 2)
        .ForDefaultGroup(10)
        .ForOtherGroups(5));

3.x compatibility

Uses established opportunistic patterns — no public interface changes:

  • INextVersionTrigger for ExecutionGroup property (on AbstractTrigger)
  • INextVersionJobStore for execution-limit-aware AcquireNextTriggers overload
  • INextVersionDelegate for ADO column probing and SQL filtering
  • SchedulerExtensions for public scheduler API (SetExecutionLimits/GetExecutionLimits)

What's included

  • Core: ExecutionLimits model, TriggerBuilder.WithExecutionGroup(), scheduler thread enforcement with per-group tracking
  • RAMJobStore: In-memory filtering during trigger acquisition
  • ADO JobStore: Optional EXECUTION_GROUP column probing, SQL-aware acquisition, graceful degradation when column is absent
  • Configuration: quartz.executionLimit.* property parsing in StdSchedulerFactory
  • DI: UseExecutionLimits() on IServiceCollectionQuartzConfigurator
  • Dashboard: Execution group column on trigger list, trigger detail, and currently-executing pages; GetExecutionLimits API
  • Docs: Full tutorial (execution-groups.md), trigger docs update, configuration reference
  • DB migration: schema_30_add_execution_group.sql with scripts for all supported databases
  • Tests: 19 unit tests covering model, limits logic, builder round-tripping

Test plan

  • All 1246 existing unit tests pass (0 failures)
  • 19 new execution groups tests pass
  • Full solution builds with 0 warnings, 0 errors
  • Integration test: end-to-end with RAMJobStore — schedule jobs with different execution groups, configure limits, verify concurrency
  • Integration test: ADO job store round-trip with EXECUTION_GROUP column
  • Manual test: Dashboard shows execution group in trigger list and currently executing views

🤖 Generated with Claude Code

lahma and others added 4 commits April 4, 2026 16:49
Execution groups allow each scheduler node to limit how many threads
a category of job can consume concurrently, preventing resource-intensive
jobs from starving lightweight work.

A trigger is optionally tagged with an execution group via
TriggerBuilder.WithExecutionGroup("batch-jobs"). Per-node limits are
configured through properties (quartz.executionLimit.*), DI
(UseExecutionLimits), or the runtime API (SetExecutionLimits extension).

Implementation uses the established 3.x opportunistic patterns:
- INextVersionTrigger for the ExecutionGroup property
- INextVersionJobStore for execution-limit-aware trigger acquisition
- INextVersionDelegate for ADO column probing and SQL filtering
- SchedulerExtensions for the public scheduler API

The scheduler thread tracks running counts per execution group by
wrapping job execution delegates, requiring zero IThreadPool changes.
RAMJobStore filters candidates in-memory; ADO job store probes for
an optional EXECUTION_GROUP column and applies SQL + in-memory filtering.

Includes Dashboard UI (execution group column/filter on trigger and
currently-executing pages), documentation (tutorial, config reference),
database migration scripts, and unit tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Fix count leak: decrement running execution group count when
  RunInThread returns false (thread pool rejected the job)
- Fix mutable-after-share: snapshot ExecutionLimits in SetExecutionLimits
  so callers cannot mutate the instance after setting it
- Validate negative limits: ForGroup/ForDefaultGroup/ForOtherGroups now
  reject negative maxConcurrent values with ArgumentOutOfRangeException
- Fix ADO noEarlierThan: pass MisfireTime instead of computing from
  timeWindow, matching the original SelectTriggerToAcquire behavior
- Fix bare catch: catch only SchedulerException in dashboard
  GetExecutionLimits instead of swallowing all exceptions
- Fix DTO: use IReadOnlyDictionary in ExecutionLimitsDto
- Deduplicate: original AcquireNextTriggers delegates to the new
  overload instead of duplicating the validation lambda
- Register quartz.executionLimit prefix in known properties list
- Add tests: ParseExecutionLimits property parsing (numeric, unlimited,
  invalid, empty), ComputeAvailableLimits subtraction logic, negative
  value rejection, snapshot independence

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Add SqlSelectNextTriggerToAcquireWithExecutionGroup SQL constant that
  includes t.EXECUTION_GROUP in the SELECT clause
- Add GetSelectNextTriggerToAcquireWithExecutionGroupSql virtual method
  with overrides in all 6 DB-specific delegates (SqlServer, PostgreSQL,
  MySQL, Oracle, SQLite, Firebird) applying their LIMIT/TOP/ROWS syntax
- SelectTriggerToAcquire now selects the wider query when
  HasExecutionGroupColumn is true, falling back to the standard query
  (without EXECUTION_GROUP) when the column doesn't exist
- Add EXECUTION_GROUP column to all 8 database table creation scripts
  (tables_sqlite, tables_sqlServer, tables_sqlServerMOT,
  tables_sqlServer_Below2016, tables_postgres, tables_mysql_innodb,
  tables_oracle, tables_firebird) in both QRTZ_TRIGGERS and
  QRTZ_FIRED_TRIGGERS tables
- Add TestExecutionGroups to SmokeTestPerformer exercising execution
  group round-trip through store/retrieve and the limits API
- Remove trailing blank lines in IQuartzApiClient.cs
- Remove unused NullSignaler from unit tests

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- InsertTrigger: conditionally appends EXECUTION_GROUP column and
  parameter when HasExecutionGroupColumn is true
- UpdateTrigger: separate UPDATE for EXECUTION_GROUP after main update
  (avoids modifying the complex SqlUpdateTrigger/SqlUpdateTriggerSkipData
  constants that use positional parameter ordering)
- SelectTrigger: reads execution group via separate scalar query after
  trigger load when column is available
- Add SqlSelectTriggerExecutionGroup and SqlUpdateTriggerExecutionGroup
  SQL constants

Verified end-to-end with SQLite integration tests: execution group
round-trips correctly through store → retrieve.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds execution groups and per-node execution limits so a scheduler can cap concurrent executions for categories of triggers (e.g., “batch-jobs”) to prevent heavy workloads from consuming all available threads.

Changes:

  • Introduces ExecutionLimits model + configuration via properties, DI, and runtime scheduler extensions.
  • Plumbs execution-group-aware trigger acquisition through QuartzSchedulerThread, RAMJobStore, and AdoJobStore (with optional DB column probing).
  • Updates dashboard UI/DTOs and docs/schema scripts to surface and persist execution group data.

Reviewed changes

Copilot reviewed 43 out of 43 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
src/Quartz/TriggerBuilder.cs Adds WithExecutionGroup() and applies it to built triggers.
src/Quartz/SPI/IOperableTrigger.cs Adds ExecutionGroup to INextVersionTrigger (4.x-forward compat).
src/Quartz/SPI/IJobStore.cs Adds execution-limit-aware AcquireNextTriggers to INextVersionJobStore.
src/Quartz/Simpl/RAMJobStore.cs Filters acquired triggers by execution limits during acquisition.
src/Quartz/SchedulerExtensions.cs Adds SetExecutionLimits / GetExecutionLimits public scheduler API.
src/Quartz/Impl/Triggers/AbstractTrigger.cs Stores and round-trips ExecutionGroup via GetTriggerBuilder().
src/Quartz/Impl/StdSchedulerFactory.cs Parses quartz.executionLimit.* properties and applies limits.
src/Quartz/Impl/StdScheduler.cs Forwards execution limits calls to core scheduler.
src/Quartz/Impl/AdoJobStore/StdAdoDelegate.Triggers.cs Persists/probes EXECUTION_GROUP and supports exec-group-aware acquisition.
src/Quartz/Impl/AdoJobStore/StdAdoConstants.cs Adds SQL constants for execution group probing/select/update and acquisition query.
src/Quartz/Impl/AdoJobStore/SqlServerDelegate.cs Adds TOP-limited acquisition SQL including execution group.
src/Quartz/Impl/AdoJobStore/SQLiteDelegate.cs Adds LIMIT acquisition SQL including execution group.
src/Quartz/Impl/AdoJobStore/PostgreSQLDelegate.cs Adds LIMIT acquisition SQL including execution group.
src/Quartz/Impl/AdoJobStore/OracleDelegate.cs Adds ROWNUM-limited acquisition SQL including execution group.
src/Quartz/Impl/AdoJobStore/MySQLDelegate.cs Adds LIMIT acquisition SQL including execution group (with index hint).
src/Quartz/Impl/AdoJobStore/JobStoreSupport.cs Threads execution limits through ADO acquisition + probes optional column.
src/Quartz/Impl/AdoJobStore/IDriverDelegate.cs Extends INextVersionDelegate + TriggerAcquireResult to carry execution group.
src/Quartz/Impl/AdoJobStore/FirebirdDelegate.cs Adds ROWS-limited acquisition SQL including execution group.
src/Quartz/Impl/AdoJobStore/AdoConstants.cs Adds ColumnExecutionGroup constant.
src/Quartz/ExecutionLimits.cs New model for per-group limits + limit checking/decrement logic.
src/Quartz/Core/QuartzSchedulerThread.cs Computes available slots per group and tracks running counts per group.
src/Quartz/Core/QuartzScheduler.cs Stores execution limits snapshot in scheduler state.
src/Quartz.Tests.Unit/ExecutionGroupsTest.cs Adds unit tests for builder, limits logic, and property parsing.
src/Quartz.Tests.Integration/Impl/SmokeTestPerformer.cs Adds smoke test covering execution group round-trip + API calls.
src/Quartz.Extensions.DependencyInjection/ServiceCollectionQuartzConfigurator.cs Adds UseExecutionLimits() DI configuration -> properties mapping.
src/Quartz.Extensions.DependencyInjection/IServiceCollectionQuartzConfigurator.cs Exposes UseExecutionLimits() on the DI configurator interface.
src/Quartz.Dashboard/Services/IQuartzApiClient.cs Extends DTOs with ExecutionGroup and adds GetExecutionLimits().
src/Quartz.Dashboard/Services/InProcessQuartzApiClient.cs Populates execution group DTO fields and implements GetExecutionLimits().
src/Quartz.Dashboard/Components/Pages/Triggers.razor Displays execution group column in trigger list.
src/Quartz.Dashboard/Components/Pages/TriggerDetail.razor Displays execution group on trigger detail page.
src/Quartz.Dashboard/Components/Pages/CurrentlyExecuting.razor Displays execution group for currently executing jobs.
docs/documentation/quartz-3.x/tutorial/more-about-triggers.md Adds execution groups section and links tutorial.
docs/documentation/quartz-3.x/tutorial/execution-groups.md New tutorial page describing execution groups/limits and migration.
docs/documentation/quartz-3.x/configuration/reference.md Documents quartz.executionLimit.{group} configuration.
database/tables/tables_sqlServerMOT.sql Adds EXECUTION_GROUP columns to SQL Server MOT schema.
database/tables/tables_sqlServer.sql Adds EXECUTION_GROUP columns to SQL Server schema.
database/tables/tables_sqlServer_Below2016.sql Adds EXECUTION_GROUP columns to legacy SQL Server schema.
database/tables/tables_sqlite.sql Adds EXECUTION_GROUP columns to SQLite schema.
database/tables/tables_postgres.sql Adds execution_group columns to PostgreSQL schema.
database/tables/tables_oracle.sql Adds EXECUTION_GROUP columns to Oracle schema.
database/tables/tables_mysql_innodb.sql Adds EXECUTION_GROUP columns to MySQL/InnoDB schema.
database/tables/tables_firebird.sql Adds EXECUTION_GROUP columns to Firebird schema.
database/schema_30_add_execution_group.sql Adds optional migration script template for EXECUTION_GROUP columns.

Comment thread src/Quartz/Core/QuartzSchedulerThread.cs Outdated
Comment thread src/Quartz/Impl/AdoJobStore/JobStoreSupport.cs
Comment thread src/Quartz/Impl/AdoJobStore/JobStoreSupport.cs
Comment thread src/Quartz/Impl/AdoJobStore/StdAdoDelegate.Triggers.cs
Comment thread src/Quartz/Impl/AdoJobStore/StdAdoDelegate.Triggers.cs
Comment thread src/Quartz.Extensions.DependencyInjection/ServiceCollectionQuartzConfigurator.cs Outdated
Comment thread src/Quartz/ExecutionLimits.cs
Comment thread src/Quartz/TriggerBuilder.cs
Comment thread src/Quartz.Tests.Unit/ExecutionGroupsTest.cs
Comment thread docs/documentation/quartz-3.x/tutorial/execution-groups.md Outdated
- Always track execution group counts unconditionally so limits enabled
  at runtime see accurate in-flight counts immediately
- Downgrade missing EXECUTION_GROUP column log from Warn to Debug (only
  relevant when execution groups are actually used)
- Hoist rs.GetOrdinal() out of the per-row loop in SelectTriggerToAcquire
- Validate configure delegate is not null in UseExecutionLimits
- Reject reserved keys ("" and "*") in ForGroup/Unlimited with clear
  error message directing to ForDefaultGroup/ForOtherGroups
- Normalize empty/whitespace execution group to null in TriggerBuilder
- Fix test for "null" value alias and add coverage for "_" key alias

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 6 comments.

Comment thread src/Quartz/ExecutionLimits.cs Outdated
Comment thread src/Quartz/Impl/StdSchedulerFactory.cs
Comment thread src/Quartz/TriggerBuilder.cs Outdated
Comment thread src/Quartz/Core/QuartzSchedulerThread.cs
Comment thread docs/documentation/quartz-3.x/tutorial/execution-groups.md Outdated
Comment thread docs/documentation/quartz-3.x/tutorial/execution-groups.md Outdated
- Exclude default (null) group from OtherGroups fallback so configuring
  quartz.executionLimit.* = N does not implicitly cap ungrouped triggers
- Use INextVersionTrigger instead of AbstractTrigger in TriggerBuilder to
  avoid coupling public API to implementation types
- Remove zero-count entries from runningExecutionGroupCounts to prevent
  unbounded growth with high-cardinality group names
- Clarify docs: QRTZ_FIRED_TRIGGERS column is for forward compatibility
  only (not currently read/written), fix log level description

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Comment thread src/Quartz/Impl/StdSchedulerFactory.cs Outdated
Comment thread src/Quartz/Core/QuartzSchedulerThread.cs Outdated
…ult group

- Reject negative values in ParseExecutionLimits with SchedulerConfigException
  instead of letting ArgumentOutOfRangeException escape from the model
- Skip OtherGroups fallback for the default (ungrouped) key in
  ComputeAvailableExecutionGroupLimits, matching the CheckExecutionLimits
  semantics where "*" only applies to named groups

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Comment thread src/Quartz/Impl/AdoJobStore/IDriverDelegate.cs Outdated
Comment thread src/Quartz.Tests.Integration/Impl/SmokeTestPerformer.cs Outdated
- Restore original 3-parameter TriggerAcquireResult constructor as an
  overload forwarding to the new 4-parameter one, preserving binary
  compatibility for custom IDriverDelegate implementations
- Wrap execution limits API assertions in SmokeTestPerformer with
  try/catch for SchedulerException so the test works with remote proxy
  schedulers that don't support execution limits

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 3 comments.

Comment thread src/Quartz/Core/QuartzScheduler.cs
Comment thread src/Quartz/Core/QuartzSchedulerThread.cs Outdated
Comment thread src/Quartz.Extensions.DependencyInjection/IServiceCollectionQuartzConfigurator.cs Outdated
- GetExecutionLimits returns a snapshot through the public API path
  (StdScheduler) while the internal QuartzScheduler path stays
  zero-copy for the scheduler thread hot path
- Use INextVersionTrigger instead of AbstractTrigger in
  QuartzSchedulerThread for reading ExecutionGroup
- Move UseExecutionLimits from IServiceCollectionQuartzConfigurator
  interface to an extension method to preserve binary compatibility
  for external interface implementations

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 6 comments.

Comment thread src/Quartz.Extensions.DependencyInjection/ServiceCollectionQuartzConfigurator.cs Outdated
Comment thread src/Quartz/Simpl/RAMJobStore.cs
Comment thread src/Quartz/Impl/AdoJobStore/StdAdoDelegate.Triggers.cs
Comment thread src/Quartz/Impl/AdoJobStore/StdAdoDelegate.Triggers.cs
Comment thread src/Quartz/Impl/AdoJobStore/StdAdoDelegate.Triggers.cs
Comment thread docs/documentation/quartz-3.x/tutorial/execution-groups.md Outdated
- Replace all AbstractTrigger casts for ExecutionGroup with
  INextVersionTrigger in RAMJobStore, StdAdoDelegate (insert, update,
  select), keeping the feature decoupled from concrete trigger types
- Remove unused 'using Quartz.Impl' from ServiceCollectionQuartzConfigurator
- Clarify docs: ADO job stores need the EXECUTION_GROUP column for
  persistence; without it triggers appear ungrouped after restart

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated 3 comments.

Comment thread src/Quartz.Dashboard/Services/IQuartzApiClient.cs
Comment thread src/Quartz.Dashboard/Services/IQuartzApiClient.cs Outdated
Comment thread src/Quartz.Dashboard/Services/IQuartzApiClient.cs Outdated
- Move GetExecutionLimits from IQuartzApiClient to a separate
  IQuartzApiClientExecutionLimits interface to avoid breaking external
  implementations
- Change TriggerHeaderDto and CurrentlyExecutingJobDto to use init-only
  ExecutionGroup property instead of adding constructor parameters,
  preserving binary compatibility for existing consumers
- InProcessQuartzApiClient implements IQuartzApiClientExecutionLimits
- Use INextVersionTrigger instead of AbstractTrigger in dashboard client

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated 4 comments.

Comment thread src/Quartz/TriggerBuilder.cs
Comment thread src/Quartz/Impl/Triggers/AbstractTrigger.cs
Comment thread docs/documentation/quartz-3.x/configuration/reference.md
Comment thread src/Quartz.Tests.Unit/ExecutionGroupsTest.cs
- TriggerBuilder.WithExecutionGroup trims whitespace and rejects "*"
- AbstractTrigger.ExecutionGroup setter normalizes whitespace to null
  and trims valid values
- Document "null" as case-insensitive key alias for "_" (default group)
  in configuration reference, distinct from value "null" (unlimited)
- Add test for quartz.executionLimit.null key alias

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@sonarqubecloud

sonarqubecloud Bot commented Apr 4, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated no new comments.

@lahma
lahma merged commit be13972 into 3.x Apr 4, 2026
19 of 20 checks passed
@lahma
lahma deleted the execution-groups branch April 4, 2026 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants