ActiveJob and retries API #34337

ActiveJob::Exceptions promotes the following API:

class RemoteServiceJob < ActiveJob::Base
  retry_on ActiveRecord::Deadlocked, attempts: 3
  retry_on Net::OpenTimeout, attempts: 10

It may seem that the job would be retried 10 times on Net::OpenTimeout and 3 times on ActiveRecord::Deadlocked - however, since the exceptions counter (#executions) is incremented globally but not per exception class, it's currently possible that:

1. First 3 times the job raises with Net::OpenTimeout, each time it's retried and executions gets incremented to 3
2. On 4th execution the job raises with ActiveRecord::Deadlocked but it won't be retried because executions (3) is greater than attempts for ActiveRecord::Deadlocked class.

IMO this side of the API can be misleading. It also doesn't provide a shortcut to declare "retry any exception N times" other than retry_on StandardError, attempts: 10 - which is what we'd want many jobs to do to enforce idempotency.

At Shopify we have our own Retry module with the following API:

class ShopifyJob < AJ::Base
  # we don't want the API to encourage zero retries for idempotency reasons so we offer "minimum" number of retries
  retries :minimum
end

class ShopifyJob < AJ::Base
  # in most of cases developer doesn't have to come up with a custom number of retries, so we support "default" number of retries per platform. In this example they'll be applied exponentially.
  retries :default, backoff: :exponential
end

class ShopifyJob < AJ::Base
  # 10 retries, each with 1 minute backoff
  retries 10, backoff: [1.minute] * 50
end

class ShopifyJob < AJ::Base
  retries 10, backoff: :exponential
  def perform
    # ...
  rescue RecordAlreadyExists
    # don't retry this exception by "swallowing it"
  end
end

class ShopifyJob < AJ::Base
  retries 10, backoff: :exponential
  def perform
    # ...
  rescue StandardError => e
    raise if e == Billing::BackendBusyRetryLaterError
    # retry only one kind of exception, swallow the rest
  end
end

Some parts of this API are more flexible than ActiveJob::Exceptions but some are less expressive (especially retrying only one exceptions class).

I'd like to discuss options how we could merge the best parts of both APIs and address the misleading parts of ActiveJob::Exceptions, and at the same time push all Rails developers to opt-in for idempotency by default.