Skip to content

Enqueuing jobs inside a job can lead to memory bloat #21036

@pixeltrix

Description

@pixeltrix

On the UK Parliament and Government petitions website we're queuing up emails to be sent using a background job and deliver_later so that the email sending is not limited to one worker thread. Essentially the code looks somewhat like this:

class EmailSignees < ActiveJob::Base
  def perform(petition)
    petition.signatures.find_each do |signature|
      PetitionMailer.notify_signee_of_response(petition, signature),deliver_later
    end
  end
end

The problem is that even though we are using find_each, because Active Job uses a single log subscriber for both enqueue and perform events the deliver_later call gets treated as a child event of the performevent so eventually we end up with 100,000+ objects in the event object even though we're using find_each. This obviously causes the worker to eventually crash due to a lack of memory.

The questions are:

  1. is this behaviour intentional?
  2. do we want to keep this behaviour, given the above scenario?
  3. if it is intentional and we want to keep it, what's the solution?

For question 3 there are multiple ways of working round it I can think of, just wondering what our recommendation would be.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions