-
Notifications
You must be signed in to change notification settings - Fork 22.2k
Description
On the UK Parliament and Government petitions website we're queuing up emails to be sent using a background job and deliver_later so that the email sending is not limited to one worker thread. Essentially the code looks somewhat like this:
class EmailSignees < ActiveJob::Base
def perform(petition)
petition.signatures.find_each do |signature|
PetitionMailer.notify_signee_of_response(petition, signature),deliver_later
end
end
endThe problem is that even though we are using find_each, because Active Job uses a single log subscriber for both enqueue and perform events the deliver_later call gets treated as a child event of the performevent so eventually we end up with 100,000+ objects in the event object even though we're using find_each. This obviously causes the worker to eventually crash due to a lack of memory.
The questions are:
- is this behaviour intentional?
- do we want to keep this behaviour, given the above scenario?
- if it is intentional and we want to keep it, what's the solution?
For question 3 there are multiple ways of working round it I can think of, just wondering what our recommendation would be.