Skip to content

Email: render template before sending it to the task - #9538

Merged
stsewd merged 10 commits into
mainfrom
render-template-before-task
Oct 4, 2022
Merged

stsewd merged 10 commits into
mainfrom
render-template-before-task

Conversation

@stsewd

@stsewd stsewd commented Aug 23, 2022 •

Copy link
Copy Markdown
Member

The txt formats are the same as the html ones,
but with tags removed and small changes to include
links along the text.

Closes #9466


📚 Documentation previews 📚

@stsewd
stsewd requested review from a team as code owners August 23, 2022 16:17
@stsewd
stsewd requested review from agjohnson and humitos August 23, 2022 16:17
@stsewd
stsewd force-pushed the render-template-before-task branch from ed93d2b to 8642e5a Compare August 23, 2022 16:20
@stsewd
stsewd marked this pull request as draft August 23, 2022 16:39
stsewd added 3 commits August 23, 2022 13:59
The txt formats are the same as the html ones,
but with tags removed and small changes to include
links along the text.

Closes #9466
@stsewd
stsewd force-pushed the render-template-before-task branch from 7b8854a to d5307af Compare August 23, 2022 19:00
@stsewd
stsewd marked this pull request as ready for review August 23, 2022 19:04
self.assertEqual(mock_send_email.call_count, 1)
mock_send_email.assert_has_calls([
mock.call(
request=mock.ANY,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the request parameter isn't being used at all withing the send email function or task, I think we should just remove it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏼 on removing it, then.

@humitos humitos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine. I left some comments to talk a little more before merging, but I think this is good work and going in the right direction.

Comment on lines -545 to -546
# We send only what we need from the Django model objects here to avoid
# serialization problems in the ``readthedocs.core.tasks.send_email_task``

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are not doing this anymore, we should probably also stop creating the context dictionary, and just send the Django objects instead. Then, from the other function we can grab whatever we need from these objects.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are not doing this anymore, we should probably also stop creating the context dictionary, and just send the Django objects instead.

Do you mean just for this case? If you mean for everything, we still need to pass some additional things in the context that aren't in just one object.

Comment thread readthedocs/core/tasks.py Resolved
self.assertEqual(mock_send_email.call_count, 1)
mock_send_email.assert_has_calls([
mock.call(
request=mock.ANY,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏼 on removing it, then.

@agjohnson agjohnson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Reminder that there might be emails hiding in ext as well, if you haven't already looked.

I'll let @humitos give the 👍 on this, as you'll be closer to release and QA, but seems like a better approach. These emails were all templated and sent from a task to avoid synchronous delays with web requests. So, just doing the templating step before inserting into our task queue doesn't seem like a huge change we need to be worried about.

@humitos

humitos commented Aug 25, 2022

Copy link
Copy Markdown
Member

@agjohnson

These emails were all templated and sent from a task to avoid synchronous delays with web requests.

This is a good point that I didn't think about. I don't think it will be a problem since these templates are small and I don't think they will take too much time to render that could produce a timeout on the web request.

@stsewd One thing that we should test before deploying this is if the Celery message inserted in the queue is not too big due the whole HTML + txt files are going to be on the message itself. Does Celery have a setting for the max size of the message that we can check?

@stsewd

stsewd commented Aug 25, 2022

Copy link
Copy Markdown
Member Author

Does Celery have a setting for the max size of the message that we can check?

Didn't find a setting related to that

@agjohnson

Copy link
Copy Markdown
Contributor

I don't think it will be a problem since these templates are small and I don't think they will take too much time to render that could produce a timeout on the web request.

Aye, I feel this is safe as well, though don't quote me on this either. At some point we must have had some variable that required an external lookup or something that would block the web request during template render.

@humitos

humitos commented Sep 6, 2022

Copy link
Copy Markdown
Member

@stsewd @agjohnson are we happy moving forward with this PR?

@stsewd have you tested this locally to check the rendered template does fit on the Celery queue without problems?

@stsewd

stsewd commented Sep 6, 2022

Copy link
Copy Markdown
Member Author

@stsewd have you tested this locally to check the rendered template does fit on the Celery queue without problems?

It works locally.

@stsewd

stsewd commented Sep 6, 2022

Copy link
Copy Markdown
Member Author

Another approach would be to change the notifications classes to pass the attributes of the object to be used in the context instead of the object itself, but that feels more error-prone than just passing the object.

@agjohnson

agjohnson commented Sep 6, 2022 •

Copy link
Copy Markdown
Contributor

I don't have any objections, I was just raising a note about the performance. If you all feel this is safe, I'd agree this shouldn't be an issue.

@humitos brings up a good point on the Celery internals. I'm not aware of a Celery specific limit here, but we are using AWS Elasticache in production (as opposed to Redis in development). Elasticache, or Elasticache + Celery, could have different limits here.

@benjaoming

Copy link
Copy Markdown
Contributor

Re: queue object and size limits

The rendered email content is likely smaller than a Django context object, since it for instance contains request from django.template.context_processors.request.

If there is a known limit to celery task data, it would be nice of Celery to raise that exception. Moreover, Redis and RabbitMQ both have a maximum value size of 512 MB: https://stackoverflow.com/a/5624569/405682

I was reading a bit about Kombu, and I think there's some "Quality of Service" involved when a worker receives messages as well. For all I can tell, these message brokers and task queues are designed to handle significantly larger task data than some text email.

I don't think this is an issue -- and if it is, then we might find more issues.

@stsewd
stsewd merged commit bfef827 into main Oct 4, 2022
@stsewd
stsewd deleted the render-template-before-task branch October 4, 2022 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Email: render template before sending it to the task

4 participants