SMTP connection configuration from_email is ignored by email_on_failure and email_on_retry #69262

Under which category would you file this issue?

Task SDK

Apache Airflow version

3.2.2

What happened and how to reproduce it?

Automatic task failure/retry emails sent through the email_on_failure and email_on_retry task options do not honor the SMTP connection extra from_email value.

The SMTP provider's SmtpNotifier supports falling back to the SMTP connection extra from_email when its own from_email argument is None, but the task runner always passes a non-None fallback value from [email] from_email or the hard-coded fallback airflow@airflow. As a result, the SmtpNotifier connection-extra fallback path is never reached for these automatic task error emails.

Steps to reproduce

1. Configure the SMTP provider as the email provider and use an SMTP connection for task email notifications.

2. Configure the SMTP connection extra with a sender address, for example:

   {
     "conn_type": "smtp",
     "extra": {
       "from_email": "[email protected]"
     }
   }

3. Do not set [email] from_email.

4. Create a task that uses automatic failure or retry emails, for example:

   EmptyOperator(
       task_id="failing_task",
       email=["[email protected]"],
       email_on_failure=True,
   )

5. Make the task fail.

6. Airflow trys to send the email from airflow@airflow

The reason is that task-sdk/src/airflow/sdk/execution_time/task_runner.py constructs the notifier with:

notifier = SmtpNotifier(
    to=to_emails,
    subject=subject,
    html_content=html_content,
    from_email=conf.get("email", "from_email", fallback="airflow@airflow"),
)

Since SmtpNotifier.from_email is never None, the following fallback in providers/smtp/src/airflow/providers/smtp/notifications/smtp.py is skipped:

if self.from_email is None:
    if smtp.from_email is not None:
        self.from_email = smtp.from_email
    else:
        raise ValueError("You should provide `from_email` or define it in the connection")
    fields_to_re_render.append("from_email")

What you think should happen instead?

Automatic task failure/retry emails sent using the SmtpNotifier should use the most specific configuration available.

I would expect the following outcomes based on the different config fields being set.

1. No [email] from_email config set, and from_email extra field set for the SMTP connection config: use from_email from the SMTP connection config (case from Steps to reproduce, not happening)

Other case for completeness reasons

2. No [email] from_email config set, and no from_email extra field set for the SMTP connection config: use the existing fallback airflow@airflow (this is already the case, altough I personally think an error would make more sense)

3. [email] from_email config set, and from_email extra field not set for the SMTP connection config: use the [email] from_email config. (this is already the case)

4. Both [email] from_email and the SMTP connection extra from_email are set: use from_email from the SMTP connection config, because the more specific config should take precedence over the more general config (currently the more general config [email] from email is used)

While it is debatable if all of my expected outcomes make sense, I would argue that the current configuration can cause quite a lot of confusion as it did for me, because:

• Airflow 3 uses the SmtpNotifier for sending email triggered by email_on_failure and email_on_retry

• The SMTP provider changelog for 2.0.0 explicitely states:

  You can instead use the SMTP connection configuration to set the default values, where you can use:

  the connection extra field from_email instead of the configuration smtp.smtp_mail_from in SmtpNotifier.

Operating System

No response

Deployment

None

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

I know that both task options email_on_failure and email_on_retry will be deprecated in Airflow 4 (e.g. see this PR). Nonetheless I thought I should create this issue as Airflow 3 still will be around for quite some time.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct