Apache Airflow version
2.6.3
What happened
A task instance's log_url does not contain the full URL defined in base_url.
What you think should happen instead
The base_url may contain paths that should be acknowledged when build the log_url.
The log_url is built with urljoin. Due to how urljoin builds URLs, any existing paths are ignored leading to a faulty URL.
How to reproduce
This snippet showcases how urljoin ignores existing paths when building the url.
>>> from urllib.parse import urljoin
>>>
>>>
>>> urljoin(
... "https://my.astronomer.run/path",
... f"log?execution_date=test"
... f"&task_id=wow"
... f"&dag_id=super"
... f"&map_index=-1",
... )
'https://my.astronomer.run/log?execution_date=test&task_id=wow&dag_id=super&map_index=-1'
Operating System
n/a
Versions of Apache Airflow Providers
No response
Deployment
Astronomer
Deployment details
No response
Anything else
This was introduced by #31833.
A way to fix this can be to utilize urlsplit and urlunsplit to account for existing paths.
from urllib.parse import urlsplit, urlunsplit
parts = urlsplit("https://my.astronomer.run/paths")
urlunsplit((
parts.scheme,
parts.netloc,
f"{parts.path}/log",
f"execution_date=test"
f"&task_id=wow"
f"&dag_id=super"
f"&map_index=-1",
""
)
)
Here is the fix in action.
>>> parts = urlsplit("https://my.astronomer.run/paths")
>>> urlunsplit((
... parts.scheme,
... parts.netloc,
... f"{parts.path}/log",
... f"execution_date=test"
... f"&task_id=wow"
... f"&dag_id=super"
... f"&map_index=-1",
... ''))
'https://my.astronomer.run/paths/log?execution_date=test&task_id=wow&dag_id=super&map_index=-1'
>>>
>>> parts = urlsplit("https://my.astronomer.run/paths/test")
>>> urlunsplit((
... parts.scheme,
... parts.netloc,
... f"{parts.path}/log",
... f"execution_date=test"
... f"&task_id=wow"
... f"&dag_id=super"
... f"&map_index=-1",
... ''))
'https://my.astronomer.run/paths/test/log?execution_date=test&task_id=wow&dag_id=super&map_index=-1'
Are you willing to submit PR?
Code of Conduct
Apache Airflow version
2.6.3
What happened
A task instance's log_url does not contain the full URL defined in base_url.
What you think should happen instead
The base_url may contain paths that should be acknowledged when build the log_url.
The log_url is built with urljoin. Due to how urljoin builds URLs, any existing paths are ignored leading to a faulty URL.
How to reproduce
This snippet showcases how urljoin ignores existing paths when building the url.
Operating System
n/a
Versions of Apache Airflow Providers
No response
Deployment
Astronomer
Deployment details
No response
Anything else
This was introduced by #31833.
A way to fix this can be to utilize urlsplit and urlunsplit to account for existing paths.
Here is the fix in action.
Are you willing to submit PR?
Code of Conduct