-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Description
Description
We currently have integration between web UI and Elasticsearch. It was added by this PR: #5164

This allows easier access to the Elasticsearch interface. We recently added a task handler that allows you to save logs to the Stackdriver service
#6660
I would be happy if a similar button also appears when StackdriverTaskHandler is used. Stackdriver has a much more friendly interface than Airflow, so access to it would be very helpful. It is also a fully managed service, so anyone can use it very easily.
To generate the Stackdriver URL, you can use the following code:
LOG_VIEWER_BASE_URL = "https://console.cloud.google.com/logs/viewer"
@property
def _resource_path(self):
segments = [self.resource.type]
for key, value in self.resource.labels:
segments += [key]
segments += [value]
return "/".join(segments)
def get_external_log_url(self, task_instance: TaskInstance, try_number: int) -> str:
"""
Creates an address for an external log collecting service.
:param task_instance: task instance object
:type: task_instance: TaskInstance
:param try_number: task instance try_number to read logs from.
:type try_number: Optional[int]
:return: URL to the external log collection service
:rtype: str
"""
project_id = self._client.project
ti_labels = self._task_instance_to_labels(task_instance)
ti_labels[self.LABEL_TRY_NUMBER] = str(try_number)
log_filter = self._prepare_log_filter(ti_labels)
url_query_string = {
'project': project_id,
'interval': 'NO_LIMIT',
'resource': self._resource_path,
'advancedFilter': log_filter,
}
url = f"{self.LOG_VIEWER_BASE_URL}?{urllib.parse.urlencode(url_query_string)}"
return urlBefore making this change, I think it's worth adding the missing tests PR, which added external links for Elasticsearch does not introduce any new tests, so it is advisable to supplement them before starting work.
You can use capture_templates method to do it.
#8505
If anyone is interested in this task, I am willing to provide all the necessary tips and information.
If you haven't used the GCP yet, after creating the account you will get $300, which will allow you to test change and get to know this service better.