This is nice, thank you @kunalyogenshah ! Users using Airflow with Elasticsearch backend do likely want to utilize more those aggregation features Elasticsearch frontends, e.g. Kibana, provide.

Before looking into the code, my opinion on the UI change is that we make the layout similar to how we do the log downloading--we allow users to choose the try number of the task instances. Do you think that make sense?

@kunalyogenshah , not sure if this is the right way to show the UI. I think another approach(Lyft) is to leverage #5059 and #5094 which allows you to customize your task instance panel in generic way.

And we have already did that at Lyft in prod for quite some time(https://eng.lyft.com/running-apache-airflow-at-lyft-6e53bb8fccff). cc @milton0825

Ah. Thanks @feng-tao . This seems to be exactly similar to what we've been trying to do. Let me look into using extra links instead.

@feng-tao : A quick naive clarification question. Does this mean that the DAG operator itself must specify the link to Kibana as an extra link in its operator definition? Or is there a way to make that universal for all DAGs and tasks?

@kunalyogenshah I think this is what you are looking for: #5094
We can register an extra link that is global to all the operators.

+1 for @milton0825 , cc @Fokko as well

Perfect. Thanks @milton0825 . I'll move my questions over to that PR for better context :)

Bringing this thread back up to share some alternate thoughts. I'd like to advocate for this approach specifically for just logs look up for the following reasons :

• Using elasticsearch for now is quite well baked into the log handling, and Kibana is possibly the most common frontend for this. Using that as a frontend for visualizing / searching logs seems like quite a useful central feature to have as compared to a plugin. Semantically we'd use extra links plugin for features that are more specific to each individual Airflow DAG / Task / Provider.
• This can be quite easily integrated and extended like the existing download logs / view logs, and seems on the same plane w.r.t. utility.
• Also referring to the proposed metrics changes in this PR ([AIRFLOW-4360] Add performance metrics for Webserver UI. #5139), having a standalone endpoint for serving logs allows us to track metrics for just that endpoint better than a common extra links section. (like tracking views and failures, etc).

Thoughts?

@kunalyogenshah I am thinking about making extra_links aware of try-number. Once that is done, we can probably consolidate all the download logs and view logs and register them through extra_links. download logs and view logs will by default be there and can be disabled through airflow.cfg. What do you think?

I am fine with shipping this PR now and rework later to support what I proposed.

Codecov Report

Merging #5164 into master will decrease coverage by 0.02%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master    #5164      +/-   ##
==========================================
- Coverage   78.67%   78.65%   -0.03%     
==========================================
  Files         470      470              
  Lines       30026    30044      +18     
==========================================
+ Hits        23623    23631       +8     
- Misses       6403     6413      +10

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e187625...b599d8b. Read the comment docs.

Yeah, I tested this by just changing the cfg file to include the frontend URL. The frontend URL I used is also included in the CFG file.

Tailored example URL from default cfg file. Caused parsing errors for the default file. For Kibana specifically, the format I used in testing was :

elasticsearch_frontend = <Domain>/_plugin/kibana/app/kibana#/discover?_g=()&_a=(filters:!((query:(match:(log_id:(query:'{log_id}',type:phrase))))))

@kunalyogenshah This features is very interesting. Can you write documentation for this feature? The description in the configuration file is very little visible. I think you should add a note to the file: doc/howto/write-logs.rst.

Hey @mik-laj . I can definitely write documentation if needed. Will send a followup PR.

@kaxil can this PR cheery picked to 1.10.4?

I am thinking about making extra_links aware of try-number. Once that is done, we can probably consolidate all the download logs and view logs and register them through extra_links. download logs and view logs will by default be there and can be disabled through airflow.cfg. What do you think?

I am fine with shipping this PR now and rework later to support what I proposed.

@milton0825 Yes please, that would be great.

@KevinYang21 When you merge PRs can you double check the commit message correctly references the jira - the release scripts I use to make sure we get everything in to the release need it. In this case it was [AIRFLOW] rather than [AIRFLOW-4396] - you can edit the commit message in the github UI when merging. Thanks