Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.6.2

What happened?

When opening task log from grid menu - loading animation is played and logs won't show. Webserver logs show:
webserver [2024-04-30T09:51:42.825+0000] {base.py:274} DEBUG - < {"took":8,"timed_out":false,"_shards":{"total":53,"successful":53,"skipped":51,"failed":0},"hits":{"total":{"value":17,"relation":"eq"},"max_score":null,"hits":[{"_index": .......
Here I can see all the actual logs that are in elastic an then webserver just bombards elastic with:
webserver [2024-04-30T09:48:34.688+0000] {base.py:270} INFO - POST https://elastic:9200/_all/_count [status:200 request:0.009s] webserver [2024-04-30T09:48:34.688+0000] {base.py:273} DEBUG - > {"query":{"bool":{"filter":[{"range":{"offset":{"gt":1714470190001012992}}}],"must":[{"match_phrase":{"log_id":"Redis_Populate_Cities_populate_redis_cities_2024_04_30T09_43_06_661556_1"}}]}}} webserver [2024-04-30T09:48:34.689+0000] {base.py:274} DEBUG - < {"count":0,"_shards":{"total":53,"successful":53,"skipped":0,"failed":0}}

Here is my config from values.yaml regarding elastic:

elasticsearch:
  enabled: true
  secretName: airflow-elastic-secret
config:
  elasticsearch:
    end_of_log_mark: end_of_log
    json_fields: asctime, filename, lineno, levelname, message
    json_format: "True"
    log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"
    write_stdout: "True"
  elasticsearch_configs:
    max_retries: 2
    retry_timeout: "True"
    timeout: 10
    use_ssl: "True"
    verify_certs: "False"

What you think should happen instead?

No response

How to reproduce

Can be reproduced by clicking dag name in ui and then from grid selecting task and selecting task's logs:

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow==2.6.2
apache-airflow-providers-amazon==8.1.0
apache-airflow-providers-celery==3.2.0
apache-airflow-providers-cncf-kubernetes==7.0.0
apache-airflow-providers-common-sql==1.5.1
apache-airflow-providers-docker==3.7.0
apache-airflow-providers-elasticsearch==4.5.0
apache-airflow-providers-ftp==3.4.1
apache-airflow-providers-google==10.1.1
apache-airflow-providers-grpc==3.2.0
apache-airflow-providers-hashicorp==3.4.0
apache-airflow-providers-http==4.4.1
apache-airflow-providers-imap==3.2.1
apache-airflow-providers-microsoft-azure==6.1.1
apache-airflow-providers-microsoft-mssql==3.4.1
apache-airflow-providers-mongo==3.2.1
apache-airflow-providers-mysql==5.1.0
apache-airflow-providers-odbc==3.3.0
apache-airflow-providers-postgres==5.5.0
apache-airflow-providers-redis==3.2.0
apache-airflow-providers-sendgrid==3.2.0
apache-airflow-providers-sftp==4.3.1
apache-airflow-providers-slack==7.3.0
apache-airflow-providers-snowflake==4.1.0
apache-airflow-providers-sqlite==3.4.1
apache-airflow-providers-ssh==3.7.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Airflow is deployed with official helm chart with changes in values.yaml Dockerfile is customized with pip installs
Please let me know if any additional info is required

Anything else?

Couple of caveats:

1. When selecting "See More" log is shown.
2. Log also can be viewed from Graph - Task - Log. (But here i have another bug - my logs doesnt contain end_of_log so it keeps polling elastic.)
3. When dag is in running state - i can see its log in grid view.
4. It's only pending when there is a log in elastic (and i can see it with right log_id ) if there is no log in elastic - message will show Log not found in Elasticsearch... etc

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct