Description
Right now, the get_log endpoint at /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/logs/{task_try_number} does not have any pagination and therefore we can be forced to load extremely large text blocks, which makes everything slow. (see the workaround fix we needed to do in the UI: #29390)
In task_log_reader, we do have log_pos and offset (see here). It would be great to expose those parameters in the REST API in order to break apart task instance logs into more manageable pieces.
Use case/motivation
No response
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
Description
Right now, the
get_logendpoint at/dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/logs/{task_try_number}does not have any pagination and therefore we can be forced to load extremely large text blocks, which makes everything slow. (see the workaround fix we needed to do in the UI: #29390)In
task_log_reader, we do havelog_posandoffset(see here). It would be great to expose those parameters in the REST API in order to break apart task instance logs into more manageable pieces.Use case/motivation
No response
Related issues
No response
Are you willing to submit a PR?
Code of Conduct