Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

When submitting spark / flink task to Kubernetes, we collect driver pod logs after task execution. There are two potential problems with this way:

• Collect large amounts of pod logs will cause OOM.
• User can not read pod log in-time, this problem can be exaggerated especially for long-running task.

Thus, we should support real-time pod log collection.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct