Currently logs are printed to stdout only and it's truncated in the notebook cell. We need a second logging destination for troubleshooting

This problem persists in python wheel tasks as well:

use {{job_id}} and {{run_id}} and {{task_key}} as injected parameters to distinguish the logs.

https://docs.databricks.com/en/workflows/jobs/task-parameter-variables.html

use try ... finally ... to ensure the log file is flushed

https://docs.python.org/3/library/logging.handlers.html#filehandler