Right now, if worker/scheduler pods run for long periods of time, they may accumulate so many logs under logs.path that they may cause issues.

We should allow users to deploy a sidecar that is responsible to truncate log files after some period of time.

Here are some example values for what this feature may look like:

scheduler:
  logCleanup:
    # if the sidecar container is added to the scheduler Pod (default: true)
    enabled: true

    # resources for the ContainerSpec
    resources: {}

    ## the number of minutes to retain log files (by last-modified time)
    ##
    retentionMinutes: 21600

    ## the number of seconds between each check for files to delete
    ##
    intervalSeconds: 900

workers:
  logCleanup:
    ## SAME AS ABOVE

WARNING: we should NOT allow {scheduler,workers}.logCleanup.enabled and logs.persistence.enabled to be true at the same time (using ./_helpers/validate-values.tpl) as multiple of the sidecars may fight with each other. In the error message, tell users that they could create an airflow job instead.