What issues does your PR fix?

• related to Scheduler cpu usage. #512
  • (NOTE: this PR probably does not fix that issue fully)
• supersedes feat: make scheduler livenessProbe use less resources #541

What does your PR do?

• Adds an optional "task creation check" to the scheduler liveness probe:
  • scheduler.livenessProbe.taskCreationCheck.enabled (default: false)
  • scheduler.livenessProbe.taskCreationCheck.thresholdSeconds (default: 300)
• Adds a new FAQ "How to configure the scheduler liveness probe?" with sub-sections:
  • Scheduler "Heartbeat Check"
  • Scheduler "Task Creation Check"
• Runs the scheduler livenessProbe with /entrypoint:
  • NOTE: this ensures any fixes inside /entrypoint are applied in our livenessProbe
  • NOTE: this improves support for running as root (airflow python packages will not be visible otherwise)
• Makes the scheduler livenessProbe use fewer resources by:
  1. Only querying the .first() instance of SchedulerJob from the database
  2. Setting airflow log-level to ERROR
  3. Running python under dumb-init (so processes are cleaned up)

Checklist

For all Pull Requests

• Commits are signed off
• Commits have semantic messages
• Documentation updated
• Passes ct linting

For releasing ONLY

• Chart.yaml version bumped
• CHANGELOG.md updated