Checks

• I have checked for existing issues.
• This report is about the User-Community Airflow Helm Chart.

Chart Version

8.6.0

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6+vmware.1", GitCommit:"088f01db2ffab397a290be443902918b59ee032c", GitTreeState:"clean", BuildDate:"2021-04-17T01:01:00Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}

Helm Version

version.BuildInfo{Version:"v3.6.1", GitCommit:"61d8e8c4a6f95540c15c6a65f36a6dd0a45e7a2f", GitTreeState:"clean", GoVersion:"go1.16.5"}

Description

We use NAS as a storage for our logs for every instance of Airflow we use. We mount this storage as a volume:

extraVolumes:
  - name: nfs-volume
    nfs:
      server: myserver.com
      path: /path/to/folder/on/nfs

with volumeMount defined as follows:

extraVolumeMounts:
  - name: nfs-volume
    mountPath: /opt/airflow/storage

Then this mountPath is being used as our logs path:

logs:
  path: /opt/airflow/storage/logs

This setup worked flawlessly up until 8.6.0 upgrade when we suddenly 'lost' all of our older logs from the UI. New ones were created and displayed successfully. We investigated why is that happening and found out that log cleanup functionality introduced in 8.6.0 is basically overriding our setup with logs being created in workers/schedulers.

Even if we disabled this functionality on a values level with hope it reverts back, unfortunately it still didn't work. Then we investigated and with kubectl describe we finally saw that our 'nfs-volume' mount is being ignored (replaced) by logs-data storage which is basically an empty dir:

/charts/airflow/templates/_helpers/pods.tpl

{{/*
The list of `volumeMounts` for web/scheduler/worker/flower container
EXAMPLE USAGE: {{ include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
*/}}
{{- define "airflow.volumeMounts" }}

...

{{- /* logs */ -}}
{{- if .Values.logs.persistence.enabled }}
# doesn't apply for us as we don't use PV's for logs
{{- else }}
- name: logs-data
  mountPath: {{ .Values.logs.path }}
{{- end }}
{{- end }}

{{/*
The list of `volumes` for web/scheduler/worker/flower Pods
EXAMPLE USAGE: {{ include "airflow.volumes" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumes" $extraVolumes) }}
*/}}
{{- define "airflow.volumes" }}

...

{{- /* logs */ -}}
{{- if .Values.logs.persistence.enabled }}
# doesn't apply for us as we don't use PV's for logs
{{- else }}
- name: logs-data
  emptyDir: {}
{{- end }}
{{- end }}

From these settings it's visible that logs-data mount is effectively replacing {{ .Values.logs.path }} with an emptyDir, thus 'destroying' our NAS volume mount.

Maybe our use case is not very common, so it's not being considered when developing the chart, but we see this NAS volume scenario as a valid and useful one for many reasons. So we consider this issue as a bug unfortunately and needed to remove all 'log cleanup' related lines from the chart to to make it work again.

Do you see anything we might be doing wrong here or is our external storage use case just not supported anymore?

Apart from that we love the chart, using it almost from the beginning and this is the first issue we have with it. Thank you!

Relevant Logs

No response

Custom Helm Values

No response