You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Use private docker repository with K8S operator and XCOM sidecar container #10605
I've submitted a PR, do nopt hesitate to add comment or if I've implemented this badly.
The sidecar container declaration is low level in K8S operator so I've tried to not add complexity.
Hi @mickaelgervais
To understand your issue better, are you having trouble turning on xcom because 'alpine' give you 403 error? Because in your description, you are not able to use private registry, but in my case, I can pass the whole string to the image in my KubernetesPodOperator like below. I just have problem with the xcom-sidecar, because class PodDefaults hard coded image name to be 'alpine'. image="<private-registry>/<dir>/<dir>/<image>:<tag>"
I have left some comments in your pr for adding a explicit image tag, we have something like private-registry/alpine:3.10, and would like to add a variable for image tags.
A work around would be to add this code block in the /home/airflow/airflow_local_settings.py
PodDefaults.SIDECAR_CONTAINER = k8s.V1Container(
name=PodDefaults.SIDECAR_CONTAINER_NAME,
command=['sh', '-c', PodDefaults.XCOM_CMD],
image='',
volume_mounts=[PodDefaults.VOLUME_MOUNT],
resources=k8s.V1ResourceRequirements(
requests={
"cpu": "10m",
"memory":"32Mi"
},
limits={
"cpu": "0.25",
"memory":"32Mi"
}
),
)
Is there a reason why this has not been implemented yet?
It was waiting for someone like you to take that over and complete it possibly? Airflow has > 1700 contributors and you can become one if you want some features to be implemented faster :)
Feel free to take it over and contribute! we even have Contributors workshops (3 of them till the end of the year). On Airflow Slack there is a #contributors-workshop dedicated channel for it! Anyone is welcome!
sure, I think it's covered in a different feature request, let me find out
and update the comments.
Can you let me know when are the workshops and how do I attend them?
Thanks,
Zhicong
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
You can verify the custom airflow_local_settings.py file is loaded because any code written in that file will be executed by teach task instance, i.e., those are the logs to check. That's really the only annoying/hard part, is just knowing whether your instance is actually executing/including the local_settings file.
Also note that this works even as early as at least 2.1.0 for those of you who are having issues with 2.2.0 :)
Hi @ZackingIt , could I know how do you overrote the code? Do you go to pod and change the airflow_local_settings.py file in root/config folder or you add it in values.yaml file as below:
Your code
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
in values.yaml
airflow
localSettings:
## the full content of the `airflow_local_settings.py` file (as a string)
stringOverride: |
# use a custom `xcom_sidecar` image for KubernetesPodOperator()
from airflow.kubernetes.pod_generator import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "private_artifactory/alpine"
@dzungdev
I created an actual file, I did not overwrite a variable in the values.yaml i.e., part of my Docker build involves: cp /airflow-config/config/airflow_local_settings.py ${AIRFLOW_HOME}/config/airflow_local_settings.py
When I say "due to this yaml" I mean that that yaml tells us that this is already a location,${AIRFLOW_HOME}/config is already one of the paths being read by default, as I did not need to explicitly set a PYTHONPATH environment variable to read that location.
@ZackingIt
Where exactly did you put the configuration? When I put the PodDefaults.SIDECAR_CONTAINER.image (as mentioned several times here and everywhere else) into the values.yaml it is just used by the helm chart. But I can't figure out where to put the airflow_local_settings.py to configure the worker pod image pulled by the KubernetesExecutor. I even can't build a Docker image with Airflow 2.1.4 which is configured as needed.
My current Dockerfile looks like the following:
FROM apache/airflow:2.1.4-python3.8@sha256:ae98e5e9eba596e24bce820e5f3a109274b82cd347bb9fdfca1e9d3f0dda4bb0
COPY --chown=airflow:root airflow_local_settings.py /opt/airflow/config/airflow_local_settings.py
USER airflow
But as soon as the KubernetesExecutor is pulling my image (because I configured it in the values.yaml) it runs into the Docker pull limit very easy, because I can't configure the image correctly.
We are facing the same issue, but in our case the error is related the absent of an option to define resources to the sidecar container. Our kubernetes cluster has this resource enforcement active and them even trying to set the resouces by airflow_local_settings.py, pod_template_file, it sounds like the code is ignoring these options.
Here is the workaround (tested with Airflow 2.1.4) if you are using airflow helm chart:
values.yaml:
airflowLocalSettings: |
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
But where do I provide the Image Pull Secret for the private registry in the values.yaml? So far I can't find anything and would really appreciate advice @fritzb?
Incase anyone else is encountering this issue still and not using helm, you can update the image value without have to recreate the whole V1Container. I put this in my custom EKSPodOperator. FYI, using this on MWAA.
if PodDefaults.SIDECAR_CONTAINER.image != "<your-custom-image>":
PodDefaults.SIDECAR_CONTAINER.image = "<your-custom-image>"
Here is the workaround (tested with Airflow 2.1.4) if you are using airflow helm chart:
values.yaml:
airflowLocalSettings: |
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
But where do I provide the Image Pull Secret for the private registry in the values.yaml? So far I can't find anything and would really appreciate advice @fritzb?
Did you ever find an answer to this? I'm still struggling to use an image from our private repository for the sidecar.
Activity
boring-cyborg commentedon Aug 27, 2020
Thanks for opening your first issue here! Be sure to follow the issue template!
potiuk commentedon Aug 28, 2020
would you like to add this feature @mickaelgervais ? I am happy to review it
mickaelgervais commentedon Aug 31, 2020
Hi @potiuk ,
I've submitted a PR, do nopt hesitate to add comment or if I've implemented this badly.
The sidecar container declaration is low level in K8S operator so I've tried to not add complexity.
Regards
zoomzoomTnT commentedon Sep 3, 2020
Hi @mickaelgervais
To understand your issue better, are you having trouble turning on xcom because 'alpine' give you 403 error? Because in your description, you are not able to use private registry, but in my case, I can pass the whole string to the image in my KubernetesPodOperator like below. I just have problem with the xcom-sidecar, because class PodDefaults hard coded image name to be 'alpine'.
image="<private-registry>/<dir>/<dir>/<image>:<tag>"I have left some comments in your pr for adding a explicit image tag, we have something like private-registry/alpine:3.10, and would like to add a variable for image tags.
Hi @potiuk
Is this feature going live in 1.10.13 or later versions?
Shivarp1 commentedon Sep 7, 2020
A work around would be to add this code block in the /home/airflow/airflow_local_settings.py
PodDefaults.SIDECAR_CONTAINER = k8s.V1Container(
name=PodDefaults.SIDECAR_CONTAINER_NAME,
command=['sh', '-c', PodDefaults.XCOM_CMD],
image='',
volume_mounts=[PodDefaults.VOLUME_MOUNT],
resources=k8s.V1ResourceRequirements(
requests={
"cpu": "10m",
"memory":"32Mi"
},
limits={
"cpu": "0.25",
"memory":"32Mi"
}
),
)
fritzb commentedon Nov 21, 2020
Docker Inc started image pull rate limiting around October 30th 2020 (https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status/). As a result, all KubernetesPodPerator tasks with XCOM started to fail in large deployment. Can we have the fix in Airflow 1.10 branch as well?
Based on the severity of the failure, the bug priority should be raised higher.
evanhlavaty commentedon Sep 22, 2021
Is there a reason why this has not been implemented yet?
potiuk commentedon Sep 22, 2021
It was waiting for someone like you to take that over and complete it possibly? Airflow has > 1700 contributors and you can become one if you want some features to be implemented faster :)
potiuk commentedon Sep 22, 2021
Feel free to take it over and contribute! we even have Contributors workshops (3 of them till the end of the year). On Airflow Slack there is a #contributors-workshop dedicated channel for it! Anyone is welcome!
zoomzoomTnT commentedon Sep 22, 2021
fritzb commentedon Sep 23, 2021
Here is the workaround (tested with Airflow 2.1.4) if you are using airflow helm chart:
values.yaml:
potiuk commentedon Sep 25, 2021
Just join the #contributors-workshop channel - we announce the workshops there.
evanhlavaty commentedon Oct 12, 2021
@fritzb
I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
ZackingIt commentedon Nov 2, 2021
I was able to get my xcom to use a non-DockerHub repo also by creating a
airflow_local_settings.pyfile in the root/config folder, due to this yaml:airflow/chart/templates/_helpers.yaml
Line 416 in 73044af
I then simply overwrote with the code:
You can verify the custom
airflow_local_settings.pyfile is loaded because any code written in that file will be executed by teach task instance, i.e., those are the logs to check. That's really the only annoying/hard part, is just knowing whether your instance is actually executing/including the local_settings file.Also note that this works even as early as at least 2.1.0 for those of you who are having issues with 2.2.0 :)
dzungdev commentedon Nov 11, 2021
Hi @ZackingIt , could I know how do you overrote the code? Do you go to pod and change the airflow_local_settings.py file in root/config folder or you add it in values.yaml file as below:
Your code
in values.yaml
ZackingIt commentedon Nov 11, 2021
@dzungdev
I created an actual file, I did not overwrite a variable in the
values.yamli.e., part of my Docker build involves:cp /airflow-config/config/airflow_local_settings.py ${AIRFLOW_HOME}/config/airflow_local_settings.pyWhen I say "due to this yaml" I mean that that yaml tells us that this is already a location,
${AIRFLOW_HOME}/configis already one of the paths being read by default, as I did not need to explicitly set a PYTHONPATH environment variable to read that location.lshw42 commentedon Nov 14, 2021
@ZackingIt
Where exactly did you put the configuration? When I put the
PodDefaults.SIDECAR_CONTAINER.image(as mentioned several times here and everywhere else) into thevalues.yamlit is just used by the helm chart. But I can't figure out where to put theairflow_local_settings.pyto configure the worker pod image pulled by theKubernetesExecutor. I even can't build a Docker image with Airflow 2.1.4 which is configured as needed.My current Dockerfile looks like the following:
But as soon as the
KubernetesExecutoris pulling my image (because I configured it in thevalues.yaml) it runs into the Docker pull limit very easy, because I can't configure the image correctly.alexandrebunn commentedon Jan 17, 2022
We are facing the same issue, but in our case the error is related the absent of an option to define resources to the sidecar container. Our kubernetes cluster has this resource enforcement active and them even trying to set the resouces by airflow_local_settings.py, pod_template_file, it sounds like the code is ignoring these options.
robkuehl commentedon Aug 17, 2023
But where do I provide the Image Pull Secret for the private registry in the values.yaml? So far I can't find anything and would really appreciate advice @fritzb?
raphaelauv commentedon Jan 18, 2024
you can setup the xcom container image in the kubernetes connection ( airflow connection )
marlena-hammond commentedon Mar 29, 2024
Incase anyone else is encountering this issue still and not using helm, you can update the image value without have to recreate the whole V1Container. I put this in my custom EKSPodOperator. FYI, using this on MWAA.
BjornvandeSandRWS commentedon Apr 3, 2025
Did you ever find an answer to this? I'm still struggling to use an image from our private repository for the sidecar.