Skip to content

Use private docker repository with K8S operator and XCOM sidecar container #10605

Closed

Description

@mickaelgervais

Use private docker repository with K8S operator and XCOM sidecar container

An extra parameter to KubernetesPodOperator: docker_repository, this allows to specify the repository where the sidecar container is located

My company force docker proxy usage for K8S

I need to use my company docker repository, images that are not proxifed by the company docker repository are not allowed

Activity

  1. boring-cyborg commented on Aug 27, 2020

    @boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template!

  2. potiuk commented on Aug 28, 2020

    @potiuk
    Member

    would you like to add this feature @mickaelgervais ? I am happy to review it

  3. mickaelgervais commented on Aug 31, 2020

    @mickaelgervais
    Author

    Hi @potiuk ,

    I've submitted a PR, do nopt hesitate to add comment or if I've implemented this badly.
    The sidecar container declaration is low level in K8S operator so I've tried to not add complexity.

    Regards

  4. zoomzoomTnT commented on Sep 3, 2020

    @zoomzoomTnT

    Hi @mickaelgervais
    To understand your issue better, are you having trouble turning on xcom because 'alpine' give you 403 error? Because in your description, you are not able to use private registry, but in my case, I can pass the whole string to the image in my KubernetesPodOperator like below. I just have problem with the xcom-sidecar, because class PodDefaults hard coded image name to be 'alpine'.
    image="<private-registry>/<dir>/<dir>/<image>:<tag>"

    I have left some comments in your pr for adding a explicit image tag, we have something like private-registry/alpine:3.10, and would like to add a variable for image tags.

    Hi @potiuk

    Is this feature going live in 1.10.13 or later versions?

  5. Shivarp1 commented on Sep 7, 2020

    @Shivarp1

    A work around would be to add this code block in the /home/airflow/airflow_local_settings.py
    PodDefaults.SIDECAR_CONTAINER = k8s.V1Container(
    name=PodDefaults.SIDECAR_CONTAINER_NAME,
    command=['sh', '-c', PodDefaults.XCOM_CMD],
    image='',
    volume_mounts=[PodDefaults.VOLUME_MOUNT],
    resources=k8s.V1ResourceRequirements(
    requests={
    "cpu": "10m",
    "memory":"32Mi"
    },
    limits={
    "cpu": "0.25",
    "memory":"32Mi"
    }
    ),
    )

  6. fritzb commented on Nov 21, 2020

    @fritzb

    Docker Inc started image pull rate limiting around October 30th 2020 (https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status/). As a result, all KubernetesPodPerator tasks with XCOM started to fail in large deployment. Can we have the fix in Airflow 1.10 branch as well?

    Based on the severity of the failure, the bug priority should be raised higher.

      - image: alpine
        imageID: ""
        lastState: {}
        name: airflow-xcom-sidecar
        ready: false
        restartCount: 0
        started: false
        state:
          waiting:
            message: 'rpc error: code = Unknown desc = Error response from daemon: toomanyrequests:
              You have reached your pull rate limit. You may increase the limit by authenticating
              and upgrading: https://www.docker.com/increase-rate-limit'
            reason: ErrImagePull
    
  7. evanhlavaty commented on Sep 22, 2021

    @evanhlavaty

    Is there a reason why this has not been implemented yet?

  8. potiuk commented on Sep 22, 2021

    @potiuk
    Member

    Is there a reason why this has not been implemented yet?

    It was waiting for someone like you to take that over and complete it possibly? Airflow has > 1700 contributors and you can become one if you want some features to be implemented faster :)

  9. potiuk commented on Sep 22, 2021

    @potiuk
    Member

    Feel free to take it over and contribute! we even have Contributors workshops (3 of them till the end of the year). On Airflow Slack there is a #contributors-workshop dedicated channel for it! Anyone is welcome!

  10. zoomzoomTnT commented on Sep 22, 2021

    @zoomzoomTnT
  11. fritzb commented on Sep 23, 2021

    @fritzb

    Here is the workaround (tested with Airflow 2.1.4) if you are using airflow helm chart:

    values.yaml:

    airflowLocalSettings: |
      from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
      PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
    
  12. potiuk commented on Sep 25, 2021

    @potiuk
    Member

    Can you let me know when are the workshops and how do I attend them?

    Just join the #contributors-workshop channel - we announce the workshops there.

  13. evanhlavaty commented on Oct 12, 2021

    @evanhlavaty

    airflowLocalSettings: |
    from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
    PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine

    @fritzb

    I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!

  14. ZackingIt commented on Nov 2, 2021

    @ZackingIt

    airflowLocalSettings: |
    from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
    PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine

    @fritzb

    I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!

    I was able to get my xcom to use a non-DockerHub repo also by creating a airflow_local_settings.py file in the root/config folder, due to this yaml:

    {{ (printf "%s/config/airflow_local_settings.py" .Values.airflowHome) | quote }}
    .

    I then simply overwrote with the code:

    from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
    PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
    

    You can verify the custom airflow_local_settings.py file is loaded because any code written in that file will be executed by teach task instance, i.e., those are the logs to check. That's really the only annoying/hard part, is just knowing whether your instance is actually executing/including the local_settings file.

    Also note that this works even as early as at least 2.1.0 for those of you who are having issues with 2.2.0 :)

  15. dzungdev commented on Nov 11, 2021

    @dzungdev

    Hi @ZackingIt , could I know how do you overrote the code? Do you go to pod and change the airflow_local_settings.py file in root/config folder or you add it in values.yaml file as below:

    Your code

    from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
    PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
    

    in values.yaml

    airflow
        localSettings:
          ## the full content of the `airflow_local_settings.py` file (as a string)
          stringOverride: |
            # use a custom `xcom_sidecar` image for KubernetesPodOperator()
            from airflow.kubernetes.pod_generator import PodDefaults
            PodDefaults.SIDECAR_CONTAINER.image = "private_artifactory/alpine"
    
  16. ZackingIt commented on Nov 11, 2021

    @ZackingIt

    @dzungdev
    I created an actual file, I did not overwrite a variable in the values.yaml i.e., part of my Docker build involves:
    cp /airflow-config/config/airflow_local_settings.py ${AIRFLOW_HOME}/config/airflow_local_settings.py

    When I say "due to this yaml" I mean that that yaml tells us that this is already a location,${AIRFLOW_HOME}/config is already one of the paths being read by default, as I did not need to explicitly set a PYTHONPATH environment variable to read that location.

  17. lshw42 commented on Nov 14, 2021

    @lshw42

    @ZackingIt
    Where exactly did you put the configuration? When I put the PodDefaults.SIDECAR_CONTAINER.image (as mentioned several times here and everywhere else) into the values.yaml it is just used by the helm chart. But I can't figure out where to put the airflow_local_settings.py to configure the worker pod image pulled by the KubernetesExecutor. I even can't build a Docker image with Airflow 2.1.4 which is configured as needed.

    My current Dockerfile looks like the following:

    FROM apache/airflow:2.1.4-python3.8@sha256:ae98e5e9eba596e24bce820e5f3a109274b82cd347bb9fdfca1e9d3f0dda4bb0
    
    COPY --chown=airflow:root airflow_local_settings.py /opt/airflow/config/airflow_local_settings.py
    
    USER airflow

    But as soon as the KubernetesExecutor is pulling my image (because I configured it in the values.yaml) it runs into the Docker pull limit very easy, because I can't configure the image correctly.

  18. alexandrebunn commented on Jan 17, 2022

    @alexandrebunn

    We are facing the same issue, but in our case the error is related the absent of an option to define resources to the sidecar container. Our kubernetes cluster has this resource enforcement active and them even trying to set the resouces by airflow_local_settings.py, pod_template_file, it sounds like the code is ignoring these options.

  19. robkuehl commented on Aug 17, 2023

    @robkuehl

    Here is the workaround (tested with Airflow 2.1.4) if you are using airflow helm chart:

    values.yaml:

    airflowLocalSettings: |
      from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
      PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
    

    But where do I provide the Image Pull Secret for the private registry in the values.yaml? So far I can't find anything and would really appreciate advice @fritzb?

  20. raphaelauv commented on Jan 18, 2024

    @raphaelauv
    Contributor

    you can setup the xcom container image in the kubernetes connection ( airflow connection )

    image

  21. marlena-hammond commented on Mar 29, 2024

    @marlena-hammond

    Incase anyone else is encountering this issue still and not using helm, you can update the image value without have to recreate the whole V1Container. I put this in my custom EKSPodOperator. FYI, using this on MWAA.

    if PodDefaults.SIDECAR_CONTAINER.image != "<your-custom-image>":
        PodDefaults.SIDECAR_CONTAINER.image = "<your-custom-image>"
    
  22. BjornvandeSandRWS commented on Apr 3, 2025

    @BjornvandeSandRWS

    Here is the workaround (tested with Airflow 2.1.4) if you are using airflow helm chart:
    values.yaml:

    airflowLocalSettings: |
      from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
      PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
    

    But where do I provide the Image Pull Secret for the private registry in the values.yaml? So far I can't find anything and would really appreciate advice @fritzb?

    Did you ever find an answer to this? I'm still struggling to use an image from our private repository for the sidecar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions