Skip to content

docker-compose copy file or directory to container #5523

Closed

Description

we miss a possibility to copy a file or directory using docker-compose. I find this really useful.
Please check many +1 in premature closed #2105

Activity

  1. shin- commented on Jan 2, 2018

    @shin-

    What's the usecase? Most of the suggested usage I've seen were antipatterns.

  2. ukbe commented on Jan 2, 2018

    @ukbe

    +1

  3. shin- commented on Jan 2, 2018

    @shin-

    This will make it available/runnable "globally" not only on machine where it was composed as in case of mounting volume(s)

    The problem with this is that it is incredibly short-sighted (hence the term "anti-pattern"), as it will force you to repeat the operation every time the containers to be recreated. Not to mention the fact that it scales very poorly (what if you have 10 containers? 20? 100?)

    The actual solution to your issue is to include those necessary files in your build (Dockerfile) and rebuild when an update is needed.

  4. dkinzer commented on Jan 15, 2018

    @dkinzer

    I'm running into an issue where copy would come in handy (at least as an override). I mostly develop on mac so I almost never see an issue with commands running as root in the container and exporting to a mounted volume. However, recently using the same workflow on a CentOs has caused some major pain because files owned by the root user are being added to the host via the mounted volume. I would like in these cases to just be able to copy the host files to the container instead of mounting them.

    The related issue: #1532

    update

    I think in my case I can get away with using COPY in the Dockerfile and having multiple docker-compose files one of which uses a volume mount.

  5. nrdmn commented on Feb 26, 2018

    @nrdmn

    Use case: starting multiple docker containers simultaneously from .gitlab-ci.yml which need to write into the git repository's directory.

    If the process inside a container fails or if the ci job is cancelled before the container has cleaned up after itself, the remaining files can't be deleted by gitlab-runner due to lack of permissions. Now I could copy the files within the container out of the volume into another directory, but that would be an antipattern, wouldn't it?

  6. harpratap commented on Mar 12, 2018

    @harpratap

    Is this different from volumes: - ./folder_on_host/ :/folder_in_container/ ?
    I am able to copy files from host to container (equivalent of COPY) this way in my compose file

  7. justin-vanwinkle commented on Mar 13, 2018

    @justin-vanwinkle

    @harpratap you are right, but the drawback is that /folder_in_container must not exist or must be empty or else it will be overwritten. If you have a bash script as your entry point, you could circumvent this by symlinking your files into the originally intended directory after you create a volume at /some_empty_location

    +1 for having a COPY functionality. Our use case is for rapidly standing up local development environments and copying in configs for the dev settings.

  8. djelroy commented on Mar 15, 2018

    @djelroy

    +1 for COPY. This would really be a helpful feature.

    Use case: in swarm mode, I have a service using mysql image. I need to copy my initialization scripst in /docker-entrypoint-initdb.d/ so that MySQL can execute them.

    Though it is possible to create an image on top of mysql, copy the files and use it or connect to the mysql
    task in swarm and then manually run the scripts, it's kinda unnecessary in my opinion.

  9. AnishAnil commented on Mar 26, 2018

    @AnishAnil

    +1 for COPY/ADD,

    Use Case:
    Fluentd requires the configuration files to be moved into the container during run time. These config files are created on the run time by our Jenkins Engine and without a COPY/ADD in docker compose it simply fails.

  10. mbao01 commented on May 7, 2018

    @mbao01
  11. 141 remaining items

  12. evbo commented on Jun 17, 2021

    @evbo

    ok, thanks for clarifying @ndeloof

    If I understand correctly, the main benefit is that docker-compose cp recognizes services (not just containers), so does that mean if I have a scale > 1 it would cp the data to each and every instance?

    What if I have a scale: 0? No cp happens? Are there other benefits over docker cp?

  13. ndeloof commented on Jun 17, 2021

    @ndeloof
    Contributor

    this is indeed "just" docker cp applied to services
    for scale != 1 it will copy to all containers, or you can pass --index

  14. natotech commented on Jul 6, 2021

    @natotech

    What's the usecase? Most of the suggested usage I've seen were antipatterns.

    One use case for me is to put the file custom.cnf into /etc/mysql/conf.d inside mysql container. Volume is not an answer to that but file copy would be more appropriate. So you expect me to create a Dockerfile just copy that one file?

  15. jadon1979 commented on Jul 6, 2021

    @jadon1979

    Common Scenario: 1 or more files that live in non-empty directories need to be changed on-the-fly. You can't use volume with these directories as you'd lose all the other files in the directory. You most definitely do not want to copy the entire directory down to work around this. A suggestion of rebuilding the image every time you make a change completely ignores the real world, time, and the rage induced because you missed a comma. A basic copy command isn't an anti-pattern by any stretch.

    copy:
      - ./some_dir/some_file:/etc/app/some_config/:<chmod>:<chgrp>:<chown>
      
    

    A quick implementation of this might be an underlying volume that takes these files and creates a symlink to their respective destinations.

  16. eldarj commented on Feb 14, 2022

    @eldarj

    why not add this to docker-compose and enable for use in docker-compose.yml configs as well?

  17. XeJames commented on Dec 13, 2022

    @XeJames

    I solved it. Gave up on Docker entirely and am using LXC. Solved everything for my use case AND no more people insulting me for "antipatterns". Win-win.

  18. alamar commented on Jul 19, 2023

    @alamar

    It is disingenous to write that the feature was implemented, because it was not.

    The feature that was asked for have not been not implemented, and instead the feature that nobody has asked for and which was already fully covered by underlying docker was added.

    The central issue of this feature request is that one does not own most of the images they have in their Docker Compose file, and when they bring up an image like grafana/grafana they cannot supply its config file in a hygienic fashion, and the container would not boot without that file making subsequent docker cp (or docker compose cp for all what it is worth) impossible. Bind mount of that file will lead to the config file hanging around in the host file system, which does seem like a huge anti-pattern WRT docker to me.

  19. washtubs commented on Nov 30, 2023

    @washtubs

    compose-spec/compose-spec#346

    This ends the nearly 6 year long holy war. The reasonable among us were just asking for a way to put configs in our containers without volume mounts and without rolling new images.

    Apparently it's available in docker compose 2.23.1. I would follow that thread if you are still seriously interested in this feature.

  20. ndeloof commented on Nov 30, 2023

    @ndeloof
    Contributor

    @washtubs while inlined config indeed don't rely on bind mount, this is not the reason this feature was introduced. We could support file-based configs without a bind mount, just this would have some side effects on existing users.

    @alamar the example you mention suggest we should investigate lifecycle hooks. A pre-start hook to cp required file in container would solve this use-case.

  21. washtubs commented on Nov 30, 2023

    @washtubs

    @ndeloof I haven't used docker compose in some time. I got renewed interest in it just now after realizing it was rewritten in go (yeah it's been a while), and seeing the output of docker compose up which honestly looks gorgeous, which is why I got curious to see if anything became of this issue.

    So forgive me for my lack of ecosystem knowledge here. I assume those who are used to helm and kubernetes would like to have some kind of templating pre-processor engine on top of docker compose and use that to insert file contents inline. I'm not sure if compose has a "helm"-like sister software like kubernetes does, or perhaps even builtin support. But that's an approach that would be familiar to me which I would leverage to insert config files inline as a pre-processing step.

  22. ndeloof commented on Nov 30, 2023

    @ndeloof
    Contributor

    @washtubs docker compose do support variable interpolation to offer some flexibility, but we don't want this to become a templating system like the one helm uses. Obviously someone could built another tool to produce compose.yaml from another source.

  23. andoks commented on Dec 1, 2023

    @andoks

    @washtubs / @ndeloof : in a way docker compose already can be a lightweight templating tool for itself, by building your docker-compose config with variables, then "rendering" the complete result by using docker compose config 😅

  24. washtubs commented on Dec 2, 2023

    @washtubs

    That doesn't seem quite the same. I think that is basically coalescing multiple yaml files together into one, another thing that helm does. But helm uses go's template engine which gives a ton of flexibility on the kinds of things you can insert, and manipulations you can do, like quoting. A common pattern is inserting the contents of a file inline with indent. You can even define your own custom template functions to be reused throughout your project.

    Honestly you could probably use helm to do this for compose, just make a helm chart alongside your docker-compose file, and you can use the helm template command to get the output docker-compose file with your file contents rendered and script the deployment yourself. (won't be able to use helm install or upgrade obviously, since those are k8 specific)

  25. sahlex commented on Apr 4, 2024

    @sahlex

    Another usecase: I have a compose file on a local machine and my dockerhost ist a remote machine. How do I reference a file to be mounted from my local directory? Answer: not at all without building a specific image! A solution would be to copy it into the container.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions