You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docker-compose copy file or directory to container #5523
we miss a possibility to copy a file or directory using docker-compose. I find this really useful.
Please check many +1 in premature closed #2105
Reacted by david (he/him), chenchuxin, Rafal Krypa, nrdmn, Thomas, alex-eri, Amos, Raffaello, zwessels, Nick Heap and 643 more
Reacted by ekow, Alexander Dobrynin, foxoffire33 , Vladislav Paskar and Biju George
Reacted by Jasmine Hegman, Aaron Davis, kousgi, Austin Jackson, Igal Sapir, Boris D. Teoharov, Serkan, Harsh Donga, Nicolas Beuchat, Sergey Ukolov and 45 more
Reacted by Ilya Evseev, James Muehlner, emmanuel-ashaye, Serkan, Sergey Ukolov, EL KESRI Othmane, klausfuchssprig, Ciaran McGauran, Vinay Sharma, Gianluca and 19 more
What's the usecase? Most of the suggested usage I've seen were antipatterns.
Reacted by Zsolt Katona, James Pudney, FromSi and Pavel Kalashnikov
Reacted by Jan Hertsens, Alexander Löhr, Dmitry Seleznyov, John Mora, Ryan, Mike, Maksim Makhniuk, Chris Heney, Pavan Keerthi, Greg Brown and 300 more
Reacted by Al
Reacted by Igal Sapir, jpmaligne, Dongdong Li (Don), Sébastien Treille, Iqbal, Myroslav, Danila Mihalkovich, Antoine Viallon, -, Giorgos Stamatakis and 14 more
Reacted by Alexander Löhr, Stein-Bjarne Johansen, ClaudioGisch, Abdo, pallinger, Simone Cogno, SGINJ, Viktor Shchelochkov , Yuri Volchkov and Stefan Mototolea
Reacted by Karl-Philipp Richter, _bran, Alexander Schwankner, Jorge Rodriguez Flores, Christopher Dingli, Sergei Ermakov, Michael Sanford, Zh Zh Zh, random-webdeveloper, Einar Gangsø and 3 more
This will make it available/runnable "globally" not only on machine where it was composed as in case of mounting volume(s)
The problem with this is that it is incredibly short-sighted (hence the term "anti-pattern"), as it will force you to repeat the operation every time the containers to be recreated. Not to mention the fact that it scales very poorly (what if you have 10 containers? 20? 100?)
The actual solution to your issue is to include those necessary files in your build (Dockerfile) and rebuild when an update is needed.
Reacted by Frank Sachsenheim, Dutch van Andel, Nk, Jose Angel Sanchez Castillejos , Julian Mojico, Sergei Ermakov, Trần Hữu Lâm, subash adhikari, Nick Brady, Luiz and 5 more
Reacted by Pavan Keerthi, Ingo Keck, jtdevos, Matthew Parrott, Alexander Löhr, georgewhewell, Jules Morelli, Marcos Soutullo Rodriguez, Dmitry Matveev, gary6xu and 86 more
I'm running into an issue where copy would come in handy (at least as an override). I mostly develop on mac so I almost never see an issue with commands running as root in the container and exporting to a mounted volume. However, recently using the same workflow on a CentOs has caused some major pain because files owned by the root user are being added to the host via the mounted volume. I would like in these cases to just be able to copy the host files to the container instead of mounting them.
Use case: starting multiple docker containers simultaneously from .gitlab-ci.yml which need to write into the git repository's directory.
If the process inside a container fails or if the ci job is cancelled before the container has cleaned up after itself, the remaining files can't be deleted by gitlab-runner due to lack of permissions. Now I could copy the files within the container out of the volume into another directory, but that would be an antipattern, wouldn't it?
Reacted by Gavril Tonev, rndrr, carterminearIMT, alexhiggins732, Pavel Argentov, RJ Garcia, pallinger, yodapotatofly and user-llmm
Is this different from volumes: - ./folder_on_host/ :/folder_in_container/ ?
I am able to copy files from host to container (equivalent of COPY) this way in my compose file
Reacted by Thomas, Alex, Mykhail Martsyniuk, MikeLaza, kernitus and Roman
@harpratap you are right, but the drawback is that /folder_in_container must not exist or must be empty or else it will be overwritten. If you have a bash script as your entry point, you could circumvent this by symlinking your files into the originally intended directory after you create a volume at /some_empty_location
+1 for having a COPY functionality. Our use case is for rapidly standing up local development environments and copying in configs for the dev settings.
Reacted by Harpratap Singh (Hari), Ettapp, hud, Nouman Khan Behttani, Aden Narkowicz, rndrr, Theo Paul, Vittorio Passuello-Dussault, magick93, Jonathan Lee Komar and 25 more
+1 for COPY. This would really be a helpful feature.
Use case: in swarm mode, I have a service using mysql image. I need to copy my initialization scripst in /docker-entrypoint-initdb.d/ so that MySQL can execute them.
Though it is possible to create an image on top of mysql, copy the files and use it or connect to the mysql
task in swarm and then manually run the scripts, it's kinda unnecessary in my opinion.
Reacted by Ingo Keck, ragnese, Maxim Kot, Martin Majewski, Prasannjeet Singh, John Furr, Andrés Mata Suárez, Mihai Ungureanu, Johnny Miller, jared-hardy and 14 more
Use Case:
Fluentd requires the configuration files to be moved into the container during run time. These config files are created on the run time by our Jenkins Engine and without a COPY/ADD in docker compose it simply fails.
If I understand correctly, the main benefit is that docker-compose cp recognizes services (not just containers), so does that mean if I have a scale > 1 it would cp the data to each and every instance?
What if I have a scale: 0? No cp happens? Are there other benefits over docker cp?
What's the usecase? Most of the suggested usage I've seen were antipatterns.
One use case for me is to put the file custom.cnf into /etc/mysql/conf.d inside mysql container. Volume is not an answer to that but file copy would be more appropriate. So you expect me to create a Dockerfile just copy that one file?
Common Scenario: 1 or more files that live in non-empty directories need to be changed on-the-fly. You can't use volume with these directories as you'd lose all the other files in the directory. You most definitely do not want to copy the entire directory down to work around this. A suggestion of rebuilding the image every time you make a change completely ignores the real world, time, and the rage induced because you missed a comma. A basic copy command isn't an anti-pattern by any stretch.
why not add this to docker-compose and enable for use in docker-compose.yml configs as well?
Reacted by jihu, Nicolas Penot, Antonio Marques, garyng2000, Joao Rodrigues Dias Neto, shemanov, Alistair1231, jofrey007, Melvin Dors, jtheuerkauf-zz and 21 more
I solved it. Gave up on Docker entirely and am using LXC. Solved everything for my use case AND no more people insulting me for "antipatterns". Win-win.
Reacted by James Coole, Nato, Crown0815, Joaquim, borlandconnect, Vaughan, Rfrancis13, R S, Nick McCullum, Ilya Kaznacheev and 7 more
It is disingenous to write that the feature was implemented, because it was not.
The feature that was asked for have not been not implemented, and instead the feature that nobody has asked for and which was already fully covered by underlying docker was added.
The central issue of this feature request is that one does not own most of the images they have in their Docker Compose file, and when they bring up an image like grafana/grafana they cannot supply its config file in a hygienic fashion, and the container would not boot without that file making subsequent docker cp (or docker compose cp for all what it is worth) impossible. Bind mount of that file will lead to the config file hanging around in the host file system, which does seem like a huge anti-pattern WRT docker to me.
Reacted by Zoltán Gaál, Rfrancis13, Eric Ferreira, Caleb Reyes, aslze, andoks, Matin ShayanFar and Heath Raftery
This ends the nearly 6 year long holy war. The reasonable among us were just asking for a way to put configs in our containers without volume mounts and without rolling new images.
Apparently it's available in docker compose 2.23.1. I would follow that thread if you are still seriously interested in this feature.
@washtubs while inlined config indeed don't rely on bind mount, this is not the reason this feature was introduced. We could support file-based configs without a bind mount, just this would have some side effects on existing users.
@alamar the example you mention suggest we should investigate lifecycle hooks. A pre-start hook to cp required file in container would solve this use-case.
@ndeloof I haven't used docker compose in some time. I got renewed interest in it just now after realizing it was rewritten in go (yeah it's been a while), and seeing the output of docker compose up which honestly looks gorgeous, which is why I got curious to see if anything became of this issue.
So forgive me for my lack of ecosystem knowledge here. I assume those who are used to helm and kubernetes would like to have some kind of templating pre-processor engine on top of docker compose and use that to insert file contents inline. I'm not sure if compose has a "helm"-like sister software like kubernetes does, or perhaps even builtin support. But that's an approach that would be familiar to me which I would leverage to insert config files inline as a pre-processing step.
@washtubs docker compose do support variable interpolation to offer some flexibility, but we don't want this to become a templating system like the one helm uses. Obviously someone could built another tool to produce compose.yaml from another source.
@washtubs / @ndeloof : in a way docker compose already can be a lightweight templating tool for itself, by building your docker-compose config with variables, then "rendering" the complete result by using docker compose config 😅
That doesn't seem quite the same. I think that is basically coalescing multiple yaml files together into one, another thing that helm does. But helm uses go's template engine which gives a ton of flexibility on the kinds of things you can insert, and manipulations you can do, like quoting. A common pattern is inserting the contents of a file inline with indent. You can even define your own custom template functions to be reused throughout your project.
Honestly you could probably use helm to do this for compose, just make a helm chart alongside your docker-compose file, and you can use the helm template command to get the output docker-compose file with your file contents rendered and script the deployment yourself. (won't be able to use helm install or upgrade obviously, since those are k8 specific)
Another usecase: I have a compose file on a local machine and my dockerhost ist a remote machine. How do I reference a file to be mounted from my local directory? Answer: not at all without building a specific image! A solution would be to copy it into the container.
Activity
shin- commentedon Jan 2, 2018
What's the usecase? Most of the suggested usage I've seen were antipatterns.
ukbe commentedon Jan 2, 2018
+1
shin- commentedon Jan 2, 2018
The problem with this is that it is incredibly short-sighted (hence the term "anti-pattern"), as it will force you to repeat the operation every time the containers to be recreated. Not to mention the fact that it scales very poorly (what if you have 10 containers? 20? 100?)
The actual solution to your issue is to include those necessary files in your build (Dockerfile) and rebuild when an update is needed.
dkinzer commentedon Jan 15, 2018
I'm running into an issue where copy would come in handy (at least as an override). I mostly develop on mac so I almost never see an issue with commands running as root in the container and exporting to a mounted volume. However, recently using the same workflow on a CentOs has caused some major pain because files owned by the root user are being added to the host via the mounted volume. I would like in these cases to just be able to copy the host files to the container instead of mounting them.
The related issue: #1532
update
I think in my case I can get away with using COPY in the Dockerfile and having multiple docker-compose files one of which uses a volume mount.
nrdmn commentedon Feb 26, 2018
Use case: starting multiple docker containers simultaneously from .gitlab-ci.yml which need to write into the git repository's directory.
If the process inside a container fails or if the ci job is cancelled before the container has cleaned up after itself, the remaining files can't be deleted by gitlab-runner due to lack of permissions. Now I could copy the files within the container out of the volume into another directory, but that would be an antipattern, wouldn't it?
harpratap commentedon Mar 12, 2018
Is this different from
volumes: - ./folder_on_host/ :/folder_in_container/?I am able to copy files from host to container (equivalent of COPY) this way in my compose file
justin-vanwinkle commentedon Mar 13, 2018
@harpratap you are right, but the drawback is that /folder_in_container must not exist or must be empty or else it will be overwritten. If you have a bash script as your entry point, you could circumvent this by symlinking your files into the originally intended directory after you create a volume at /some_empty_location
+1 for having a COPY functionality. Our use case is for rapidly standing up local development environments and copying in configs for the dev settings.
djelroy commentedon Mar 15, 2018
+1 for COPY. This would really be a helpful feature.
Use case: in swarm mode, I have a service using mysql image. I need to copy my initialization scripst in /docker-entrypoint-initdb.d/ so that MySQL can execute them.
Though it is possible to create an image on top of mysql, copy the files and use it or connect to the mysql
task in swarm and then manually run the scripts, it's kinda unnecessary in my opinion.
AnishAnil commentedon Mar 26, 2018
+1 for COPY/ADD,
Use Case:
Fluentd requires the configuration files to be moved into the container during run time. These config files are created on the run time by our Jenkins Engine and without a COPY/ADD in docker compose it simply fails.
mbao01 commentedon May 7, 2018
141 remaining items
evbo commentedon Jun 17, 2021
ok, thanks for clarifying @ndeloof
If I understand correctly, the main benefit is that
docker-compose cprecognizes services (not just containers), so does that mean if I have ascale> 1 it wouldcpthe data to each and every instance?What if I have a
scale: 0? Nocphappens? Are there other benefits overdocker cp?ndeloof commentedon Jun 17, 2021
this is indeed "just"
docker cpapplied to servicesfor scale != 1 it will copy to all containers, or you can pass
--indexnatotech commentedon Jul 6, 2021
One use case for me is to put the file custom.cnf into /etc/mysql/conf.d inside mysql container. Volume is not an answer to that but file copy would be more appropriate. So you expect me to create a Dockerfile just copy that one file?
jadon1979 commentedon Jul 6, 2021
Common Scenario: 1 or more files that live in non-empty directories need to be changed on-the-fly. You can't use volume with these directories as you'd lose all the other files in the directory. You most definitely do not want to copy the entire directory down to work around this. A suggestion of rebuilding the image every time you make a change completely ignores the real world, time, and the rage induced because you missed a comma. A basic copy command isn't an anti-pattern by any stretch.
A quick implementation of this might be an underlying volume that takes these files and creates a symlink to their respective destinations.
eldarj commentedon Feb 14, 2022
why not add this to docker-compose and enable for use in docker-compose.yml configs as well?
XeJames commentedon Dec 13, 2022
I solved it. Gave up on Docker entirely and am using LXC. Solved everything for my use case AND no more people insulting me for "antipatterns". Win-win.
alamar commentedon Jul 19, 2023
It is disingenous to write that the feature was implemented, because it was not.
The feature that was asked for have not been not implemented, and instead the feature that nobody has asked for and which was already fully covered by underlying docker was added.
The central issue of this feature request is that one does not own most of the images they have in their Docker Compose file, and when they bring up an image like
grafana/grafanathey cannot supply its config file in a hygienic fashion, and the container would not boot without that file making subsequentdocker cp(ordocker compose cpfor all what it is worth) impossible. Bind mount of that file will lead to the config file hanging around in the host file system, which does seem like a huge anti-pattern WRT docker to me.washtubs commentedon Nov 30, 2023
compose-spec/compose-spec#346
This ends the nearly 6 year long holy war. The reasonable among us were just asking for a way to put configs in our containers without volume mounts and without rolling new images.
Apparently it's available in docker compose 2.23.1. I would follow that thread if you are still seriously interested in this feature.
ndeloof commentedon Nov 30, 2023
@washtubs while inlined config indeed don't rely on bind mount, this is not the reason this feature was introduced. We could support file-based configs without a bind mount, just this would have some side effects on existing users.
@alamar the example you mention suggest we should investigate lifecycle hooks. A pre-start hook to
cprequired file in container would solve this use-case.washtubs commentedon Nov 30, 2023
@ndeloof I haven't used docker compose in some time. I got renewed interest in it just now after realizing it was rewritten in go (yeah it's been a while), and seeing the output of
docker compose upwhich honestly looks gorgeous, which is why I got curious to see if anything became of this issue.So forgive me for my lack of ecosystem knowledge here. I assume those who are used to helm and kubernetes would like to have some kind of templating pre-processor engine on top of docker compose and use that to insert file contents inline. I'm not sure if compose has a "helm"-like sister software like kubernetes does, or perhaps even builtin support. But that's an approach that would be familiar to me which I would leverage to insert config files inline as a pre-processing step.
ndeloof commentedon Nov 30, 2023
@washtubs docker compose do support variable interpolation to offer some flexibility, but we don't want this to become a templating system like the one helm uses. Obviously someone could built another tool to produce compose.yaml from another source.
andoks commentedon Dec 1, 2023
@washtubs / @ndeloof : in a way docker compose already can be a lightweight templating tool for itself, by building your docker-compose config with variables, then "rendering" the complete result by using
docker compose config😅washtubs commentedon Dec 2, 2023
That doesn't seem quite the same. I think that is basically coalescing multiple yaml files together into one, another thing that helm does. But helm uses go's template engine which gives a ton of flexibility on the kinds of things you can insert, and manipulations you can do, like quoting. A common pattern is inserting the contents of a file inline with indent. You can even define your own custom template functions to be reused throughout your project.
Honestly you could probably use helm to do this for compose, just make a helm chart alongside your docker-compose file, and you can use the helm template command to get the output docker-compose file with your file contents rendered and script the deployment yourself. (won't be able to use helm install or upgrade obviously, since those are k8 specific)
sahlex commentedon Apr 4, 2024
Another usecase: I have a compose file on a local machine and my dockerhost ist a remote machine. How do I reference a file to be mounted from my local directory? Answer: not at all without building a specific image! A solution would be to copy it into the container.