Repository navigation
Add copy to the yaml configuration #1664
Description
I want to deploy different services using the same image but with a different configuration file.
Currently to achieve that I can
- build as many images that inherit from a common image plus a different COPY in each
- mount a single file volume
The first solution isn't feasible and the second it's overkill. I just want to copy a file, not keep it in sync
Proposed configuration:
myservice:
image: foo/bar
copy:
- src:dest
Reacted by Tevin, gRey, NicklasWallgren, Jesper Rønn-Jensen, AnthonyWC, Viktor Alenkov, Eric LERY, Aurélien Lavorel, ajthemacboy, Spence and 119 more
Activity
prologic commentedon Jul 7, 2015
What is the semantics of
copyhere? Are we copying files from the host wheredocker-composeis run from into the newly running container usingdocker cp?prologic commentedon Jul 7, 2015
AFAIK there is no current way of doing this:
See:
riquito commentedon Jul 7, 2015
Yes
I see that the
docker cpcommand is unable to do it. Yet it's still possible by accessing the container data on the host side (which I ammit is ugly)http://stackoverflow.com/a/24805696/210090
prologic commentedon Jul 7, 2015
Yeah I'm not going to advocate we do this at all. This will break a lot of things and assumptions about Docker. This is also highly dependent on the underlying storage driver being used as well. Docker is meant to be agnostic. I think we should a different solution to your problem :)
ljrittle commentedon Aug 16, 2015
Now that docker-1.8 added support for copy into a container via
docker cpand swarm-0.4.0-rc2 added support fordocker cp, it would be great to revisit this issue at the compose level. Here is the use case (which mirrors what we actually do in almost production now):A docker-compose.yml file that mentions many containers by CI-built image tag (i.e. we don't use
build:in production; maybe we could now but it didn't play nice with swarm in past releases) that all need e.g. static hadoop configuration files mapped in which exactly match the deployment environment. Currently, one must manually sync the directory in which docker-compose.yml resides to the exact path on each target docker-machine in the swarm. Then one adds a stack ofvolume:lines.Supporting
docker cpwould allow the removal of a custom configuration file sync in our system deployment method and allow better control of when configuration changes are injected (since anyone changing the files mentioned in thevolume:lines are affecting all prior deployed containers (and whatever time the internal implementations re-read said configuration, perhaps only upon the next container restart, perhaps when various tools inside the container start; which may be unexpected) and future (which is expected).OTOH, (as mentioned briefly above) maybe we should use
build:. The downside here is that we need to write an extra Dockerfile per deployment container type. One for the CI-built images and a second for the static configuration file injector.copy:in compose backed by the recent changes todocker cpwould neatly allow us to avoid all this duplication.dnephin commentedon Aug 25, 2015
It still seems like using a volume is a better way to do this. It doesn't have to be a host volume, it should work with a data volume container.
prologic commentedon Aug 25, 2015
Bedies
docker-machinehasdocker-machine scpanyway which you can do with the preparation of the machine; and then you can do host volumes the normal way eithdockerordocker-composeljrittle commentedon Aug 26, 2015
"It still seems like using a volume is a better way to do this. It doesn't have to be a host volume, it should work with a data volume container."
I fail to see how that is best when the target is a swarm. AFAICS, one has no way to express that a data volume container should exist at every possible swarm node.
prologic commentedon Aug 26, 2015
Can't you deploy a data volume container on very node via swarm itself anyway like you can with any other container ala scheduling strategy?
ljrittle commentedon Aug 26, 2015
"Bedies docker-machine has docker-machine scp anyway which you can do with the preparation of the machine; and then you can do host volumes the normal way eith docker or docker-compose"
I currently do this. It is painful. People managing deployments forget to to move the configuration files. If it was supported in docker-compose yml then it would happen whenever a container was recreated without the manual step(s).
ljrittle commentedon Aug 26, 2015
"Can't you deploy a data volume container on very node via swarm itself anyway like you can with any other container ala scheduling strategy?"
Maybe we could but I admit that I don't see how in light of scaling.
ljrittle commentedon Aug 26, 2015
"Can't you deploy a data volume container on very node via swarm itself anyway like you can with any other container ala scheduling strategy?"
Humm, if I know that my swarm is size 10 nodes; can I just scale the data volume container to be size 10 with a policy that no dups are allowed on a given docker engine node? The problem (unless I have missed something) is that e.g. containers that need to reference that data volume have no way to --volume-from a container with an index that matches. Guys, I have been looking at this problem for 2 months now. The best I could do is create a script that uses
docker-machine scp. But it is a manual step upon after editing a configuration file and beforedocker-compose up. It also requires that one can write the exact path on the target machines of the swarm as the root of the docker-compose.yml (i.e. it smells like a major hack).prologic commentedon Aug 26, 2015
Perhaps my efforts in factory can help automate this process of spinning up docker machines and swarm clsuters with the "right" data sitting there ready to use? (_albeit it's still in early development and I'm using it to combine
docker-machineanddocker-composein a single setp_0.dnephin commentedon Oct 16, 2015
copyis an operation, not part of the service configuration, so it doesn't belong in the compose file.Docker 1.9 is adding a new volumes api, and volume drivers are in already. Volumes are really the correct way to support this. Not trying to copy files around.
Thanks for the suggestion! but I don't think this really fits with the design of compose right now.
121 remaining items
elemir commentedon May 31, 2018
+1
avgoeid commentedon Jun 7, 2018
+1
umedjan commentedon Jun 11, 2018
+1
ascourtas commentedon Jun 15, 2018
+1 this is low key driving me nuts, it seems silly to have to mount a volume just for one config file.
Payden-Pringle commentedon Jul 13, 2018
+1
We have a way to add files to a container using docker compose with volumes. I've noticed that the Compose documentation does have a
configssetting now, so we can add single files as well.https://docs.docker.com/compose/compose-file/#configs
However, what I believe people want is a way to merge files from the host into a directory in the container with preference being
overwrite if exists, the way you'd expect a copy to work.It's strange to me that the
volumesconfiguration in the compose file has anocopyoption:https://docs.docker.com/compose/compose-file/#volumes
Which is described like this:
volume: configure additional volume optionsnocopy: flag to disable copying of data from a container when a volume is createdSo it disables exactly what is desired?
My desired use case is web applications for development. I want the image to contain everything it needs to serve a website including a default website to serve, but have an option to merge/overwrite those files with the current working state of a website without other configuration (the key) beyond having the files in the right place on the host. Files that, when copied to the container, are ephemeral. Meaning not in a network share.
An example of a framework that would benefit from that is something like Laravel. Where I make an image that will be capable of serving the Laravel splash page with no other configuration (volumes, network shares, etc), but the compose file is written to copy files from the host to the container's Laravel root so that they can be overwritten.
Using a network share for this is extra upkeep for an ephemeral environment (development isn't always constant). Using a volume mount for it, with a framework like Laravel, requires building the app in that volume which means installing PHP and composer on the host or sharing the location on the host the app would be created at. Both creating more maintenance.
Edit:
After testing, it seems that when you mount a volume, the files that existed in the image at that location are copied over to the persistent volume with preference to the volume's original contents. Meaning it doesn't overwrite.
I actually think this is a solution for the desired outcome.
su-narthur commentedon Aug 9, 2018
Hey, guys, just FYI, it's better to give a thumb-up reaction to the top message on this issue than add ing a "+1" comment, as issues can be sorted by how many reactions they've received.
sentinel09 commentedon Oct 17, 2018
+1
flavienbwk commentedon Mar 5, 2019
+1 Please Docker team, consider your community.
masaeedu commentedon Mar 12, 2019
@shin- The problem right now is that the volumes feature makes it impossible to plug in a single config file from an external volume into a container.
Suppose I want to plug an nginx config file into
/etc/something/whatever.confwithout annihilating everything else that's in that folder. You can't create a named volume for a single file (see moby/moby#30310), and you also can't create a directory volume containing a single file, then mount that particular file into the container (see moby/moby#32582).The only option you're left with is to pollute fixed locations in your actual Docker host with the config files and then bind mount them in. This of course goes to pieces if you want to move around between Docker hosts, or work with a Docker host you don't have filesystem access to (e.g. online CI), or even just run multiple stacks in parallel.
Something has to give way here, either the volumes feature in Docker needs to change so that we can actually use it to accomplish injecting config files into our stack, or we need to work around it in compose using something like a
copyconfig option.shin- commentedon Mar 12, 2019
FWIW I'm no longer a Compose maintainer, but it seems to me that that use case would be best served by injecting the configuration file at build time using a Dockerfile.
masaeedu commentedon Mar 12, 2019
@shin- Unfortunately the configuration files are not known at build time, the images are prepared on a CI server long before we know where we're going to be deploying them and in what configuration.
WebMatrixware commentedon Mar 12, 2019
I'm with @masaeedu, I understand the conceptual ideal of keeping the function of compose from creeping and therefore not wanting to add this feature, but in real life the loss of this feature GREATLY reduces the circumstances where compose can be used without
A) having to cobble together some kind of of hideous script work around to stand up the containers in question (the very thing compose was meant to help standardize and keep us from having to do),
B) having to add some other tool that handles all of our config (which while workable adds a huge amount of overhead for small to mid-sized projects that really just need to copy a specific .conf and .env to each container, something you do not want to do at image build time, that is equivalent to saying that rather than your software having a config file you should hard code all of your options into the C++, it leads to dozens of extra images to have to maintain), or
C) having to maintain a separate image for every possible config of a given piece of software that we will ever need / build a custom image every time we have to make a minor config change....which is both unworkable in the real world of production software, and also makes it impossible to have the kind of robust QA and testing pipeline that we need for production software.
Spunturn commentedon Dec 4, 2024
https://github.com/langgenius/dify.git
egorlitvinenko commentedon Jan 15, 2025
So, 2025, ten years over, still no copy, still same questions. :)
brian-farrell commentedon Jun 6, 2025
@shin- This doesn't make any sense - configuration occurs at run time, not at build time.