Thanks @aboch !

LGTM

fixes #17190

This does not fix #17190. It just makes the corruption less likely. There are still several race conditions.

@phemmer the internal corruption issue is addressed via #17278. and by turning off the /etc/hosts update for the default bridge, it addresses #17190, which is reported for the default bridge. With this PR the functionality is exactly the same as before and hence it does fix #17190.

Now, your proposal (which I like a lot) : #17195 will hopefully solve the general /etc/hosts update issue by using embedded DNS. That is the rationale behind the fixes assertion here.

LGTM

This also addresses #16619 and #16445

Updated PR description with "fixes" for the two issues @mavenugo mentioned above.

LGTM

Just want to briefly re-state what I said on #17190 (comment) - this is a change that breaks behaviour observable in three releases (1.8.X) and offers no alternative for people who need it to permit previously impossible use-cases.

(I particularly object to this breaking change being added during the 1.9 RC process)

@aidanhs that behavior should never have found its way into the a regular release; it was part of the "experimental" builds, but leaked into the regular builds, which is the bug being fixed here. If you need "service discovery", create a network and attach containers to that.

@thaJeztah I appreciate it was unintentional, but it did happen, has been present for two months and people now rely on the behaviour!

As mentioned in the blog post (from the comment I linked to above), it lets docker build do a number of helpful new things, like retrieve files over SSH without baking in credentials and perform nested docker build. These are things people really do want (there are links to issues in that post). Attaching to a network is not possible for docker build.

If there was a way to set a different default network (and therefore get back to service discovery) I'd have no problem - does a change like this have any chance of getting into 1.9 to help alleviate the breakage of this PR?

@aidanhs relying on external services during build is not something we advocate, because it makes a build not reproducible in different environments. I'm not aware of a --default-network (e.g.) flag for the daemon being worked on. I know it has been discussed at some point, but don't think it reached consensus.

@thaJeztah it's a nuanced discussion...but from that point of view our builds aren't reproducible anyway because they have HTTP requests to .internal addresses - people would also object if this got broken! Anyway, the only reason we need this service discovery is because the necessary features aren't in dockerfiles (and never will be, due to the freeze). The alternative (restarting the daemon and adding --dns $BRIDGE_IP) is really painful in comparison.

Manually adding containers to a network isn't exactly delightful either.

I was more thinking of something like docker network create --set-as-default newbridge.

@aidanhs Why not use --dns on the daemon and then register your containers with a DNS service?

@aidanhs also one can launch a container in a user-created network using the --net=<network> option (which automatically connects to the network and SD in-built).
If your service that uses Docker API (instead of CLI), then you could make --net=<network> as default and your users wont know the difference. Will that work ?
I would like to understand your service better. am available in IRC (@Madhu) and we can discuss more in detail.

@cpuguy83

The alternative (restarting the daemon and adding --dns $BRIDGE_IP) is really painful in comparison.

Because it's really painful :)

To put this into context, I'm sharing images and dockerfiles with people who have no idea how docker works behind the scenes and don't even have root access on their machines (yes, they can run containers so could get root, but they're not at that skill level). Getting them to find the bridge IP, change their daemon arguments to have dns pointing at the bridge IP, restart their daemon, start a DNS service container listening on the bridge IP, then do their build...they're just not going to bother. This also wouldn't be so bad if restarting the daemon didn't kill all containers!

Right now I can just say "run this command (to start a container), then do docker build". If I could set a default network for all future container startups, that's three commands (which is fine). If it were possible to pass --net to docker build, that's two commands. If it were possible to intercept the API calls the builder makes (i.e. when #14298 happens) to alter --net myself, that's two commands. If it were possible to enable service discovery on the default bridge, that's also two commands. Any of these would be great. But none are available right now!

@tiborvass suggested an interesting approach on IRC:

docker run --net=<network> $OTHER_RUN_OPTIONS builder-image | docker build -t finalimage -

This is pretty cool, but requires splitting a build into two parts (the pre build to gather resources, the actual build with the gathered resources) and doesn't cover some use-cases, like nested builds.

as of 1.9.0 (planned) the feature from 1.8.0 will be disabled on the default bridge, and will only be available on networks you create yourself

Could this be made a configuration option for Docker instead? So should the default bridge have this feature or not? And then the host can decide? In this way it can be made backwards compatible, with Docker 1.9 hosts just have to enable it, while the new default would be that it is disabled?

@mitar see #17375

However I think in the long run moby/libnetwork#722 is more promising, as changing things for an existing network is something people are going to want to do at some point anyway.

@aidanhs: That looks great! We really have a problem because without a way to turn on the 1.8 behavior for the whole Docker daemon we will have issues migrating to 1.9 all our infrastructure. It is really sad that a feature is being removed just so, without any easy backwards compatibility in mind.