vendor: docker/libnetwork b3507428be5b458cb0e2b4086b13531fb0706e46 by thaJeztah · Pull Request #41908 · moby/moby

thaJeztah · 2021-01-20T16:52:00Z

also somewhat related to #28589, which discusses the "implicit" binding of IPv6

depends on:

Update rootlesskit to v0.13.1 to fix handling of IPv6 addresses #42021 Update rootlesskit to v0.13.1 to fix handling of IPv6 addresses
- Refactor ParsePortSpec to handle IPv6 addresses, and improve validation rootless-containers/rootlesskit#213 Refactor ParsePortSpec to handle IPv6 addresses, and improve validation
CI: update tests to be more resilient against CLI output format and for libnetwork changes #42050 CI: update tests to be more resilient against CLI output format and for libnetwork changes
~~[DNM] trigger CI with rootlesskit v0.14.0-pre #42102~~ update rootlesskit to v0.14.0 #42179 update rootlesskit to v0.14.0
- Port API: support specifying IP version explicitly ("tcp4", "tcp6") rootless-containers/rootlesskit#232 Port API: support specifying IP version explicitly ("tcp4", "tcp6")

full diff: moby/libnetwork@fa125a3...b350742

fixed IPv6 iptables rules for enabled firewalld (fixed IPv6 iptables rules for enabled firewalld libnetwork#2609)
- fixes Docker uses 'iptables' instead of 'ip6tables' for IPv6 NAT rule, crashes #41861 "Docker uses 'iptables' instead of 'ip6tables' for IPv6 NAT rule, crashes"
Fix regression in Fix regression in docker-proxy libnetwork#2608
- introduced in "Fix IPv6 Port Forwarding for the Bridge Driver" (Fix IPv6 Port Forwarding for the Bridge Driver libnetwork#2604)
- fixes IPv4 and IPv6 addresses are not bound by default anymore libnetwork#2607 "IPv4 and IPv6 addresses are not bound by default anymore"
- fixes IPv6 is no longer proxied by default anymore #41858 "IPv6 is no longer proxied by default anymore"
Use hostIP to decide on Portmapper version (Use hostIP to decide on Portmapper version libnetwork#2616)
- fixes docker-proxy not being stopped correctly

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

thaJeztah · 2021-01-20T16:53:45Z

@arkodg PTAL; can you check if I have the correct "fixes" in place or if I missed any / linked the wrong ones?

thaJeztah · 2021-01-21T09:33:58Z

Lots of failures; look related / legit failures;

=== FAIL: arm64.integration.container TestNetworkLoopbackNat (0.28s)
    nat_test.go:66: assertion failed: error is not nil: Error response from daemon: driver failed programming external connectivity on endpoint server-TestNetworkLoopbackNat (5a4eaaf3ff84ad959dc875a4829767a89dd3da6bc472e3f6feba32241b54d982): Bind for :::8080 failed: port is already allocated

=== RUN   TestNetworkLocalhostTCPNat
--- FAIL: TestNetworkLocalhostTCPNat (0.15s)
    nat_test.go:48: assertion failed: error is not nil: Error response from daemon: driver failed programming external connectivity on endpoint server-TestNetworkLocalhostTCPNat (09844ee44b347259889cac7df6f0a9ac78af25af4a690166d4e404eed2d01845): Error starting userland proxy: error while calling PortManager.AddPort(): listen tcp: address :::8081: too many colons in address

=== RUN   TestDockerSuite/TestPortExposeHostBinding
    --- FAIL: TestDockerSuite/TestPortExposeHostBinding (0.54s)
        docker_cli_port_test.go:314: assertion failed: expression is false: err == nil: out: 0.0.0.0:49153
            :::49153

=== RUN   TestDockerSuite/TestPortHostBinding
   --- FAIL: TestDockerSuite/TestPortHostBinding (0.53s)
       docker_cli_port_test.go:293: assertion failed: error is not nil: different size lists 0.0.0.0:9876
           :::9876
           , 2, 1

=== RUN   TestDockerSuite/TestPsNotShowPortsOfStoppedContainer
    --- FAIL: TestDockerSuite/TestPsNotShowPortsOfStoppedContainer (0.50s)
        docker_cli_ps_test.go:603: assertion failed: :::5000->5000/tcp (fields[len(fields)-2] string) != 0.0.0.0:5000->5000/tcp (expected string): Expected: 0.0.0.0:5000->5000/tcp, got: :::5000->5000/tcp

=== RUN   TestDockerSuite/TestRunAllocatePortInReservedRange
    --- FAIL: TestDockerSuite/TestRunAllocatePortInReservedRange (0.80s)
        docker_cli_run_test.go:2071: invalid port, got: 49155
            , error: strconv.ParseInt: parsing "49155\n": invalid syntax

=== RUN   TestDockerSuite/TestRunAllowPortRangeThroughExpose
    --- FAIL: TestDockerSuite/TestRunAllowPortRangeThroughExpose (0.71s)
        docker_cli_run_test.go:2282: Port is not mapped for the port 3000/tcp

thaJeztah · 2021-01-21T19:34:31Z

@bboehmke PTAL

bboehmke · 2021-01-22T14:36:29Z

failure:

=== FAIL: arm64.integration.container TestNetworkLoopbackNat (0.28s)
    nat_test.go:66: assertion failed: error is not nil: Error response from daemon: driver failed programming external connectivity on endpoint server-TestNetworkLoopbackNat (5a4eaaf3ff84ad959dc875a4829767a89dd3da6bc472e3f6feba32241b54d982): Bind for :::8080 failed: port is already allocated

is caused because the docker-proxy process for IPv6 is not removed after the container is stopped.

I think this is caused because the releasePort selects the wrong portMapper.

On creation the bnd.IP is used: https://github.com/moby/libnetwork/blob/master/drivers/bridge/port_mapping.go#L142
On release the bnd.HostIP is used: https://github.com/moby/libnetwork/blob/master/drivers/bridge/port_mapping.go#L212

For the IPv6 userland proxy the bnd.HostIP is :: and the bnd.IP is a IPv4.
This causes also an error:

could not release {tcp 172.17.0.2 80 :: 80 80} because of port is not mapped

bboehmke · 2021-01-22T14:58:52Z

failure 3

=== RUN   TestDockerSuite/TestPortExposeHostBinding
    --- FAIL: TestDockerSuite/TestPortExposeHostBinding (0.54s)
        docker_cli_port_test.go:314: assertion failed: expression is false: err == nil: out: 0.0.0.0:49153
            :::49153

failure 4

=== RUN   TestDockerSuite/TestPortHostBinding
   --- FAIL: TestDockerSuite/TestPortHostBinding (0.53s)
       docker_cli_port_test.go:293: assertion failed: error is not nil: different size lists 0.0.0.0:9876
           :::9876
           , 2, 1

failure 5

=== RUN   TestDockerSuite/TestPsNotShowPortsOfStoppedContainer
    --- FAIL: TestDockerSuite/TestPsNotShowPortsOfStoppedContainer (0.50s)
        docker_cli_ps_test.go:603: assertion failed: :::5000->5000/tcp (fields[len(fields)-2] string) != 0.0.0.0:5000->5000/tcp (expected string): Expected: 0.0.0.0:5000->5000/tcp, got: :::5000->5000/tcp

failure 6

=== RUN   TestDockerSuite/TestRunAllocatePortInReservedRange
    --- FAIL: TestDockerSuite/TestRunAllocatePortInReservedRange (0.80s)
        docker_cli_run_test.go:2071: invalid port, got: 49155
            , error: strconv.ParseInt: parsing "49155\n": invalid syntax

I think they are caused because the container now also shows the IPv6 publish (previously only IPv4).
Here an example:

root@docker-test:~# docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS                               NAMES
2eea9954bb0a   nginx     "/docker-entrypoint.…"   9 seconds ago   Up 8 seconds   0.0.0.0:80->80/tcp, :::80->80/tcp   nginx_test

bboehmke · 2021-01-22T15:06:44Z

failure 7:

=== RUN   TestDockerSuite/TestRunAllowPortRangeThroughExpose
    --- FAIL: TestDockerSuite/TestRunAllowPortRangeThroughExpose (0.71s)
        docker_cli_run_test.go:2282: Port is not mapped for the port 3000/tcp

similar to the ones before but this time it expects that the binding is only correct if there is exactly one binding:
https://github.com/moby/moby/blob/master/integration-cli/docker_cli_run_test.go#L2281

The inspect output would look like this:

            "Ports": {
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "80"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "80"
                    }
                ]
            },

bboehmke · 2021-01-22T15:27:31Z

Sadly for the second one I have no idea what is going on.
Maybe it is also related to the first one but I can not test this locally because the integration test is not running on my system.

Sadly I don't think I will have the time to fix this on my own in the next days.
But at least I hope this information helps to fix the issues/tests.

arkodg · 2021-01-25T16:49:38Z

thanks for triaging the issues @bboehmke !
@thaJeztah can you please include moby/libnetwork#2616 in this PR ?

The port bindings have now been split up into 2 - for ipv4 and ipv6

This helps set up port mapping using iptables for ipv4 bindings and ip6tables for ipv6 bindings in a cleaner way
earlier a binding using something like -p 3000:3000 set up 2 bindings if using userland-proxy and 1 if using
iptables .
To resolve the tests failures, we can either look for 2 bindings - for 0.0.0.0 and :: or disable userland proxy & ipv6

thaJeztah · 2021-01-26T13:00:45Z

The port bindings have now been split up into 2 - for ipv4 and ipv6

Looks like we should at least mention that in the changelog. Also /cc @simonferquel @djs55 as I think Docker Desktop makes use of that information to setup networking

thaJeztah · 2021-01-26T13:05:33Z

@thaJeztah can you please include moby/libnetwork#2616 in this PR ?

updated the commit to include that PR. Will have a look later at fixing the tests

thaJeztah · 2021-02-05T15:44:37Z

Looks like this failure is an issue in rootless kit;

=== RUN   TestNetworkNat
--- FAIL: TestNetworkNat (0.22s)
    nat_test.go:30: assertion failed: error is not nil: Error response from daemon: driver failed programming external connectivity on endpoint server-TestNetworkNat (65cc966cfc2fd79fcfaf0063f3afa7174e8854ce9dfce04e7f9cd4cd0b1e0268): Error starting userland proxy: error while calling PortManager.AddPort(): listen tcp: address :::8080: too many colons in address

Probably somewhere in this function, which doesn't look to be handling IPv6 addresses; https://github.com/rootless-containers/rootlesskit/blob/27c9611a3931149dd91fabd2d559140c351bf36f/pkg/port/portutil/portutil.go#L13-L15

thaJeztah · 2021-02-05T18:14:38Z

I thought I'd have a look at the rootless kit code (thought; replace with the go-connections code used in Docker and Libnetwork), but I think this change; rootless-containers/rootlesskit#207 is problematic; the "port spec" string format is already ambiguous in docker ([ip-address]:[port or port-range]:[port or port-range][/protocol]), but could still be handled (knowing that port or port-range would always be either "empty", "integer", or "integer-integer"). Having two IP addresses in the format makes things a lot more complicated if IPv6 is added to the equation (using [] could somewhat help for ::, but [::], or 2001:db8::1 I think would still be problematic.
Can (for now) probably work around it by making sure we use the IPv4 for that test. (test also looks to not put [] around IPv6 addresses, which probably should, as its <ip>:<port>)

AkihiroSuda · 2021-02-26T05:18:12Z

$ docker --context=rootless run --rm -p 8080:80 nginx:alpine
docker: Error response from daemon: driver failed programming external connectivity on endpoint vigorous_archimedes (8ca94868817b1f0398ae1c96aa4caa3eedc60753b826797ce164fa33a159a966): Error starting userland proxy: error while calling PortManager.AddPort(): listen tcp 0.0.0.0:8080: bind: address already in use.

$ sudo execsnoop.bt 
Attaching 2 probes...
TIME(ms)   PID   ARGS
22491      43131 docker --context=rootless run --rm -p 8080:80 nginx:alpine
22542      43141 /sbin/iptables --wait -t nat -C DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:80
22545      43142 /sbin/iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:80
22547      43143 /sbin/iptables --wait -t filter -C DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport
22548      43144 /sbin/iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport
22550      43145 /sbin/iptables --wait -t nat -C POSTROUTING -p tcp -s 172.17.0.2 -d 172.17.0.2 --dport 80 -j MASQUERADE
22551      43146 /sbin/iptables --wait -t nat -A POSTROUTING -p tcp -s 172.17.0.2 -d 172.17.0.2 --dport 80 -j MASQUERADE
22553      43147 /usr/local/bin/rootlesskit-docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.17.0.2 -container-port 80
22555      43152 docker-proxy -container-ip 172.17.0.2 -container-port 80 -host-ip 127.0.0.1 -host-port 8080 -proto tcp
22556      43160 /usr/local/bin/rootlesskit-docker-proxy -proto tcp -host-ip :: -host-port 8080 -container-ip 172.17.0.2 -container-port 80
22564      43167 /sbin/iptables --wait -t nat -C DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:80
22568      43168 /sbin/iptables --wait -t nat -D DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:80
22571      43169 /sbin/iptables --wait -t filter -C DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport
22573      43170 /sbin/iptables --wait -t filter -D DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport
22574      43171 /sbin/iptables --wait -t nat -C POSTROUTING -p tcp -s 172.17.0.2 -d 172.17.0.2 --dport 80 -j MASQUERADE
22576      43172 /sbin/iptables --wait -t nat -D POSTROUTING -p tcp -s 172.17.0.2 -d 172.17.0.2 --dport 80 -j MASQUERADE

rootlesskit-docker-proxy is unexpectedly called twice:

/usr/local/bin/rootlesskit-docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.17.0.2 -container-port 80
/usr/local/bin/rootlesskit-docker-proxy -proto tcp -host-ip :: -host-port 8080 -container-ip 172.17.0.2 -container-port 80

Seems a bug on libnetwork side.

arkodg · 2021-02-26T06:31:23Z

@AkihiroSuda the behavior in libnetwork has recently changed to spawn a separate docker-proxy process for each port binding type - ipv4 & ipv6, this might look non optimal when in userland-proxy mode, but enables creating correct port forwarding rules for ipv4 and ipv6 bindings using iptables & ip6tables respectively

AkihiroSuda · 2021-03-01T07:18:26Z

Opened rootless-containers/rootlesskit#231 for discussion

AkihiroSuda · 2021-03-01T16:40:10Z

Please try this version of RootlessKit: rootless-containers/rootlesskit#232

thaJeztah · 2021-03-01T21:14:48Z

included #42102 to test

thaJeztah · 2021-03-24T21:08:18Z

Rebased; this should be ready for review: @AkihiroSuda @tonistiigi @tiborvass @cpuguy83 ptal

The normalizing was updated with the output of the "docker port" command in mind, but we're normalizing the "expected" output, which is passed without the "->" in front of the mapping, causing some tests to fail; === RUN TestDockerSuite/TestPortHostBinding --- FAIL: TestDockerSuite/TestPortHostBinding (1.21s) docker_cli_port_test.go:324: assertion failed: error is not nil: |:::9876!=[::]:9876| === RUN TestDockerSuite/TestPortList --- FAIL: TestDockerSuite/TestPortList (0.96s) docker_cli_port_test.go:25: assertion failed: error is not nil: |:::9876!=[::]:9876| Signed-off-by: Sebastiaan van Stijn <[email protected]>

full diff: moby/libnetwork@fa125a3...b350742 - fixed IPv6 iptables rules for enabled firewalld (libnetwork#2609) - fixes "Docker uses 'iptables' instead of 'ip6tables' for IPv6 NAT rule, crashes" - Fix regression in docker-proxy - introduced in "Fix IPv6 Port Forwarding for the Bridge Driver" (libnetwork#2604) - fixes/addresses: "IPv4 and IPv6 addresses are not bound by default anymore" (libnetwork#2607) - fixes/addresses "IPv6 is no longer proxied by default anymore" (moby#41858) - Use hostIP to decide on Portmapper version - fixes docker-proxy not being stopped correctly Port mapping of containers now contain separatet mappings for IPv4 and IPv6 addresses, when listening on "any" IP address. Various tests had to be updated to take multiple mappings into account. Signed-off-by: Sebastiaan van Stijn <[email protected]>

thaJeztah · 2021-03-25T08:39:39Z

-		old := strings.Replace(mapping, "-> [", "-> ", 1)
+		old := strings.Replace(mapping, "[", "", 1)


I made the normalisation too strict in #42050, and forgot the "expected" format was sometimes passed as a "full" format (80/tcp -> [::]:80), and sometimes only the mapping ([::]:80); pushed this fix as a separate commit in this PR, but I can move to a separate PR if we think that makes more sense

thaJeztah · 2021-03-25T21:32:55Z

ok this is all green now; also prepared a draft backport for 20.10 #42205

vendor: docker/libnetwork b3507428be5b458cb0e2b4086b13531fb0706e46 Signed-off-by: wuguoguoya <[email protected]>

thaJeztah added status/2-code-review area/networking Networking kind/bugfix PR's that fix bugs labels Jan 20, 2021

thaJeztah added this to the 20.10.3 milestone Jan 20, 2021

thaJeztah force-pushed the bump_libnetwork branch from b3c7687 to 43abca6 Compare January 20, 2021 16:52

thaJeztah requested a review from tianon as a code owner January 20, 2021 16:52

thaJeztah added the status/failing-ci Indicates that the PR in its current state fails the test suite label Jan 21, 2021

thaJeztah added the impact/changelog label Jan 26, 2021

thaJeztah changed the title ~~vendor: docker/libnetwork 448016ef11309bd67541dcf4d72f1f5b7de94862~~ vendor: docker/libnetwork b3507428be5b458cb0e2b4086b13531fb0706e46 Jan 26, 2021

thaJeztah force-pushed the bump_libnetwork branch from 43abca6 to 64d023c Compare January 26, 2021 13:05

terencehonles mentioned this pull request Jan 27, 2021

IPv4 and IPv6 addresses are not bound by default anymore moby/libnetwork#2607

Closed

thaJeztah modified the milestones: 20.10.3, 20.10.4 Feb 2, 2021

thaJeztah added the process/cherry-pick label Feb 2, 2021

thaJeztah modified the milestones: 20.10.4, 21.xx Feb 2, 2021

thaJeztah mentioned this pull request Feb 11, 2021

Refactor ParsePortSpec to handle IPv6 addresses, and improve validation rootless-containers/rootlesskit#213

Merged

thaJeztah force-pushed the bump_libnetwork branch from 64d023c to f29af39 Compare February 11, 2021 13:46

AkihiroSuda mentioned this pull request Mar 1, 2021

Port API: support specifying IP version explicitly ("tcp4", "tcp6") rootless-containers/rootlesskit#231

Closed

thaJeztah force-pushed the bump_libnetwork branch from 24b7ffe to f90314f Compare March 1, 2021 21:13

AkihiroSuda removed the status/failing-ci Indicates that the PR in its current state fails the test suite label Mar 4, 2021

AkihiroSuda approved these changes Mar 4, 2021

View reviewed changes

thaJeztah force-pushed the bump_libnetwork branch from f90314f to 72991ac Compare March 11, 2021 18:06

thaJeztah mentioned this pull request Mar 22, 2021

update rootlesskit to v0.14.0 #42179

Merged

thaJeztah force-pushed the bump_libnetwork branch from 72991ac to 62cb4b2 Compare March 22, 2021 18:34

This was referenced Mar 22, 2021

[20.10 backport] update rootlesskit to v0.14.0 #42186

Merged

[20.10 backport] CI: update tests to be more resilient against CLI output format and for libnetwork changes #42196

Merged

thaJeztah force-pushed the bump_libnetwork branch from 62cb4b2 to b072776 Compare March 24, 2021 21:07

thaJeztah marked this pull request as ready for review March 24, 2021 21:07

thaJeztah added 2 commits March 25, 2021 00:19

thaJeztah force-pushed the bump_libnetwork branch from b072776 to 0450728 Compare March 24, 2021 23:19

thaJeztah commented Mar 25, 2021

View reviewed changes

thaJeztah mentioned this pull request Mar 25, 2021

[20.10 backport] vendor: docker/libnetwork b3507428be5b458cb0e2b4086b13531fb0706e46 #42205

Merged

5 tasks

tiborvass approved these changes Mar 29, 2021

View reviewed changes

tiborvass merged commit d7a5abe into moby:master Mar 29, 2021

thaJeztah deleted the bump_libnetwork branch March 29, 2021 20:51

winiiiii pushed a commit to winiiiii/moby that referenced this pull request Mar 30, 2021

Merge pull request moby#41908 from thaJeztah/bump_libnetwork

cd9dfbf

vendor: docker/libnetwork b3507428be5b458cb0e2b4086b13531fb0706e46 Signed-off-by: wuguoguoya <[email protected]>

winiiiii pushed a commit to winiiiii/moby that referenced this pull request Mar 30, 2021

Merge pull request moby#41908 from thaJeztah/bump_libnetwork

9fe2e8d

vendor: docker/libnetwork b3507428be5b458cb0e2b4086b13531fb0706e46 Signed-off-by: wuguoguoya <[email protected]>

AkihiroSuda added process/cherry-picked and removed process/cherry-pick labels Apr 1, 2021

		old := strings.Replace(mapping, "-> [", "-> ", 1)
		old := strings.Replace(mapping, "[", "", 1)

Uh oh!

Conversation

thaJeztah commented Jan 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Jan 20, 2021

Uh oh!

thaJeztah commented Jan 21, 2021

Uh oh!

thaJeztah commented Jan 21, 2021

Uh oh!

bboehmke commented Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bboehmke commented Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bboehmke commented Jan 22, 2021

Uh oh!

bboehmke commented Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arkodg commented Jan 25, 2021

Uh oh!

thaJeztah commented Jan 26, 2021

Uh oh!

thaJeztah commented Jan 26, 2021

Uh oh!

thaJeztah commented Feb 5, 2021

Uh oh!

thaJeztah commented Feb 5, 2021

Uh oh!

AkihiroSuda commented Feb 26, 2021

Uh oh!

arkodg commented Feb 26, 2021

Uh oh!

AkihiroSuda commented Mar 1, 2021

Uh oh!

AkihiroSuda commented Mar 1, 2021

Uh oh!

thaJeztah commented Mar 1, 2021

Uh oh!

thaJeztah commented Mar 24, 2021

Uh oh!

thaJeztah Mar 25, 2021

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Mar 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

thaJeztah commented Jan 20, 2021 •

edited

Loading

bboehmke commented Jan 22, 2021 •

edited

Loading

bboehmke commented Jan 22, 2021 •

edited

Loading

bboehmke commented Jan 22, 2021 •

edited

Loading