Migrate etcd driver from 'go-etcd' to `etcd/client` by abronan · Pull Request #75 · docker/libkv

abronan · 2015-10-02T22:13:19Z

Migrates the etcd driver codebase from coreos/go-etcd (deprecated) to coreos/etcd/client.

Amongst the notable changes you can now specify an IsDir parameter to WriteOptions which allows to circumvent the directory/key distinction on creating a key with Put and watching that key with Watch.

Until etcd APIv3 is finalized, a conservative measure is to pass isDir:true to Put before calling Watch on that key so that it is considered as a directory for etcd. Consul and zookeeper are ignoring this parameter.

(Tests have been slightly changed notably for AtomicPut because 0 is interpreted as ignore with the new client)

This needs to be tested against swarm and libnetwork first.

Fixes: #53
Related to moby/libnetwork#577

Signed-off-by: Alexandre Beslic [email protected]

abronan · 2015-10-02T22:20:54Z

/cc @mavenugo @tomdee

With this PR, we can solve the create key + watch issue in libnetwork by just giving IsDir:true to WriteOptions when doing a Put. Much more reasonable than my first patch until this is fixed with the new etcd API.

mavenugo · 2015-10-05T08:19:12Z

@abronan that is a good idea.
@tomdee can you give it a try asap and let us know if that solves your issue in libnetwork ?
(pls use this along with moby/libnetwork#582).

This commit migrates the old 'go-etcd' client, which is deprecated to the new 'coreos/etcd/client'. One notable change is the ability to specify an 'IsDir' parameter to the 'Put' call. This allows to circumvent the limitations of etcd regarding the key/directory distinction while setting up Watches on a directory. A conservative measure to set up a watch that should be used the same way for etcd/consul/zookeeper is to enforce the 'IsDir' parameter with 'WriteOptions' on 'Put' to avoid the 'NotANode' error thrown by etcd on Watch call. Consul and zookeeper are not using the 'IsDir' parameter. Signed-off-by: Alexandre Beslic <[email protected]>

tomdee · 2015-10-05T17:51:47Z

I'll try it out this afternoon.

tomdee · 2015-10-05T20:36:12Z

Using the ux branch of docker with this libkv branch and the linked libnetwork branch, I'm hitting the following build errors.

Using Building: bundles/1.9.0-dev/binary/docker-1.9.0-dev
vendor/src/github.com/docker/libkv/store/etcd/etcd.go:14:2: cannot find package "github.com/coreos/etcd/client" in any of:
    /usr/local/go/src/github.com/coreos/etcd/client (from $GOROOT)
    /go/src/github.com/coreos/etcd/client (from $GOPATH)
    /go/src/github.com/docker/docker/vendor/src/github.com/coreos/etcd/client
vendor/src/github.com/docker/libnetwork/hostdiscovery/hostdiscovery.go:9:2: cannot find package "github.com/deckarep/golang-set" in any of:
    /usr/local/go/src/github.com/deckarep/golang-set (from $GOROOT)
    /go/src/github.com/deckarep/golang-set (from $GOPATH)
    /go/src/github.com/docker/docker/vendor/src/github.com/deckarep/golang-set
Makefile:44: recipe for target 'binary' failed
make: *** [binary] Error 1

abronan · 2015-10-05T20:39:39Z

@tomdee I had the same issue when testing, you need to make sure those packages are in the vendor folder of docker, either by adding the entries in vendor.sh or go get -u those repositories + copy them in the vendor folder. Then make binary should work 😉

tomdee · 2015-10-05T20:43:46Z

vendor.sh takes so long to run since it deletes and clones everything so I just checked the packages out manually, e.g. git clone [email protected]:coreos/etcd.git vendor/src/github.com/coreos/etcd

I'm hitting another issue now, I guess because I'm using the ux branch on docker/docker and it's incompatible with your libnetwork branch.

# github.com/docker/docker/api/server/router/network
api/server/router/network/network_routes.go:205: calling method String with receiver &ip (type **net.IPNet) requires explicit dereference
api/server/router/network/network_routes.go:209: calling method String with receiver &ipv6 (type **net.IPNet) requires explicit dereference
Makefile:44: recipe for target 'binary' failed
make: *** [binary] Error 1

I can fix up those two lines and get it to build, so I'm testing with it now.

tomdee · 2015-10-05T21:12:30Z

Docker was crashing every time I tried to create a container.

I've switched to using the master branch of docker, but docker is still crashing.

git checkout master
git pull
git reset --hard
rm -rf vendor/src/github.com/docker/{libnetwork,libkv}
git clone [email protected]:coreos/etcd.git vendor/src/github.com/coreos/etcd
git clone [email protected]:abronan/libkv.git -b new_etcd_client_migration vendor/src/github.com/docker/libkv
git clone [email protected]:abronan/libnetwork.git -b ensure_key_is_watchable vendor/src/github.com/docker/libnetwork

When I create a container I get

2015/10/05 14:11:51 http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 85 [running]:
net/http.func·011()
    /usr/local/go/src/net/http/server.go:1130 +0xbb
github.com/docker/docker/daemon.(*Container).buildEndpointInfo(0xc2087f0a00, 0x7f70842d8930, 0xc208843a70, 0xc2087e2fc0, 0xc2087e2fc0, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/container_unix.go:625 +0x3f0
github.com/docker/docker/daemon.(*Container).updateEndpointNetworkSettings(0xc2087f0a00, 0x7f708430d660, 0xc2080697a0, 0x7f70842d8930, 0xc208843a70, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/container_unix.go:661 +0x1c8
github.com/docker/docker/daemon.(*Container).configureNetwork(0xc2087f0a00, 0x13c2090, 0x4, 0xc208821af0, 0xb, 0xc2087ce2f0, 0x4, 0xc208821a00, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/container_unix.go:887 +0x39a
github.com/docker/docker/daemon.(*Container).allocateNetwork(0xc2087f0a00, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/container_unix.go:849 +0x4c0
github.com/docker/docker/daemon.(*Container).initializeNetworking(0xc2087f0a00, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/container_unix.go:954 +0x4c6
github.com/docker/docker/daemon.(*Container).Start(0xc2087f0a00, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/container.go:276 +0x236
github.com/docker/docker/daemon.(*Daemon).ContainerStart(0xc208065500, 0xc208826e17, 0x40, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/start.go:47 +0x24f
github.com/docker/docker/api/server/router/local.(*router).postContainersStart(0xc2086cfa00, 0x7f70842d4300, 0xc208832b10, 0x7f70842d42c8, 0xc2088361e0, 0xc2086cdad0, 0xc2088329f0, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/router/local/container.go:184 +0x162
github.com/docker/docker/api/server/router/local.*router.(github.com/docker/docker/api/server/router/local.postContainersStart)·fm(0x7f70842d4300, 0xc208832b10, 0x7f70842d42c8, 0xc2088361e0, 0xc2086cdad0, 0xc2088329f0, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/router/local/local.go:133 +0x7b
github.com/docker/docker/api/server.func·004(0x7f70842d4300, 0xc208832b10, 0x7f70842d42c8, 0xc2088361e0, 0xc2086cdad0, 0xc2088329f0, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/middleware.go:87 +0x7c7
github.com/docker/docker/api/server.func·003(0x7f70854ada50, 0xc208028fd0, 0x7f70842d42c8, 0xc2088361e0, 0xc2086cdad0, 0xc2088329f0, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/middleware.go:66 +0x10e
github.com/docker/docker/api/server.func·002(0x7f70854ada50, 0xc208028fd0, 0x7f70842d42c8, 0xc2088361e0, 0xc2086cdad0, 0xc2088329f0, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/middleware.go:49 +0x47c
github.com/docker/docker/api/server.func·001(0x7f70854ada50, 0xc208028fd0, 0x7f70842d42c8, 0xc2088361e0, 0xc2086cdad0, 0xc2088329f0, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/middleware.go:27 +0x1dc
github.com/docker/docker/api/server.func·007(0x7f70842d42c8, 0xc2088361e0, 0xc2086cdad0)
    /go/src/github.com/docker/docker/api/server/server.go:153 +0x265
net/http.HandlerFunc.ServeHTTP(0xc208120220, 0x7f70842d42c8, 0xc2088361e0, 0xc2086cdad0)
    /usr/local/go/src/net/http/server.go:1265 +0x41
github.com/gorilla/mux.(*Router).ServeHTTP(0xc2086ddc70, 0x7f70842d42c8, 0xc2088361e0, 0xc2086cdad0)
    /go/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x297
net/http.serverHandler.ServeHTTP(0xc20803e840, 0x7f70842d42c8, 0xc2088361e0, 0xc2086cdad0)
    /usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc208836140)
    /usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1751 +0x35e

mavenugo · 2015-10-05T21:15:01Z

@tomdee sorry about that. the existing changes in libnetwork needs some minor fixes on the docker side to make a clean vendor-in. Can you please do this specific change : mavenugo/docker@6107884#diff-832e27741e70367e5566a720c88b8f32L646 and it should work fine after that.

abronan · 2015-10-05T21:15:24Z

It's with the new UX right? I remember testing without it and it was working so not sure if related or not.

mavenugo · 2015-10-05T21:17:00Z

@abronan i think @tomdee is trying with the latest from libnetwork (which has the entire IPAM driver chagnes). That requires the fix that I mentioned above in docker. this will be included as part of the IPAM driver inclusion in docker when we actually do the vendor-in.

tomdee · 2015-10-05T22:04:47Z

@abronan @mavenugo I was trying exactly the branches I listed above 😄

Master docker with the two branches from abronan

git clone [email protected]:coreos/etcd.git vendor/src/github.com/coreos/etcd
git clone [email protected]:abronan/libkv.git -b new_etcd_client_migration vendor/src/github.com/docker/libkv
git clone [email protected]:abronan/libnetwork.git -b ensure_key_is_watchable vendor/src/github.com/docker/libnetwork

I'm guessing that this merge moby/moby@33e9d70 broke things with with the version of libnetwork from abronan

abronan · 2015-10-05T22:09:35Z

Yeah I was probably using a version of libnetwork without those changes being introduced 😕

Will retry with master once things are stabilized with the new IPAM driver.

vieux · 2015-10-05T22:50:49Z

LGTM

abronan · 2015-10-05T22:56:25Z

After a pass of testing on swarm and libnetwork (even if pre-IPAM..), that first step migration to the new client works fine. Will soon tag a release and enhance the doc to explain how to setup watches with this change to work cross-backends.

Merging as this is a blocker for other PRs.

Thanks @tomdee for testing! Let's move this discussion in moby/libnetwork#582

Migrate etcd driver from 'go-etcd' to `etcd/client`

tomdee · 2015-10-05T23:08:15Z

👍

abronan added kind/enhancement drivers store/etcd status/2-code-review labels Oct 2, 2015

abronan mentioned this pull request Oct 4, 2015

Ensure Keys are watchable with etcd moby/libnetwork#582

Closed

abronan force-pushed the new_etcd_client_migration branch from f19ad23 to 156c780 Compare October 5, 2015 15:52

abronan force-pushed the new_etcd_client_migration branch 2 times, most recently from 968f38c to d635a8e Compare October 5, 2015 20:00

dhiltgen pushed a commit to dhiltgen/swarm that referenced this pull request Oct 5, 2015

Sync with docker/libkv#75

79aa96f

dhiltgen mentioned this pull request Oct 5, 2015

Add TLS support for libkv docker-archive/classicswarm#1254

Merged

abronan added a commit that referenced this pull request Oct 5, 2015

Merge pull request #75 from abronan/new_etcd_client_migration

22a9d7c

Migrate etcd driver from 'go-etcd' to `etcd/client`

abronan merged commit 22a9d7c into docker:master Oct 5, 2015

abronan deleted the new_etcd_client_migration branch October 5, 2015 22:56

dhiltgen mentioned this pull request Oct 5, 2015

Add TLS support for discovery backend moby/moby#16644

Merged

code-ape mentioned this pull request Feb 2, 2017

Add support for ACL #29

Open

thaJeztah mentioned this pull request Jul 2, 2023

libnetwork: remove dead / unused code from datastore and kvstore moby/moby#45870

Merged

Conversation

abronan commented Oct 2, 2015

Uh oh!

abronan commented Oct 2, 2015

Uh oh!

mavenugo commented Oct 5, 2015

Uh oh!

tomdee commented Oct 5, 2015

Uh oh!

tomdee commented Oct 5, 2015

Uh oh!

abronan commented Oct 5, 2015

Uh oh!

tomdee commented Oct 5, 2015

Uh oh!

tomdee commented Oct 5, 2015

Uh oh!

mavenugo commented Oct 5, 2015

Uh oh!

abronan commented Oct 5, 2015

Uh oh!

mavenugo commented Oct 5, 2015

Uh oh!

tomdee commented Oct 5, 2015

Uh oh!

abronan commented Oct 5, 2015

Uh oh!

vieux commented Oct 5, 2015

Uh oh!

abronan commented Oct 5, 2015

Uh oh!

tomdee commented Oct 5, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants