Skip to content

Revendor libnetwork @ 36d3bed0e9#37843

Merged
AkihiroSuda merged 3 commits intomoby:masterfrom
microsoft:jjh/boltdb
Sep 14, 2018
Merged

Revendor libnetwork @ 36d3bed0e9#37843
AkihiroSuda merged 3 commits intomoby:masterfrom
microsoft:jjh/boltdb

Conversation

@lowenna
Copy link
Member

@lowenna lowenna commented Sep 13, 2018 •
edited by thaJeztah
Loading

Signed-off-by: John Howard [email protected]

Fixes moby/libnetwork#1950

This relates to a bunch of associated PRs, but ultimately is the fix for moby/libnetwork#1950.

There is a bug in boltdb/bolt on Windows, which is unfortunately no longer maintained. The active maintained fork is now at https://github.com/etcd-io/bbolt. See etcd-io/bbolt#121 for the bug, and etcd-io/bbolt#122 for the fix. The tagged release with the fix is https://github.com/etcd-io/bbolt/releases/tag/v1.3.1-etcd.8.

Unfortunately, boltdb is used all over the place by many components. Hence to get this fix to moby/moby, I've had to go through docker/libkv#202 which is where the database is used by libnetwork; then moby/libnetwork#2268 to pull in the libkv fix; and now this PR to pull everything into moby/moby.

I still cannot remove boltdb/bolt from vendoring here as it's used by (at least) the following components still:

vendor/github.com/containerd/containerd/metadata/bolt.go:       "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/boltutil/helpers.go:   "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/buckets.go:    "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/containers.go: "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/content.go:    "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/db.go: "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/gc.go: "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/images.go:     "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/leases.go:     "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/migrations.go:import "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/namespaces.go: "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/metadata/snapshot.go:   "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/runtime/v1/linux/runtime.go:    "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/services/server/server.go:      "github.com/boltdb/bolt"
vendor/github.com/containerd/containerd/vendor.conf:github.com/boltdb/bolt e9cf4fae01b5a8ff89d0ec6b32f0d9c9f79aefdd
vendor/github.com/docker/swarmkit/agent/config.go:      "github.com/boltdb/bolt"
vendor/github.com/docker/swarmkit/agent/storage.go:     "github.com/boltdb/bolt"
vendor/github.com/docker/swarmkit/agent/worker.go:      "github.com/boltdb/bolt"
vendor/github.com/docker/swarmkit/node/node.go: "github.com/boltdb/bolt"
vendor/github.com/docker/swarmkit/vendor.conf:github.com/boltdb/bolt fff57c100f4dea1905678da7e90d92429dff2904
vendor/github.com/moby/buildkit/cache/metadata.go:      "github.com/boltdb/bolt"
vendor/github.com/moby/buildkit/cache/metadata/metadata.go:     "github.com/boltdb/bolt"
vendor/github.com/moby/buildkit/snapshot/blobmapping/snapshotter.go:    "github.com/boltdb/bolt"
vendor/github.com/moby/buildkit/solver/boltdbcachestorage/storage.go:   "github.com/boltdb/bolt"
vendor/github.com/moby/buildkit/solver/llbsolver/ops/exec.go:   "github.com/boltdb/bolt"
vendor/github.com/moby/buildkit/source/git/gitsource.go:        "github.com/boltdb/bolt"
vendor/github.com/moby/buildkit/source/http/httpsource.go:      "github.com/boltdb/bolt"
vendor/github.com/moby/buildkit/source/local/local.go:  "github.com/boltdb/bolt"
vendor/github.com/moby/buildkit/vendor.conf:github.com/boltdb/bolt e9cf4fae01b5a8ff89d0ec6b32f0d9c9f79aefdd

I've got a PR open to containerd/containerd here: containerd/containerd#2634, but haven't yet opened PRs to docker/swarmkit or moby/buildkit. Those will follow shortly. Update - swarmkit opened. Buildkit depends on containerd PR.

There are three commits in this PR:

  • 1a6e260 pulls in the latest libkv, and adds the required vendor for go.etcd.io/bbolt @ v1.3.1-etcd.8 which has the fix.
  • 5c20890 to update packages outside of vendor in the moby/moby repo which use boltdb/bolt and moves them to go.etcd.io/bbolt
  • 4ddebbe vendors libnetwork to the latest version, and also updates the libnetwork commit in hack/dockerfile/install/proxy.installer. I couldn't find any other places the libnetwork commit was referenced.

@mavenugo @johnstep @thaJeztah PTAL.

@carlfischer1, @JMesser81, @dineshgovindasamy, @daschott FYI.

Phew! 💦 💦 💦

Full diff for Libnetwork;

John Howard added 3 commits September 13, 2018 12:37
Vendor libkv @ 458977154600b9f23984d9f4b82e79570b5ae12b
1a6e260 
Signed-off-by: John Howard <[email protected]>

This also adds go.etcd.io/bbolt as boltdb/bolt is no longer
maintained, and we need etcd-io/bbolt#122 which
was merged in https://github.com/etcd-io/bbolt/releases/tag/v1.3.1-etcd.8
in order to fix moby/libnetwork#1950.

Note that I can't entirely remove boltdb/bolt as it is still used by
other components. Still need to work my way through them.... These include
containerd/containerd (containerd/containerd#2634),
docker/swarmkit; moby/buildkit. And probably more....
boltdb/bolt --> go.etcd.io/bbolt
5c20890 
Signed-off-by: John Howard <[email protected]>
Vendor libnetwork @ 36d3bed0e9
4ddebbe 
Signed-off-by: John Howard <[email protected]>
@lowenna
Copy link
Member Author

lowenna commented Sep 13, 2018

@taylorb-microsoft FYI

@mavenugo
Copy link
Contributor

mavenugo commented Sep 13, 2018

Thanks @jhowardmsft . LGTM

It is a bit wierd though to have both boltdb and bbolt vendored into moby in order to handle the dependency chain. The question is should we wait for other PRs to be merged in other Repos and vendor them all together ? I personally dont think that is required.

@lowenna
Copy link
Member Author

lowenna commented Sep 13, 2018

The question is should we wait for other PRs to be merged in other Repos and vendor them all together ? I personally dont think that is required.

@mavenugo No, it's not required. It is going to take some time to get those through - see for example this comment. containerd/containerd#2634 (comment). I'll try my hardest though to get all components consistent ASAP.

@lowenna lowenna mentioned this pull request Sep 13, 2018
Merged
@lowenna
Copy link
Member Author

lowenna commented Sep 13, 2018 •
edited
Loading

docker/swarmkit PR opened at moby/swarmkit#2748. Just moby/buildkit to go, and then revendoring them back into moby/moby. (The buildkit one is blocked behind the containerd one as buildkit vendors containerd. This is complicated!)

@lowenna lowenna mentioned this pull request Sep 13, 2018
Merged
@thaJeztah thaJeztah added the area/networking Networking label Sep 14, 2018
thaJeztah
thaJeztah approved these changes Sep 14, 2018
View reviewed changes
Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah
Copy link
Member

thaJeztah commented Sep 14, 2018

Thanks! I looked at this a while back as well, and noticed I was opening the same can of worms (many projects using boltdb 😅

@AkihiroSuda AkihiroSuda merged commit 5adee40 into moby:master Sep 14, 2018
@thaJeztah thaJeztah mentioned this pull request Sep 14, 2018
Closed
@lowenna lowenna deleted the jjh/boltdb branch September 18, 2018 16:09
@lowenna lowenna mentioned this pull request Sep 18, 2018
Merged
@lowenna
Copy link
Member Author

lowenna commented Sep 18, 2018

containerd PR --> moby/moby is #37877

@lowenna lowenna mentioned this pull request Sep 18, 2018
Merged
@tonistiigi tonistiigi mentioned this pull request Sep 22, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@thaJeztah thaJeztah thaJeztah approved these changes

@AkihiroSuda AkihiroSuda AkihiroSuda approved these changes

@tianon tianon Awaiting requested review from tianon tianon is a code owner

@tonistiigi tonistiigi Awaiting requested review from tonistiigi tonistiigi is a code owner

@mavenugo mavenugo Awaiting requested review from mavenugo

Assignees

No one assigned

Labels

area/networking Networking status/2-code-review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Saving endpoint to store fails in windows under concurrent load

5 participants

@lowenna @mavenugo @thaJeztah @AkihiroSuda @GordonTheTurtle