feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) by marten-seemann · Pull Request #8680 · ipfs/kubo

marten-seemann · 2022-01-15T15:08:21Z

Part of #8761

go-libp2p v0.18.0 completes the deprecation of ipfs/go-log@v1 and migrates all remaining repos to ipfs/go-log@v2
Adds commands and configuration for libp2p Network Resource Manager
- ipfs swarm limit --help
- ipfs swarm stats --help
Adds config at Swarm.ResourceMgr
- Swarm.ResourceMgr.Enabled is a flag, disabled by default

Closes #8722
Closes #1482 🧙‍♂️

remove the log/tail command go-ipfs-api#263.
Decide where to put these commands
- done in feat(core): add libp2p resource manager #8735 which got merged into this PR
Make command reading config from a file work over HTTP RPC
final cleanup by @lidel – see feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) #8680 (comment)
- Add opt-in env or config flag to control resource manager instantiation, until we are comfortable with production testing to enable by default (similar to improve resource manager integration filecoin-project/lotus#8318)
- Decide where is the limit.json config file going to live: hardcoded in the repo root or configured through go-ipfs-config.

guseggert · 2022-02-09T20:19:00Z

Do you have plans to publish metrics when a RM req is rejected due to exceeded scope limits? That would be very useful operationally to help know when to scale gateway fleets.

marten-seemann · 2022-02-11T12:50:39Z

The resource manager has a tracer integration (see https://github.com/libp2p/go-libp2p-resource-manager/blob/master/trace.go), so it would be possible to hook this up with Prometheus in one way or the other.

vyzo · 2022-02-11T13:15:11Z

travis asked for the same thing so i will add metrics support to the reource manager.

…

On Fri, Feb 11, 2022, 14:50 Marten Seemann ***@***.***> wrote: The resource manager has a tracer integration (see https://github.com/libp2p/go-libp2p-resource-manager/blob/master/trace.go), so it would be possible to hook this up with Prometheus in one way or the other. — Reply to this email directly, view it on GitHub <#8680 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAI4SREFOVU7UPT2MVGGMTU2UA2XANCNFSM5MBCYZ7Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

lidel · 2022-02-21T23:24:38Z

"exchange files" → "cat file" interop test always gets stuck after go -> go2: 8.39 MB (125ms)
(was able to reproduce locally with this PRs binary)

Quick way to reproduce/debug is to run specific test with:
export IPFS_GO_EXEC=/path/to/ipfs-binary-from-libp2p-v018-branch/ipfs
npx ipfs-interop -- -t node --grep "go -> go2: 8.39 MB → works
npx ipfs-interop -- -t node --grep "go -> go2: 67.1 MB" → hangs

Given this PR adds resource manager, I guess some limit is hit?

Update: set DEBUG="ipfsd-ctl:daemon*" IPFS_LOGGING="INFO" in env to see go-ipfs daemon logs.

lidel · 2022-02-24T02:09:08Z

DEBUG="ipfsd-ctl:daemon*" IPFS_LOGGING="DEBUG"  npx ipfs-interop -- -t node --grep "go -> go2: 67.1 MB"

Revealed connection being dropped by go-libp2p-resource-manager:

2022-02-24T02:49:11.848+0100    DEBUG   rcmgr   [email protected]/scope.go:239  blocked memory reservation      {"scope": "peer:12D3KooWReFVRYtypujVVXkvictuB8PyaZnSraLEDvwQPGSDxMaL", "size": 258126, "priority": 128, "error": "resource limit exceeded"}
2022-02-24T02:49:11.849+0100    DEBUG   dht     net/message_manager.go:303      error reading message   {"error": "stream reset", "retrying": true}
2022-02-24T02:49:11.849+0100    DEBUG   swarm2  [email protected]/swarm.go:336    [12D3KooWKyDk1EymJbcr59vVjJeJUvwe6vdAUKXnRtbNVt63pvow] opening stream to peer [12D3KooWReFVRYtypujVVXkvictuB8PyaZnSraLEDvwQPGSDxMaL]
2022-02-24T02:49:11.849+0100    DEBUG   swarm2  [email protected]/swarm_dial.go:241       dialing peer    {"from": "12D3KooWKyDk1EymJbcr59vVjJeJUvwe6vdAUKXnRtbNVt63pvow", "to": "12D3KooWReFVRYtypujVVXkvictuB8PyaZnSraLEDvwQPGSDxMaL"}
2022-02-24T02:49:11.849+0100    DEBUG   swarm2  [email protected]/limiter.go:193  [limiter] adding a dial job through limiter: /ip4/127.0.0.1/tcp/44017
2022-02-24T02:49:11.849+0100    DEBUG   swarm2  [email protected]/limiter.go:161  [limiter] taking FD token: peer: 12D3KooWReFVRYtypujVVXkvictuB8PyaZnSraLEDvwQPGSDxMaL; addr: /ip4/127.0.0.1/tcp/44017; prev consuming: 0
2022-02-24T02:49:11.849+0100    DEBUG   swarm2  [email protected]/limiter.go:167  [limiter] executing dial; peer: 12D3KooWReFVRYtypujVVXkvictuB8PyaZnSraLEDvwQPGSDxMaL; addr: /ip4/127.0.0.1/tcp/44017; FD consuming: 1; waiting: 0
2022-02-24T02:49:11.849+0100    DEBUG   swarm2  [email protected]/swarm_dial.go:386       12D3KooWKyDk1EymJbcr59vVjJeJUvwe6vdAUKXnRtbNVt63pvow swarm dialing 12D3KooWReFVRYtypujVVXkvictuB8PyaZnSraLEDvwQPGSDxMaL /ip4/127.0.0.1/tcp/44017
2022-02-24T02:49:11.849+0100    INFO    bitswap [email protected]/bitswap.go:638        Bitswap ReceiveError: stream reset
2022-02-24T02:49:11.849+0100    DEBUG   bitswap_network network/ipfs_impl.go:413        bitswap net handleNewStream from 12D3KooWReFVRYtypujVVXkvictuB8PyaZnSraLEDvwQPGSDxMaL error: stream reset
2022-02-24T02:49:11.849+0100    INFO    bitswap [email protected]/bitswap.go:638        Bitswap ReceiveError: stream reset
2022-02-24T02:49:11.849+0100    DEBUG   bitswap_network network/ipfs_impl.go:413        bitswap net handleNewStream from 12D3KooWReFVRYtypujVVXkvictuB8PyaZnSraLEDvwQPGSDxMaL error: stream reset
2022-02-24T02:49:11.849+0100    INFO    connmgr connmgr/connmgr.go:546  tried to remove tag from untracked peer: 12D3KooWReFVRYtypujVVXkvictuB8PyaZnSraLEDvwQPGSDxMaL
2022-02-24T02:49:11.849+0100    DEBUG   bs:sprmgr       sessionpeermanager/sessionpeermanager.go:98     Bitswap: removed peer from session      {"session": 1, "peer": "12D3KooWReFVRYtypujVVXkvictuB8PyaZnSraLEDvwQPGSDxMaL", "peerCount": 0}

marten-seemann · 2022-02-24T08:26:09Z

Revealed connection being dropped by go-libp2p-resource-manager

There's no connection being dropped. The resource manager is blocking a memory with prio 128, which is probably a yamux stream window increase: https://github.com/libp2p/go-yamux/blob/cd22a37b789cf6e1fe4f2c2cff6da5df7a49dfac/stream.go#L229. This won't cause any problems.

Do we have logs from the other node?

lidel · 2022-02-24T13:24:14Z

These logs ipfsd-ctl:daemon are from both ends, which make it difficult to debug.

I decided to remove ipfs-interop from the picture and was still able to replicate the issue without it, using two nodes on same machine.

👉 ipfs cat of 65MB file gets stuck

With this setup you can tweak IPFS_LOGGING level for each node independently, and see logs on each end:

Node A (source)

One terminal:

$ export IPFS_PATH=/tmp/node-a
$ ipfs daemon --init --init-profile server,randomports,test

Second terminal:

$ ipfs id | jq '.Addresses[0]' 
{addrA}
$ ipfs swarm connect  {addrB}
$ head -c 65M /dev/urandom | ipfs add -Q
{cid-OK}
$ head -c 8M /dev/urandom | ipfs add -Q
{cid-FAIL}

Node B (destination)

One terminal:

$ export IPFS_PATH=/tmp/node-b
$ ipfs daemon --init --init-profile server,randomports,test

Second terminal:

$ ipfs id | jq '.Addresses[0]' 
{addrB}
$ ipfs swarm connect  {addrA}
$ ipfs cat {cid-OK} > /tmp/8M # finished instantly
$ ipfs cat {cid-FAIL} > /tmp/65M # stops in the middle (around 20-50%)
30.50 MiB / 65.00 MiB [==================================>--------------------------------------]  46.92%

aschmahmann · 2022-02-24T14:17:39Z

@marten-seemann @lidel I'm not totally sure, but am wondering if we're running into a combination of go-bitswap not gracefully handling the lack of streams and go-bitswap using more streams than it should (ipfs/boxo#80).

vyzo · 2022-02-24T15:59:50Z

or yamux not handling properly the refusal yo increase the window from the other side, that also a possibility.

…

On Thu, Feb 24, 2022, 16:17 Adin Schmahmann ***@***.***> wrote: @marten-seemann <https://github.com/marten-seemann> @lidel <https://github.com/lidel> I'm not totally sure, but am wondering if we're running into a combination of go-bitswap not gracefully handling the lack of streams and go-bitswap using more streams than it should ( ipfs/boxo#80 <ipfs/boxo#80>). — Reply to this email directly, view it on GitHub <#8680 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAI4SXHGGPWMEJAWSY2AHLU4Y4ZBANCNFSM5MBCYZ7Q> . You are receiving this because you commented.Message ID: ***@***.***>

BigLep · 2022-03-03T17:18:30Z

2022-03-03 conversation:

@BigLep : create issue for the overall user story to link to
Making config changes without recompiling will likely be key here, even for the initial gateway testing for validating go-libp2p 0.18.

core/node/libp2p/rcmgr.go

* add metrics for the resource manager * export protocol and service name in Prometheus metrics * fix: expose rcmgr metrics only when enabled Co-authored-by: Marcin Rataj <[email protected]>

This includes CI fix for go-ipfs-http-client

This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled We keep vendored copy to ensure go-ipfs is not impacted when go-libp2p decides to change defaults in any of the future releases.

Cleans up the way we initialize defaults and adds a fix for case when connection manager runs with high limits. It also hides `Swarm.ResourceMgr.Limits` until we have a better understanding what syntax makes sense.

lidel · 2022-04-06T22:14:42Z

Applied changes based on feedback from stewards sync:

core/node/libp2p/rcmgr_defaults.go vendors all implicit default limits from go-libp2p
- the number of hardcoded numbers is scary 🙈
- adjustedDefaultLimits is tweaking default limits based on external factors, such as values in existing Swarm.ConnMgr.HighWater config (ported fix from improve resource manager integration filecoin-project/lotus#8318)
Ended up keeping limit.json parser and hiding Swarm.ResourceMgr.Limits for now.
- The way defaults are handled and initialized by go-libp2p is bit complex, there are at least two layers of indirection using different structs.
- Coming up with Swarm.ResourceMgr.Limits that acts as persistence for ipfs swarm limit -s and is compatible with limit.json will improve UX, but that is additional work which should not block go-libp2p 0.18 from being merged.

Before this is merged, we should make a decision if we are ok with

hardcoding all these default limits
- DECISION: remove, and keep only the adjustedDefaultLimits logic (this is in separate PR feat: detect default limit changes in go-libp2p-resource-manager #8857)
not having Swarm.ResourceMgr.Limits in our config and using standalone limit.json for now
- I vote merging as-is, just so we land new go-libp2p. Wiring up Swarm.ResourceMgr.Limits can happen in a future PR.
- Remaining UX improvements extracted to ResourceMgr UX improvements #8858

core/commands/config.go

core/commands/swarm.go

core/node/libp2p/rcmgr_metrics.go

core/node/libp2p/rcmgr.go

docs/config.md

go.mod

* update go-libp2p to v0.18.0 * initialize the resource manager * add resource manager stats/limit commands * load limit file when building resource manager * log absent limit file * write rcmgr to file when IPFS_DEBUG_RCMGR is set * fix: mark swarm limit|stats as experimental * feat(cfg): opt-in Swarm.ResourceMgr This ensures we can safely test the resource manager without impacting default behavior. - Resource manager is disabled by default - Default for Swarm.ResourceMgr.Enabled is false for now - Swarm.ResourceMgr.Limits allows user to tweak limits per specific scope in a way that is persisted across restarts - 'ipfs swarm limit system' outputs human-readable json - 'ipfs swarm limit system new-limits.json' sets new runtime limits (but does not change Swarm.ResourceMgr.Limits in the config) Conventions to make libp2p devs life easier: - 'IPFS_RCMGR=1 ipfs daemon' overrides the config and enables resource manager - 'limit.json' overrides implicit defaults from libp2p (if present) * docs(config): small tweaks * fix: skip libp2p.ResourceManager if disabled This ensures 'ipfs swarm limit|stats' work only when enabled. * fix: use NullResourceManager when disabled This reverts commit b19f7c9eca4cee4187f8cba3389dc2c930258512. after clarification feedback from ipfs/kubo#8680 (comment) * style: rename IPFS_RCMGR to LIBP2P_RCMGR preexisting libp2p toggles use LIBP2P_ prefix * test: Swarm.ResourceMgr * fix: location of opt-in limit.json and rcmgr.json.gz Places these files inside of IPFS_PATH * Update docs/config.md * feat: expose rcmgr metrics when enabled (#8785) * add metrics for the resource manager * export protocol and service name in Prometheus metrics * fix: expose rcmgr metrics only when enabled Co-authored-by: Marcin Rataj <[email protected]> * refactor: rcmgr_metrics.go * refactor: rcmgr_defaults.go This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled We keep vendored copy to ensure go-ipfs is not impacted when go-libp2p decides to change defaults in any of the future releases. * refactor: adjustedDefaultLimits Cleans up the way we initialize defaults and adds a fix for case when connection manager runs with high limits. It also hides `Swarm.ResourceMgr.Limits` until we have a better understanding what syntax makes sense. * chore: cleanup after a review * fix: restore go-ipld-prime v0.14.2 * fix: restore go-ds-flatfs v0.5.1 Co-authored-by: Lucas Molas <[email protected]> Co-authored-by: Marcin Rataj <[email protected]>

* update go-libp2p to v0.18.0 * initialize the resource manager * add resource manager stats/limit commands * load limit file when building resource manager * log absent limit file * write rcmgr to file when IPFS_DEBUG_RCMGR is set * fix: mark swarm limit|stats as experimental * feat(cfg): opt-in Swarm.ResourceMgr This ensures we can safely test the resource manager without impacting default behavior. - Resource manager is disabled by default - Default for Swarm.ResourceMgr.Enabled is false for now - Swarm.ResourceMgr.Limits allows user to tweak limits per specific scope in a way that is persisted across restarts - 'ipfs swarm limit system' outputs human-readable json - 'ipfs swarm limit system new-limits.json' sets new runtime limits (but does not change Swarm.ResourceMgr.Limits in the config) Conventions to make libp2p devs life easier: - 'IPFS_RCMGR=1 ipfs daemon' overrides the config and enables resource manager - 'limit.json' overrides implicit defaults from libp2p (if present) * docs(config): small tweaks * fix: skip libp2p.ResourceManager if disabled This ensures 'ipfs swarm limit|stats' work only when enabled. * fix: use NullResourceManager when disabled This reverts commit b19f7c9eca4cee4187f8cba3389dc2c930258512. after clarification feedback from ipfs/kubo#8680 (comment) * style: rename IPFS_RCMGR to LIBP2P_RCMGR preexisting libp2p toggles use LIBP2P_ prefix * test: Swarm.ResourceMgr * fix: location of opt-in limit.json and rcmgr.json.gz Places these files inside of IPFS_PATH * Update docs/config.md * feat: expose rcmgr metrics when enabled (#8785) * add metrics for the resource manager * export protocol and service name in Prometheus metrics * fix: expose rcmgr metrics only when enabled Co-authored-by: Marcin Rataj <[email protected]> * refactor: rcmgr_metrics.go * refactor: rcmgr_defaults.go This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled We keep vendored copy to ensure go-ipfs is not impacted when go-libp2p decides to change defaults in any of the future releases. * refactor: adjustedDefaultLimits Cleans up the way we initialize defaults and adds a fix for case when connection manager runs with high limits. It also hides `Swarm.ResourceMgr.Limits` until we have a better understanding what syntax makes sense. * chore: cleanup after a review * fix: restore go-ipld-prime v0.14.2 * fix: restore go-ds-flatfs v0.5.1 Co-authored-by: Lucas Molas <[email protected]> Co-authored-by: Marcin Rataj <[email protected]> This commit was moved from ipfs/kubo@514411b

marten-seemann force-pushed the update-libp2p-v018 branch 3 times, most recently from 013048b to 7bd8191 Compare January 15, 2022 16:35

marten-seemann force-pushed the update-libp2p-v018 branch 6 times, most recently from 6f9c3c9 to caba218 Compare February 9, 2022 07:26

marten-seemann mentioned this pull request Feb 9, 2022

create a command for resource manager stats #8722

Closed

schomatis mentioned this pull request Feb 15, 2022

feat(core): add libp2p resource manager #8735

Merged

3 tasks

marten-seemann force-pushed the update-libp2p-v018 branch from caba218 to c3be8e9 Compare February 21, 2022 12:18

marten-seemann force-pushed the update-libp2p-v018 branch from c3be8e9 to 3bb2ecf Compare February 23, 2022 12:09

marten-seemann force-pushed the update-libp2p-v018 branch 2 times, most recently from 6ddf5bf to 0c9ddba Compare February 28, 2022 12:34

BigLep assigned marten-seemann Mar 1, 2022

BigLep mentioned this pull request Mar 1, 2022

go-libp2p v0.18.0 libp2p/go-libp2p#1267

Closed

69 tasks

marten-seemann force-pushed the update-libp2p-v018 branch from fc84c9a to c4fb623 Compare March 2, 2022 05:00

BigLep added this to the go-ipfs 0.13 milestone Mar 3, 2022

BigLep mentioned this pull request Mar 4, 2022

Integrate/expose go-libp2p resource manager #8761

Closed

20 tasks

lidel reviewed Apr 5, 2022

View reviewed changes

core/node/libp2p/rcmgr.go Outdated Show resolved Hide resolved

marten-seemann and others added 5 commits April 6, 2022 00:51

feat: expose rcmgr metrics when enabled (#8785)

41db58d

* add metrics for the resource manager * export protocol and service name in Prometheus metrics * fix: expose rcmgr metrics only when enabled Co-authored-by: Marcin Rataj <[email protected]>

Merge master into update-libp2p-v018

a5f7b2e

This includes CI fix for go-ipfs-http-client

refactor: rcmgr_metrics.go

ee0f58d

refactor: rcmgr_defaults.go

55eeab0

This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled We keep vendored copy to ensure go-ipfs is not impacted when go-libp2p decides to change defaults in any of the future releases.

refactor: adjustedDefaultLimits

bc36572

Cleans up the way we initialize defaults and adds a fix for case when connection manager runs with high limits. It also hides `Swarm.ResourceMgr.Limits` until we have a better understanding what syntax makes sense.

lidel changed the title ~~feat: Swarm.ResourceMgr (go-libp2p v0.18)~~ feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) Apr 7, 2022

Merge master into update-libp2p-v018

75dda8d

BigLep requested a review from guseggert April 7, 2022 16:08

guseggert reviewed Apr 7, 2022

View reviewed changes

lidel mentioned this pull request Apr 7, 2022

ResourceMgr UX improvements #8858

Closed

4 tasks

chore: cleanup after a review

a43a71d

lidel mentioned this pull request Apr 7, 2022

feat: detect default limit changes in go-libp2p-resource-manager #8857

Merged

lidel added 2 commits April 8, 2022 00:55

fix: restore go-ipld-prime v0.14.2

dc994c5

fix: restore go-ds-flatfs v0.5.1

3d81aa0

guseggert approved these changes Apr 8, 2022

View reviewed changes

guseggert merged commit 514411b into master Apr 8, 2022

guseggert deleted the update-libp2p-v018 branch April 8, 2022 01:06

This was referenced Apr 21, 2022

refactor: remove invalid string2code mappings ipfs/go-cid#137

Merged

feat: persist limits to Swarm.ResourceMgr.Limits #8901

Merged

Implement ipfs swarm limit [scope] --reset #8918

Closed

lidel mentioned this pull request May 31, 2022

Improve resource manager UX #9001

Closed

3 tasks

BigLep mentioned this pull request Jul 26, 2022

Release v0.14 #9032

Closed

68 tasks

Uh oh!

Conversation

marten-seemann commented Jan 15, 2022 • edited by lidel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guseggert commented Feb 9, 2022

Uh oh!

marten-seemann commented Feb 11, 2022

Uh oh!

vyzo commented Feb 11, 2022 via email

Uh oh!

lidel commented Feb 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lidel commented Feb 24, 2022

Uh oh!

marten-seemann commented Feb 24, 2022

Uh oh!

lidel commented Feb 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Node A (source)

Node B (destination)

Uh oh!

aschmahmann commented Feb 24, 2022

Uh oh!

vyzo commented Feb 24, 2022 via email

Uh oh!

BigLep commented Mar 3, 2022

Uh oh!

Uh oh!

lidel commented Apr 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

marten-seemann commented Jan 15, 2022 •

edited by lidel

Loading

lidel commented Feb 21, 2022 •

edited

Loading

lidel commented Feb 24, 2022 •

edited

Loading

lidel commented Apr 6, 2022 •

edited

Loading