Repository navigation
setup-gcp: raise node ARP table and inotify sysctls for high sandbox density - #2444
Merged
Benjamin Elder (BenTheElder) merged 4 commits intoOct 16, 2026
Conversation
Chenyi Wang (chw120)
requested review from
Benjamin Elder (BenTheElder) and
Aditya Shantanu (aditya-shantanu)
October 10, 2026 00:32
Bowei Du (bowei)
previously approved these changes
Oct 10, 2026
Contributor
is IPv6 relevant for Substrate? |
Contributor
|
/lgtm |
Chenyi Wang (chw120)
force-pushed
the
fix/raise-node-arp-inotify-sysctls
branch
from
October 14, 2026 00:02
3a75974 to
e376e54
Compare
Collaborator
Author
In Substrate, per-sandbox networking is IPv4-only today. Also looks GKE rejects |
Chenyi Wang (chw120)
force-pushed
the
fix/raise-node-arp-inotify-sysctls
branch
from
October 14, 2026 00:08
e376e54 to
5f36098
Compare
Bowei Du (bowei)
added
area/dev-infra
area/network
area/node
kind/bug
Benjamin Elder (BenTheElder)
previously approved these changes
Oct 16, 2026
Benjamin Elder (BenTheElder)
left a comment
Benjamin Elder (BenTheElder)
left a comment
Collaborator
There was a problem hiding this comment.
meta point: we should really update substrate-gke, and setup-gcp needs to go ...
Collaborator
Author
I see. Once merged, I will update substrate-gke with the same changes. |
Benjamin Elder (BenTheElder)
approved these changes
Oct 16, 2026
2 tasks done
2 tasks done
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Each awake actor sandbox creates its own network namespace, veth/tap pair, and inotify watches on the host kernel (even in multi-actor worker pods). At the default COS kernel limits (
net.ipv4.neigh.default.gc_thresh3 = 1024,fs.inotify.max_user_instances = 128,fs.inotify.max_user_watches = 8192–65536), nodes hitinotifyexhaustion around ~800 live sandboxes and ARP table overflow (neighbour: arp_cache: neighbor table overflow!) around ~1,000 live sandboxes per node, causing new sandbox creation/networking to fail (100%ping failure at 1,000 sandboxes/node;0%failure up to 1,500 sandboxes/node once raised).This PR:
LinuxNodeConfig.Sysctlson the defaultsubstrate-node-poolcreated bytools/setup-gcp/cmd/cluster.go:net.ipv4.neigh.default.gc_thresh1 = 4096net.ipv4.neigh.default.gc_thresh2 = 8192net.ipv4.neigh.default.gc_thresh3 = 16384fs.inotify.max_user_instances = 65536fs.inotify.max_user_watches = 1048576(Note: GKE creates an IPv4-only VPC cluster by default and rejects
net.ipv6.neigh.default.gc_thresh*inLinuxNodeConfig.Sysctls.)tools/setup-gcp/cmd/cluster_test.goverifying the node pool'sLinuxNodeConfig.Sysctls.tools/setup-gcp/README.md:--system-config-from-file=systemconfig.yamlwhen creating additional node pools (e.g., bare-metal pools viagcloud container node-pools create).substrate-sysctl-tunerDaemonSet on an already-running node pool without rolling/recreating nodes.Testing
go test ./tools/setup-gcp/cmd/... ./cmd/ate-setup/...