runtime: RSS seems to have increased in Go 1.24 while the runtime accounting has not #72991
Description
Go version
go version go1.24.1 linux/amd64
Output of go env in your module/workspace:
go env Output
AR='ar' CC='gcc' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_ENABLED='1' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' CXX='g++' GCCGO='gccgo' GO111MODULE='' GOAMD64='v1' GOARCH='amd64' GOAUTH='netrc' GOBIN='' GOCACHE='/home/.../.cache/go-build' GOCACHEPROG='' GODEBUG='' GOENV='/home/.../.config/go/env' GOEXE='' GOEXPERIMENT='' GOFIPS140='off' GOFLAGS='' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1182549097=/tmp/go-build -gno-record-gcc-switches' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMOD='/dev/null' GOMODCACHE='/home/.../go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/home/.../go' GOPRIVATE='' GOPROXY='https://proxy.golang.org,direct' GOROOT='/home/.../sdk/go1.24.1' GOSUMDB='sum.golang.org' GOTELEMETRY='local' GOTELEMETRYDIR='/home/.../.config/go/telemetry' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/home/.../sdk/go1.24.1/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.24.1' GOWORK='' PKG_CONFIG='pkg-config'
What did you do?
I saw @felixge report in Gopher Slack:
After upgrading to go1.24, we have one service that saw its RSS memory usage increase from 1 GiB to 1.2 GiB.
Interestingly, the Go-reported accounting did not seem to change.
I have a simple GC benchmarking utility that has been useful in the past to illustrate some Go runtime problems (or to illustrate something is just expected behavior and rule out bad behavior by the runtime).
I did a quick-and-dirty edit to it to have it run through various permutations of "interesting" memory properties that might be related to the issue @felixge reported, and then tested it via a few nested bash loops that ran through different flag combinations.
Sample run:
$ go install golang.org/dl/go1.23.7@latest && go1.23.7 download
$ go install golang.org/dl/go1.24.1@latest && go1.24.1 download
$ go1.23.7 run github.com/thepudds/heapbench@669420526 -livekind=nodechannel -livesize=large \
-baseheap=512 -garbagerate=128 -leakrate=0 -jobrate=100 -worktime=20ms -stats=2s
$ go1.24.1 run github.com/thepudds/heapbench@669420526 -livekind=nodechannel -livesize=large \
-baseheap=512 -garbagerate=128 -leakrate=0 -jobrate=100 -worktime=20ms -stats=2s
Note this is on an Ubuntu 22.04.5 LTS VM. (Given this seemingly hasn't been reported yet, I wonder if it might depend on the distro or a OS config).
What did you see happen?
RSS (MiB)
mean stdev count
nodechannel-large go1.23 688.8 89.3 44.0
nodechannel-large go1.24 1263.9 81.0 44.0
-----------------------------------------------------
nodechannel-medium go1.23 899.5 64.0 44.0
nodechannel-medium go1.24 889.1 79.7 44.0
-----------------------------------------------------
nodechannel-small go1.23 2945.7 283.3 43.0
nodechannel-small go1.24 2955.0 300.7 43.0
-----------------------------------------------------
intchannel-large go1.23 698.2 71.3 44.0
intchannel-large go1.24 697.5 76.0 44.0
-----------------------------------------------------
intchannel-medium go1.23 868.4 56.3 44.0
intchannel-medium go1.24 867.0 56.4 44.0
-----------------------------------------------------
intchannel-small go1.23 2923.1 247.2 43.0
intchannel-small go1.24 2926.6 280.6 43.0
-----------------------------------------------------
Here, we can see nodechannel-large has a much higher RSS in go1.24 compared to go1.23, whereas the other experiments above have basically the same RSS between go1.24 and go1.23.
- The "large" suffix represents individual allocations should in theory be over 32KiB at a time (so beyond the allocator size classes, and what the allocator calls a large object).
- The "medium" and "small" are below 32KiB.
- The "nodechannel" prefix represents allocations of
chan node, where thenodetype contains pointers. - The "intchannel" prefix represents allocations of
chan int64, whereint64of course does not have pointers.
The node type:
type node struct {
pointers [3]*node
next *node
_ [4]int
}I ran a git bisect, which pointed to https://go.dev/cl/614257. That's part of the big mallocgc refactor and at least in the neighborhood of plausible:
8730fcf88531152c42de9ff1e80d9b3c762d9944 is the first bad commit
Date: Wed Sep 18 21:17:03 2024 +0000
runtime: refactor mallocgc into several independent codepaths
The measured RSS results from the bisect:
RSS (MiB)
mean
20250131-085043-37f27fbecd.out: 1209.3
20241023-151814-3cb0c039e9.out: 1217.0
20240903-222652-f90f7e90b3.out: 650.7
20240925-235600-80143607f0.out: 636.4
20241007-191201-7e2487cf65.out: 633.2
20241021-133902-0492d936c5.out: 636.5
20241021-184247-971ab11ee2.out: 1211.1
20241021-155625-8730fcf885.out: 1209.3
20241021-151809-e3d372aea8.out: 644.1
20241021-154815-8df6413e11.out: 669.9
20241021-154820-6686edc0e7.out: 640.4
20241021-155537-60ee99cf5d.out: 639.3
What did you expect to see?
Ideally, similar RSS values.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status