Skip to content

client: stream containers serially to conserve memory#12846

Open
ningmingxiao wants to merge 1 commit intocontainerd:mainfrom
ningmingxiao:add_list_stream
Open

client: stream containers serially to conserve memory#12846
ningmingxiao wants to merge 1 commit intocontainerd:mainfrom
ningmingxiao:add_list_stream

Conversation

@ningmingxiao
Copy link
Copy Markdown
Contributor

@ningmingxiao ningmingxiao commented Feb 2, 2026 •
edited
Loading

stream containers serially to conserve memory
ctr/nerdctl use
https://github.com/containerd/containerd/blob/v2.2.1/client/containerstore.go#L107 to get all containers list
if every container spec size is big we have to use much memory to store the container list.
what I did:
‌Process one item from the container at a time, without adding it to any list.

fix #12858

use /usr/bin/time -v ctr -n k8s.io c ls

    Command being timed: "./ctr_old -n k8s.io c ls"
    User time (seconds): 0.33
    System time (seconds): 0.24
    Percent of CPU this job got: 79%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.73
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 338328
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 5174
    Voluntary context switches: 7363
    Involuntary context switches: 1172
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

after this pr

    Command being timed: "./ctr_new -n k8s.io c ls"
    User time (seconds): 0.47
    System time (seconds): 0.20
    Percent of CPU this job got: 79%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.85
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 45556
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 1
    Minor (reclaiming a frame) page faults: 24264
    Voluntary context switches: 8915
    Involuntary context switches: 1998
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

Maximum resident set size from 338328 reduce to 45556

@fuweid @mikebrow @mxpv @AkihiroSuda

@github-project-automation github-project-automation Bot moved this to Needs Triage in Pull Request Review Feb 2, 2026
@dosubot dosubot Bot added the area/client Go client label Feb 2, 2026
@ningmingxiao ningmingxiao force-pushed the add_list_stream branch 3 times, most recently from ad0d6c5 to e65d34f Compare February 2, 2026 14:49
@k8s-ci-robot k8s-ci-robot added size/M and removed size/S labels Feb 2, 2026
@ningmingxiao ningmingxiao changed the title client: add ListStream func client: add ListStream func to make container list use less memory Feb 2, 2026
mxpv
mxpv reviewed Feb 2, 2026
View reviewed changes
Comment thread client/client.go Outdated
Comment thread client/client.go Outdated
@mxpv mxpv moved this from Needs Triage to Needs Update in Pull Request Review Feb 2, 2026
@ningmingxiao ningmingxiao force-pushed the add_list_stream branch 3 times, most recently from 074c60f to 9b5eedf Compare February 3, 2026 06:33
@k8s-ci-robot k8s-ci-robot added size/L and removed size/M labels Feb 3, 2026
@ningmingxiao ningmingxiao mentioned this pull request Feb 4, 2026
Open
@ningmingxiao ningmingxiao force-pushed the add_list_stream branch 6 times, most recently from 4a47c4b to 89bcba9 Compare February 4, 2026 10:39
thaJeztah
thaJeztah reviewed Feb 4, 2026
View reviewed changes
Comment thread core/metadata/containers.go Outdated
@ningmingxiao ningmingxiao force-pushed the add_list_stream branch 4 times, most recently from 21641e1 to d821378 Compare February 5, 2026 04:45
@mikebrow
Copy link
Copy Markdown
Member

mikebrow commented Feb 5, 2026

Thoughts: Swapping speed for memory, and vice verse, is a time old issue best left to config option IMO. These types of performance changes are fought over with each performance change to the codebase with some contributors optimizing for speed, some for memory, some for a combination of with resource restrictions. Thus, this preference should not a this or that decision, it should be a choice, possibly even dynamic based on SLA definitions driven by the user/client.

trading time  
from 
   User time (seconds): 0.33
    System time (seconds): 0.24
    Real Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.73
to 
    User time (seconds): 0.47
    System time (seconds): 0.20
    Real Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.85
 to reduce max resident memory..

thaJeztah
thaJeztah reviewed Mar 3, 2026
View reviewed changes
Comment thread integration/client/container_test.go Outdated
@ningmingxiao ningmingxiao force-pushed the add_list_stream branch 6 times, most recently from 0ba7106 to f7bb162 Compare March 10, 2026 01:19
@ningmingxiao ningmingxiao mentioned this pull request Mar 10, 2026
Merged
@ningmingxiao ningmingxiao force-pushed the add_list_stream branch 2 times, most recently from 2df43c6 to dbe301f Compare March 12, 2026 13:05
@ningmingxiao
Copy link
Copy Markdown
Contributor Author

ningmingxiao commented Mar 12, 2026 •
edited
Loading

Here is the newest benchmark test test 10 times.
go test -test.fullpath=true -benchmem -run=^$ -bench ^BenchmarkContainerGet$ github.com/containerd/containerd/v2/integration/client -count=10 -test.root

goos: linux
goarch: amd64
pkg: github.com/containerd/containerd/v2/integration/client
cpu: 12th Gen Intel(R) Core(TM) i5-1240P
BenchmarkContainerGet/use_client.ContainersIter-16         	       1	1367609978 ns/op	1148446144 B/op	   25740 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16         	       2	 685801470 ns/op	1151093208 B/op	   22638 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16         	       2	 855170936 ns/op	1149897316 B/op	   22889 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16         	       2	 717634528 ns/op	1149832872 B/op	   23943 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16         	       2	 669174672 ns/op	1152232800 B/op	   23048 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16         	       2	 709092180 ns/op	1148837944 B/op	   22267 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16         	       2	 683709006 ns/op	1144173296 B/op	   23585 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16         	       2	 722604024 ns/op	1157932904 B/op	   22538 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16         	       2	 710639840 ns/op	1155627768 B/op	   22553 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16         	       2	 784471684 ns/op	1154485252 B/op	   22193 allocs/op
BenchmarkContainerGet/use_client.Containers-16             	       2	 766882093 ns/op	1151349384 B/op	   21831 allocs/op
BenchmarkContainerGet/use_client.Containers-16             	       2	 662555416 ns/op	1147357768 B/op	   23010 allocs/op
BenchmarkContainerGet/use_client.Containers-16             	       2	 591499147 ns/op	1155072848 B/op	   22007 allocs/op
BenchmarkContainerGet/use_client.Containers-16             	       2	 579578408 ns/op	1148404188 B/op	   22635 allocs/op
BenchmarkContainerGet/use_client.Containers-16             	       2	 618664956 ns/op	1153927416 B/op	   22043 allocs/op
BenchmarkContainerGet/use_client.Containers-16             	       2	 573116281 ns/op	1149329852 B/op	   22544 allocs/op
BenchmarkContainerGet/use_client.Containers-16             	       2	 606095926 ns/op	1149221260 B/op	   23990 allocs/op
BenchmarkContainerGet/use_client.Containers-16             	       2	 674345698 ns/op	1152261300 B/op	   23221 allocs/op
BenchmarkContainerGet/use_client.Containers-16             	       2	 612735678 ns/op	1146033480 B/op	   24050 allocs/op
BenchmarkContainerGet/use_client.Containers-16             	       2	 655414452 ns/op	1153429448 B/op	   22434 allocs/op
PASS
ok  	github.com/containerd/containerd/v2/integration/client	134.660s

if let b.Run("use client.Containers", benchmarkGetContainers(ctx, false, client)) first the first test use more resources on my computer(looks strange).

goos: linux
goarch: amd64
pkg: github.com/containerd/containerd/v2/integration/client
cpu: 12th Gen Intel(R) Core(TM) i5-1240P
BenchmarkContainerGet/use_client.Containers-16         	       1	1169215479 ns/op	1145369136 B/op	   22922 allocs/op
BenchmarkContainerGet/use_client.Containers-16         	       2	 650231393 ns/op	1152055924 B/op	   22433 allocs/op
BenchmarkContainerGet/use_client.Containers-16         	       2	 645383831 ns/op	1145809456 B/op	   23405 allocs/op
BenchmarkContainerGet/use_client.Containers-16         	       2	 648004674 ns/op	1151509492 B/op	   24001 allocs/op
BenchmarkContainerGet/use_client.Containers-16         	       2	 632736602 ns/op	1153130812 B/op	   22273 allocs/op
BenchmarkContainerGet/use_client.Containers-16         	       2	 675600908 ns/op	1156901544 B/op	   22916 allocs/op
BenchmarkContainerGet/use_client.Containers-16         	       2	 629460609 ns/op	1154335692 B/op	   21592 allocs/op
BenchmarkContainerGet/use_client.Containers-16         	       2	 609232888 ns/op	1154705712 B/op	   21674 allocs/op
BenchmarkContainerGet/use_client.Containers-16         	       2	 561363646 ns/op	1149515992 B/op	   21751 allocs/op
BenchmarkContainerGet/use_client.Containers-16         	       2	 635951789 ns/op	1145799436 B/op	   23227 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	       2	 615876850 ns/op	1160885820 B/op	   21994 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	       2	 617484624 ns/op	1158515140 B/op	   22863 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	       2	 607622336 ns/op	1150321008 B/op	   21635 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	       2	 575025780 ns/op	1149342500 B/op	   22064 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	       2	 656861868 ns/op	1156331456 B/op	   22469 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	       2	 649953342 ns/op	1154823052 B/op	   21839 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	       2	 589818291 ns/op	1156656092 B/op	   21437 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	       2	 606805622 ns/op	1152891740 B/op	   21542 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	       2	 612953382 ns/op	1151542208 B/op	   21623 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	       2	 637692201 ns/op	1147112600 B/op	   21986 allocs/op

let -benchtime=10s
go test -test.fullpath=true -benchmem -run=^$ -bench ^BenchmarkContainerGet$ github.com/containerd/containerd/v2/integration/client -benchtime=10s -test.root

cpu: 12th Gen Intel(R) Core(TM) i5-1240P
BenchmarkContainerGet/use_client.Containers-16         	      16	 635472949 ns/op	1147701598 B/op	   22838 allocs/op
BenchmarkContainerGet/use_client.ContainersIter-16     	      19	 604692542 ns/op	1151381699 B/op	   22150 allocs/op

The difference is not as obvious as when tested with "time -v".
@thaJeztah @mxpv @dmcgowan

@ningmingxiao
Copy link
Copy Markdown
Contributor Author

ningmingxiao commented Mar 17, 2026

friendly ping @dmcgowan @mxpv

@ningmingxiao
Copy link
Copy Markdown
Contributor Author

ningmingxiao commented Mar 18, 2026

done @mxpv

@ningmingxiao @mikebrow
client: stream containers serially
cfaedbf 
Co-authored-by: Sebastiaan van Stijn <[email protected]>

Signed-off-by: ningmingxiao <[email protected]>
@ningmingxiao
Copy link
Copy Markdown
Contributor Author

ningmingxiao commented Mar 26, 2026

ping @mxpv can this pr be merged?

@mxpv
Copy link
Copy Markdown
Member

mxpv commented Mar 26, 2026

@mikebrow or @thaJeztah PTAL?

This was referenced Mar 31, 2026
Open
Open
@ningmingxiao
Copy link
Copy Markdown
Contributor Author

ningmingxiao commented Apr 15, 2026

ping @mikebrow @thaJeztah

@ningmingxiao
Copy link
Copy Markdown
Contributor Author

ningmingxiao commented Apr 29, 2026

@thaJeztah can you take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mxpv mxpv mxpv approved these changes

@mikebrow mikebrow Awaiting requested review from mikebrow

@thaJeztah thaJeztah Awaiting requested review from thaJeztah

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

area/client Go client size/L

Projects

Pull Request Review
Status: Needs Update

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ctr -n k8s.io c ls use much memory

5 participants

@ningmingxiao @mikebrow @thaJeztah @mxpv @k8s-ci-robot