[release/2.0] Update remote content to break up writes to avoid grpc message size limits #11457

jsternberg · 2025-02-28T17:30:26Z

backport proxy: break up writes from the remote writer to avoid grpc limits #11441
fixes Remote content store can exceed grpc size limits when performing a write #11440

The remote content writer proxy already has the capability to break up
large files into multiple writes, but the current API doesn't recognize
when it's about to exceed the limits and attempts to send the data over
grpc in one message instead of breaking it into multiple messages.

This changes the behavior of Write to automatically break up the size
of the content based on the max send message size.

(cherry picked from commit f25f36c)

The remote content writer proxy already has the capability to break up large files into multiple writes, but the current API doesn't recognize when it's about to exceed the limits and attempts to send the data over grpc in one message instead of breaking it into multiple messages. This changes the behavior of `Write` to automatically break up the size of the content based on the max send message size. Signed-off-by: Jonathan A. Sternberg <[email protected]> (cherry picked from commit f25f36c)

k8s-ci-robot · 2025-02-28T17:30:36Z

Hi @jsternberg. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

thaJeztah · 2025-02-28T17:34:53Z

core/content/proxy/content_writer.go

+		end := i + maxBufferSize
+		if end > len(p) {
+			end = len(p)
+		}
+		data := p[i:end]


For others; the backport is slightly different than the original as this one doesn't use slices.Chunk; see #11441 (review)

thaJeztah

LGTM

k8s-ci-robot added size/M needs-ok-to-test labels Feb 28, 2025

dosubot bot added the kind/performance label Feb 28, 2025

thaJeztah requested a review from dmcgowan February 28, 2025 17:31

thaJeztah changed the title ~~proxy: break up writes from the remote writer to avoid grpc limits~~ [release/2.0 backport] proxy: break up writes from the remote writer to avoid grpc limits Feb 28, 2025

thaJeztah reviewed Feb 28, 2025

View reviewed changes

thaJeztah approved these changes Feb 28, 2025

View reviewed changes

thaJeztah added ok-to-test and removed needs-ok-to-test labels Feb 28, 2025

dmcgowan changed the title ~~[release/2.0 backport] proxy: break up writes from the remote writer to avoid grpc limits~~ [release/2.0] Update remote content to break up writes to avoid grpc message size limits Feb 28, 2025

dmcgowan approved these changes Feb 28, 2025

View reviewed changes

dmcgowan added impact/changelog and removed kind/performance labels Feb 28, 2025

dmcgowan merged commit 84e6cba into containerd:release/2.0 Feb 28, 2025
58 checks passed

jsternberg deleted the backport-remote-write-grpc-limits-exceeded branch February 28, 2025 18:52

dmcgowan mentioned this pull request Feb 28, 2025

[release/2.0] Prepare release notes for v2.0.3 #11443

Merged

github-actions bot mentioned this pull request Mar 2, 2025

Bump containerd from 1.7.21 to 2.0.3 kokyhm/kubespray#151

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[release/2.0] Update remote content to break up writes to avoid grpc message size limits #11457

[release/2.0] Update remote content to break up writes to avoid grpc message size limits #11457

jsternberg commented Feb 28, 2025 •

edited by thaJeztah

Loading

k8s-ci-robot commented Feb 28, 2025

thaJeztah Feb 28, 2025

thaJeztah left a comment

[release/2.0] Update remote content to break up writes to avoid grpc message size limits #11457

[release/2.0] Update remote content to break up writes to avoid grpc message size limits #11457

Conversation

jsternberg commented Feb 28, 2025 • edited by thaJeztah Loading

k8s-ci-robot commented Feb 28, 2025

thaJeztah Feb 28, 2025

Choose a reason for hiding this comment

thaJeztah left a comment

Choose a reason for hiding this comment

jsternberg commented Feb 28, 2025 •

edited by thaJeztah

Loading