Don't fail container list when size lookup fails by drakeo338 · Pull Request #53921 · moby/moby

drakeo338 · 2026-10-12T01:36:23Z

Fixes: containerd image store: docker ps --size and docker system df fail entirely when a file is deleted during a container's size walk (snapshotter.Usage failed … lstat … no such file or directory) #53920

Summary

With size=true, an error from GetContainerLayerSize for a single container (for example the containerd snapshotter failing on a container whose snapshot is gone) failed the whole container list, so one broken container made docker ps -s and docker system df unusable. The list now logs the error at debug level, reports that container's sizes as -1 (as the graphdriver image service does for SizeRw when Size() fails), and keeps the container in the result. TestContainerList_SizeError covers it: it fails before the change and passes after.

AI disclosure: written with Claude Code (AI-assisted, mostly AI-drafted); I reviewed the change and ran the daemon tests.

Release notes (optional)

Fix `docker ps -s` and container listing with sizes failing entirely when the size of one container cannot be computed.

A picture of a cute animal (not mandatory but encouraged)

Skipped.

vvoland · 2026-10-12T10:47:46Z

+
+	ok := setupContainerWithName(t, "ok", d)
+	bad := setupContainerWithName(t, "bad", d)
+	bad.Created = ok.Created.Add(time.Millisecond)


nit: this isn't needed

Suggested change

bad.Created = ok.Created.Add(time.Millisecond)

Removed in 31abdb8, along with the ok variable it left unused.

thaJeztah · 2026-10-12T12:27:15Z

+						// Don't fail the whole listing because the size of a
+						// single container can't be determined.
+						log.G(ctx).WithFields(log.Fields{
+							"error":       err,
+							"containerID": newC.ID,
+						}).Warn("failed to get container layer size")


Looks like in other places we made this a Debug;

moby/daemon/prune.go

Lines 71 to 77 in 553420b

cSize, _, err := daemon.imageService.GetContainerLayerSize(ctx, c.ID)

if err != nil {

// Debug log only, the size is only for informational purposes anyway

log.G(ctx).WithFields(log.Fields{

"error": err,

"container": c.ID,

}).Debug("failed to get layer size for container")

That said, I'm wondering if we should also special-case "not found" errors if they are potentially expected. In that case we could even consider setting the sizes to 0. Could there also be a situation where only one (SizeRw or SizeRootFs) cannot be found? (writable layer already gone, but rootFS not?) 🤔

Looks like there's some handling already for not found in the function we're calling, including some logging (so we could end up with multiple logs for the same error);

moby/daemon/containerd/service.go

Lines 343 to 366 in a0ae589

snapshotter := i.client.SnapshotService(ctr.Driver)

rwLayerUsage, err := snapshotter.Usage(ctx, containerID)

if err != nil {

if cerrdefs.IsNotFound(err) {

return 0, 0, errdefs.NotFound(fmt.Errorf("rw layer snapshot not found for container %s", containerID))

}

return 0, 0, errdefs.System(errors.Wrapf(err, "snapshotter.Usage failed for %s", containerID))

}

unpackedUsage, err := calculateSnapshotParentUsage(ctx, snapshotter, containerID)

if err != nil {

if cerrdefs.IsNotFound(err) {

log.G(ctx).WithField("ctr", containerID).Warn("parent of container snapshot no longer present")

} else {

log.G(ctx).WithError(err).WithField("ctr", containerID).Warn("unexpected error when calculating usage of the parent snapshots")

}

}

log.G(ctx).WithFields(log.Fields{

"rwLayerUsage": rwLayerUsage.Size,

"unpacked": unpackedUsage.Size,

}).Debug("GetContainerLayerSize")

// TODO(thaJeztah): include content-store size for the image (similar to "GET /images/json")

return rwLayerUsage.Size, rwLayerUsage.Size + unpackedUsage.Size, nil

Switched to Debug in 31abdb8. No double log: a missing rw snapshot comes back from service.go as 0, 0 plus NotFound without logging; the parent case it logs returns no error (SizeRootFs just under-counts). SizeRootFs is rw + parents, so if rw fails neither is available. Graphdriver returns 0, 0 for a missing layer, and system df sums SizeRw, so -1 skews it; happy to map NotFound to 0 if you prefer.

vvoland · 2026-10-12T16:07:54Z

@drakeo338 can you please squash the commits?

When listing containers with size=true, an error from GetContainerLayerSize for a single container (for example the containerd snapshotter failing to compute usage for a container whose snapshot is gone) was returned from the worker and made the whole request fail, so one broken container made 'docker ps -s' and 'docker system df' unusable. Log the error at debug level, as prune does for the same error, report the sizes of that container as -1, which is what the graphdriver image service already does when it cannot compute the size, and keep the container in the result. Signed-off-by: drakeo338 <[email protected]>

drakeo338 · 2026-10-13T12:26:22Z

Squashed into one commit, 9f89432, with the same tree as before.

vvoland

Thanks, LGTM!

github-actions Bot added the area/daemon Core Engine label Oct 12, 2026

vvoland reviewed Oct 12, 2026

View reviewed changes

vvoland added this to the 29.9.1 milestone Oct 12, 2026

vvoland added the kind/bugfix PR's that fix bugs label Oct 12, 2026

thaJeztah reviewed Oct 12, 2026

View reviewed changes

drakeo338 force-pushed the claude/53920-fix branch from 31abdb8 to 9f89432 Compare October 13, 2026 12:26

vvoland added the impact/changelog label Oct 13, 2026

vvoland approved these changes Oct 13, 2026

View reviewed changes

vvoland merged commit cdfbf7d into moby:master Oct 14, 2026
235 of 237 checks passed

github-actions Bot mentioned this pull request Oct 15, 2026

chore(deps): update 4 dependencies (Moshi, Docker, Rust, and more) progamesigner/sleipnir#24

Merged

3 tasks

Navigation Menu

Repository navigation

Don't fail container list when size lookup fails - #53921

Don't fail container list when size lookup fails#53921
vvoland merged 1 commit into
moby:masterfrom
drakeo338:claude/53920-fix

drakeo338 commented Oct 12, 2026 •

edited

Loading

Uh oh!

vvoland Oct 12, 2026

Uh oh!

drakeo338 Oct 12, 2026

Uh oh!

thaJeztah Oct 12, 2026

Uh oh!

thaJeztah Oct 12, 2026

Uh oh!

drakeo338 Oct 12, 2026

Uh oh!

vvoland commented Oct 12, 2026

Uh oh!

drakeo338 commented Oct 13, 2026

Uh oh!

vvoland left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	cSize, _, err := daemon.imageService.GetContainerLayerSize(ctx, c.ID)
	if err != nil {
	// Debug log only, the size is only for informational purposes anyway
	log.G(ctx).WithFields(log.Fields{
	"error": err,
	"container": c.ID,
	}).Debug("failed to get layer size for container")

	snapshotter := i.client.SnapshotService(ctr.Driver)
	rwLayerUsage, err := snapshotter.Usage(ctx, containerID)
	if err != nil {
	if cerrdefs.IsNotFound(err) {
	return 0, 0, errdefs.NotFound(fmt.Errorf("rw layer snapshot not found for container %s", containerID))
	}
	return 0, 0, errdefs.System(errors.Wrapf(err, "snapshotter.Usage failed for %s", containerID))
	}

	unpackedUsage, err := calculateSnapshotParentUsage(ctx, snapshotter, containerID)
	if err != nil {
	if cerrdefs.IsNotFound(err) {
	log.G(ctx).WithField("ctr", containerID).Warn("parent of container snapshot no longer present")
	} else {
	log.G(ctx).WithError(err).WithField("ctr", containerID).Warn("unexpected error when calculating usage of the parent snapshots")
	}
	}
	log.G(ctx).WithFields(log.Fields{
	"rwLayerUsage": rwLayerUsage.Size,
	"unpacked": unpackedUsage.Size,
	}).Debug("GetContainerLayerSize")

	// TODO(thaJeztah): include content-store size for the image (similar to "GET /images/json")
	return rwLayerUsage.Size, rwLayerUsage.Size + unpackedUsage.Size, nil

Conversation

drakeo338 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Release notes (optional)

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

vvoland Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

drakeo338 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

thaJeztah Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

thaJeztah Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

drakeo338 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

vvoland commented Oct 12, 2026

Uh oh!

drakeo338 commented Oct 13, 2026

Uh oh!

vvoland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

drakeo338 commented Oct 12, 2026 •

edited

Loading