Skip to content

Conversation

@laurazard
Copy link
Member

@laurazard laurazard commented Mar 13, 2023

Taking over #6114 as it hasn't had activity in a while. Happy to close this PR if the original author wants to do it himself though :)

related:


This adds an (exported) function to make overlay mounts readonly, which is necessary (with the overlayfs driver) to prevent the mounts from entering an unknown state and not working properly.

Also addressed some of the comments on the initial PR.

@k8s-ci-robot
Copy link

Hi @laurazard. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mikebrow
Copy link
Member

/ok-to-test

@laurazard laurazard requested review from AkihiroSuda, dmcgowan and thaJeztah and removed request for AkihiroSuda, dmcgowan and thaJeztah March 15, 2023 16:28
@laurazard
Copy link
Member Author

Oh no I'm still having issues with review requests 😅 I'll just ping @dmcgowan @AkihiroSuda @rumpl @thaJeztah

@laurazard
Copy link
Member Author

laurazard commented Mar 15, 2023

flag provided but not defined: -test.root

(https://github.com/containerd/containerd/actions/runs/4428569995/jobs/7768037954?pr=8259#step:13:287)

Not sure what's going on here, looks like the test is being run with -test.root but that doesn't work on Windows?

Looks like mount is getting picked up as a TEST_REQUIRES_ROOT_PACKAGES in

TEST_REQUIRES_ROOT_PACKAGES=$(filter \

I'll add the testutil import to register the -test.root flag so this won't fail.

@laurazard laurazard force-pushed the readonly-overlay branch 2 times, most recently from 1c1905a to 6354931 Compare March 17, 2023 01:48
This is necessary so we can mount snapshots more than once with overlayfs,
otherwise mounts enter an unknown state.

related: moby/buildkit#1100

Signed-off-by: Laura Brehm <[email protected]>
Co-authored-by: Zou Nengren <[email protected]>
@laurazard
Copy link
Member Author

laurazard commented Mar 17, 2023

The other failures were fixed, and these look unrelated –

=== FAIL: . TestContentClient/CommitErrorState (3.75s)
testsuite.go:122: Cleanup failed: failed to abort c1-commiterror-state: unlinkat /var/lib/containerd-test/io.containerd.content.v1.content/ingest/9101c5f249b4cb38a50dddda86a787c09f31099573c87c525fc50774ac21d05d: directory not empty: unknown

=== FAIL: . TestContainerPTY (10.66s)
container_test.go:2574: expected \x00 in output

@dmcgowan dmcgowan merged commit e2cb6b8 into containerd:main Mar 18, 2023
@thaJeztah thaJeztah added cherry-pick/1.6.x cherry-pick/1.7.x Change to be cherry picked to release/1.7 branch labels Mar 22, 2023
@thaJeztah
Copy link
Member

I marked this PR for cherry-picking for 1.6 and 1.7. Discussed with @laurazard and we'd like to use this functionality and as far as we can see, this should probably be safe to include.

(of course backports will still need to be reviewed, and I asked @laurazard to call out potential concerns on the backports 👍 )

@changweige
Copy link
Member

With this patch applied, the kernel throws another warning [509574.194814] overlayfs: lowerdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.. Is it normal?

@laurazard
Copy link
Member Author

laurazard commented Jun 26, 2023

I haven't noticed that warning, but before this change, as described in #6077, you would get:

kernel: [42256.430385] overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
kernel: [42256.430389] overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.

Which (in practice) is a bigger problem than remounting as lowerdir – as described in #6077:

And later for that container we observe this "undefined behavior" as the RootFS of that container becoming partially not-writable (an attempt to create a file results in "No such file or directory" error).

Are you seeing issues like that?

@changweige
Copy link
Member

I haven't noticed that warning, but before this change, as described in #6077, you would get:

kernel: [42256.430385] overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
kernel: [42256.430389] overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.

Which (in practice) is a bigger problem than remounting as lowerdir – as described in #6077:

And later for that container we observe this "undefined behavior" as the RootFS of that container becoming partially not-writable (an attempt to create a file results in "No such file or directory" error).

Are you seeing issues like that?

Yes, I also got the error "No such file or directory" when reproducing the problem as #6077 said

@laurazard
Copy link
Member Author

The only functional change introduced in this patch is in diff/walking/differ.go – which changes the behaviour there from creating the new mount "normally" to creating the new mount "read-only", which just means without an upper/workdir layer. If the issue you're describing didn't happen before this change, maybe you could share reproduction steps/explain under what scenario you're running into this so we can take a look at it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cherry-picked/1.6.x PR commits are cherry-picked into release/1.6 branch cherry-picked/1.7.x PR commits are cherry-picked into release/1.7 branch ok-to-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Creating diff of a container running on overlayfs generates duplicate mounts

9 participants