Linux GCS: Log disk info on ENOSPC errors #1297
Conversation
We've had quite a few times where creating the upper or work directories in the guest fails with ENOSPC but we don't have any view into what the mount looks like at the time we get this. This change just catches any ENOSPC errors when creating an overlayfs mount, calls statfs and logs the disk space and inode info for the mount the failed directory is on. This should make investigating these types of issues much easier. This may be followed up with a change to delete the upper and work directories for a container, as this becomes troublesome with the model for sharing a scratch volume. Signed-off-by: Daniel Canter <[email protected]>
|
@anmaxvl Assigning you as we'd talked about this :) |
helsaawy
left a comment
helsaawy
left a comment
There was a problem hiding this comment.
I am torn between processErrNoSpace and processNoSpaceError, but besides that and other nits, LGTM
@helsaawy I originally had it as processERRNOSPC so we would've had three options! I just landed on ErrNoSpace as its the closest to the actual string rep of the error and in 'Go form', but let me know if you feel strongly on the second. I'm not tied to either |
Signed-off-by: Daniel Canter <[email protected]>
I guess |
anmaxvl
left a comment
anmaxvl
left a comment
There was a problem hiding this comment.
LGTM. I was just checking the logs to see what other places could result in this error and whether we should log the stats there as well, but it looks like we'll eventually end up failing to mount overlay for container, so it's fine to just log it here.
* Linux GCS: Log disk info on ENOSPC errors We've had quite a few times where creating the upper or work directories in the guest fails with ENOSPC but we don't have any view into what the mount looks like at the time we get this. This change just catches any ENOSPC errors when creating an overlayfs mount, calls statfs and logs the disk space and inode info for the mount the failed directory is on. This should make investigating these types of issues much easier. This may be followed up with a change to delete the upper and work directories for a container, as this becomes troublesome with the model for sharing a scratch volume. Signed-off-by: Daniel Canter <[email protected]>
3 participants
We've had quite a few times where creating the upper or work directories in
the guest fails with ENOSPC but we don't have any view into what the mount
looks like at the time we get this. This change just catches any ENOSPC errors
when creating an overlayfs mount, calls statfs and logs the disk space and inode
info for the mount the failed directory is on. This should make investigating
these types of issues much easier.
This may be followed up with a change to delete the upper and work directories
for a container, as this becomes troublesome with the model for sharing a scratch volume.