implement btrfs.Usage by AkihiroSuda · Pull Request #1836 · containerd/containerd

AkihiroSuda · 2017-11-30T07:52:54Z

For active snapshot, the excl value of L0 QGroup is used, and
committed on Commit.
For view snapshot, always 0 is returned.
For committed snapshot, the committed value is returned.

Signed-off-by: Akihiro Suda [email protected]

Close #1801
Depends on containerd/btrfs#14

Result after ctr images pull docker.io/library/ubuntu:latest on btrfs:

root@ws01:~# export CONTAINERD_SNAPSHOTTER=btrfs 
root@ws01:~# ctr snapshot tree                                                                 
\_ sha256:788ce2310e2fdbbf81fe21cbcc8a44da4cf648b0339b09c221abacb4cd5fd136                     
  \_ sha256:2eb2bdc452f61ae4703cb67be52a4d595265cecb257f3210647ea1d0bf84a549                   
    \_ sha256:289566599b45e8e2a7264923d3481c6e0e7460232a4e0ed1512528272fb15c70                 
      \_ sha256:1f7b04df09e72e9b94e923567a168b438d195c4c610a335ed7320cc6dea93c3f                                                                                                               
        \_ sha256:928cadf24454047a7a1b8e8d034a539e882ed904d6627b343e2abcd338ae0123                                                                                                             
root@ws01:~# ctr snapshot usage                                                                                                                                                                
KEY                                                                     SIZE      INODES                                                                                                       
sha256:1f7b04df09e72e9b94e923567a168b438d195c4c610a335ed7320cc6dea93c3f 68.0 KiB  -1                                                                                                           
sha256:289566599b45e8e2a7264923d3481c6e0e7460232a4e0ed1512528272fb15c70 80.0 KiB  -1                                                                                                           
sha256:2eb2bdc452f61ae4703cb67be52a4d595265cecb257f3210647ea1d0bf84a549 208.0 KiB -1                                                                                                           
sha256:788ce2310e2fdbbf81fe21cbcc8a44da4cf648b0339b09c221abacb4cd5fd136 123.7 MiB -1                                                                                                           
sha256:928cadf24454047a7a1b8e8d034a539e882ed904d6627b343e2abcd338ae0123 48.0 KiB  -1

on overlayfs:

root@ws01:~# export CONTAINERD_SNAPSHOTTER=overlayfs
root@ws01:~# ctr snapshot tree
\_ sha256:788ce2310e2fdbbf81fe21cbcc8a44da4cf648b0339b09c221abacb4cd5fd136                     
  \_ sha256:2eb2bdc452f61ae4703cb67be52a4d595265cecb257f3210647ea1d0bf84a549                   
    \_ sha256:289566599b45e8e2a7264923d3481c6e0e7460232a4e0ed1512528272fb15c70                 
      \_ sha256:1f7b04df09e72e9b94e923567a168b438d195c4c610a335ed7320cc6dea93c3f               
        \_ sha256:928cadf24454047a7a1b8e8d034a539e882ed904d6627b343e2abcd338ae0123             
root@ws01:~# ctr snapshot usage                
KEY                                                                     SIZE      INODES       
sha256:1f7b04df09e72e9b94e923567a168b438d195c4c610a335ed7320cc6dea93c3f 14.7 KiB  4            
sha256:289566599b45e8e2a7264923d3481c6e0e7460232a4e0ed1512528272fb15c70 20.0 KiB  22           
sha256:2eb2bdc452f61ae4703cb67be52a4d595265cecb257f3210647ea1d0bf84a549 48.7 KiB  21           
sha256:788ce2310e2fdbbf81fe21cbcc8a44da4cf648b0339b09c221abacb4cd5fd136 119.8 MiB 5518         
sha256:928cadf24454047a7a1b8e8d034a539e882ed904d6627b343e2abcd338ae0123 12.0 KiB  4

The results are slightly different but should not hurt.

- For active snapshot, the `excl` value of L0 QGroup is used, and committed on Commit. - For view snapshot, always 0 is returned. - For committed snapshot, the committed value is returned. Signed-off-by: Akihiro Suda <[email protected]>

codecov-io · 2017-11-30T08:03:08Z

Codecov Report

Merging #1836 into master will decrease coverage by 0.1%.
The diff coverage is 32.69%.

@@            Coverage Diff             @@
##           master    #1836      +/-   ##
==========================================
- Coverage   49.17%   49.07%   -0.11%     
==========================================
  Files          86       86              
  Lines        8518     8568      +50     
==========================================
+ Hits         4189     4205      +16     
- Misses       3659     3687      +28     
- Partials      670      676       +6

Flag	Coverage Δ
#linux	`49.17% <32.69%> (-0.11%)`	⬇️
#windows	`44.25% <ø> (ø)`	⬆️

Impacted Files	Coverage Δ
snapshots/btrfs/btrfs.go	`48.83% <32.69%> (-3.36%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 96ca973...bcb66d8. Read the comment docs.

dmcgowan · 2017-11-30T23:09:49Z

+	if err != nil {
+		return snapshots.Usage{}, err
+	}
+	if err := btrfs.Sync(target); err != nil {


How confident are we that after this Sync is called the info is accurate?

As accurate as btrfs qgroup show --sync: https://github.com/kdave/btrfs-progs/blob/9fe889ac02b9c49b885c8999f5dd4e192697fa83/cmds-qgroup.c#L377-L382

man: http://manpages.ubuntu.com/manpages/zesty/man8/btrfs-qgroup.8.html

--sync To retrieve information after updating the state of qgroups, force sync of the filesystem identified by <path> before getting information.

dmcgowan · 2017-11-30T23:10:17Z

 			return nil, err
 		}
 	}
+	if err = btrfs.QuotaCtl(target, true); err != nil {


What is the cost of setting the quota in terms of extra memory, cpu, or disk?

https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-quota#PERFORMANCE_IMPLICATIONS
says:

PERFORMANCE IMPLICATIONS When the quotas are turned on, they affect all extent processing, taking a performance hit. It is not recommended to turn on qgroups unless the user intends to actually use them.

Should we make this optional?
If so, what value should we return when quota is disabled?
We might be able to use "walking" method, but it's confusing to have two different definitions of disk usage. (So I suggest returning -1)

czm4514 · 2017-12-04T08:40:38Z

Now, I want to use btrfs file system instead of overlayfs ,Which version supports btrfs，or it is still not supportted?

AkihiroSuda · 2017-12-04T09:31:42Z

@czm4514 From README:

With the understood caveat of distro kernel versioning, a reasonable starting point for Linux is a minimum 4.x kernel version.
The overlay filesystem snapshotter, used by default, uses features that were finalized in the 4.x kernel series. If you choose to use btrfs, there may be more flexibility in kernel version (minimum recommended is 3.18), but will require the btrfs kernel module and btrfs tools to be installed on your Linux distribution.

I'm using Ubuntu 17.10 kernel 4.13

implement btrfs.Usage

bcb66d8

- For active snapshot, the `excl` value of L0 QGroup is used, and committed on Commit. - For view snapshot, always 0 is returned. - For committed snapshot, the committed value is returned. Signed-off-by: Akihiro Suda <[email protected]>

AkihiroSuda mentioned this pull request Nov 30, 2017

implement btrfs.Usage #1801

Closed

dmcgowan reviewed Nov 30, 2017

View reviewed changes

dmcgowan mentioned this pull request Dec 4, 2017

Implement btrfs usage #1871

Merged

AkihiroSuda closed this Dec 5, 2017

AkihiroSuda mentioned this pull request Jan 7, 2020

implement Usage containerd/zfs#17

Closed

AkihiroSuda mentioned this pull request Nov 4, 2021

Excessive CPU usage during stat collection for BTRFS #6067

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement btrfs.Usage#1836

implement btrfs.Usage#1836
AkihiroSuda wants to merge 1 commit intocontainerd:masterfrom
AkihiroSuda:btrfs-usage

AkihiroSuda commented Nov 30, 2017

Uh oh!

codecov-io commented Nov 30, 2017 •

edited

Loading

Uh oh!

dmcgowan Nov 30, 2017

Uh oh!

AkihiroSuda Dec 1, 2017

Uh oh!

dmcgowan Nov 30, 2017

Uh oh!

AkihiroSuda Dec 1, 2017 •

edited

Loading

Uh oh!

czm4514 commented Dec 4, 2017

Uh oh!

AkihiroSuda commented Dec 4, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

AkihiroSuda commented Nov 30, 2017

Uh oh!

codecov-io commented Nov 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dmcgowan Nov 30, 2017

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Dec 1, 2017

Choose a reason for hiding this comment

Uh oh!

dmcgowan Nov 30, 2017

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Dec 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

czm4514 commented Dec 4, 2017

Uh oh!

AkihiroSuda commented Dec 4, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-io commented Nov 30, 2017 •

edited

Loading

AkihiroSuda Dec 1, 2017 •

edited

Loading