Skip to content

API: add "prune" events#41259

Merged
tiborvass merged 1 commit intomoby:masterfrom
thaJeztah:add_prune_events
Jul 29, 2020
Merged

API: add "prune" events#41259
tiborvass merged 1 commit intomoby:masterfrom
thaJeztah:add_prune_events

Conversation

@thaJeztah
Copy link
Copy Markdown
Member

@thaJeztah thaJeztah commented Jul 25, 2020

Relates to linuxkit/linuxkit#3539

This patch adds a new "prune" event type to indicate that pruning of a resource
type completed.

This event-type can be used on systems that want to perform actions after
resources have been cleaned up. For example, Docker Desktop performs an fstrim
after resources are deleted (https://github.com/linuxkit/linuxkit/tree/v0.7/pkg/trim-after-delete).

While the current (remove, destroy) events can provide information on most
resources, there is currently no event triggered after the BuildKit build-cache
is cleaned.

Prune events have a reclaimed attribute, indicating the amount of space that
was reclaimed (in bytes). The attribute can be used, for example, to use as a
threshold for performing fstrim actions. Reclaimed space for network events
will always be 0, but the field is added to be consistent with prune events for
other resources.

To test this patch:

Create some resources:

for i in foo bar baz; do \
    docker network create network_$i \
    && docker volume create volume_$i \
    && docker run -d --name container_$i -v volume_$i:/volume busybox sh -c 'truncate -s 5M somefile; truncate -s 5M /volume/file' \
    && docker tag busybox:latest image_$i; \
done;

docker pull alpine
docker pull nginx:alpine

echo -e "FROM busybox\nRUN truncate -s 50M bigfile" | DOCKER_BUILDKIT=1 docker build -

Start listening for "prune" events in another shell:

docker events --filter event=prune

Prune containers, networks, volumes, and build-cache:

docker system prune -af --volumes

See the events that are returned:

docker events --filter event=prune
2020-07-25T12:12:09.268491000Z container prune  (reclaimed=15728640)
2020-07-25T12:12:09.447890400Z network prune  (reclaimed=0)
2020-07-25T12:12:09.452323000Z volume prune  (reclaimed=15728640)
2020-07-25T12:12:09.517236200Z image prune  (reclaimed=21568540)
2020-07-25T12:12:09.566662600Z builder prune  (reclaimed=52428841)

- Description for the changelog

`GET /events` now returns `prune` events after pruning resources have completed.
  Prune events are returned for `container`, `network`, `volume`, `image`, and
  `builder`, and have a `reclaimed` attribute, indicating the amount of space
  reclaimed (in bytes).

- A picture of a cute animal (not mandatory but encouraged)

@thaJeztah
Copy link
Copy Markdown
Member Author

ping @djs55 @tiborvass @AkihiroSuda PTAL

Copy link
Copy Markdown
Contributor

@djs55 djs55 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice -- trim-after-delete could easily consume this.

@thaJeztah
Copy link
Copy Markdown
Member Author

Rebased to trigger CI (failing on a flaky test)

@thaJeztah
Copy link
Copy Markdown
Member Author

This test is flaky (again);

=== RUN   TestDockerSuite/TestLogsFollowSlowStdoutConsumer
    --- FAIL: TestDockerSuite/TestLogsFollowSlowStdoutConsumer (1.41s)
        docker_cli_logs_test.go:243: assertion failed: error is not nil: exit status 1

This patch adds a new "prune" event type to indicate that pruning of a resource
type completed.

This event-type can be used on systems that want to perform actions after
resources have been cleaned up. For example, Docker Desktop performs an fstrim
after resources are deleted (https://github.com/linuxkit/linuxkit/tree/v0.7/pkg/trim-after-delete).

While the current (remove, destroy) events can provide information on _most_
resources, there is currently no event triggered after the BuildKit build-cache
is cleaned.

Prune events have a `reclaimed` attribute, indicating the amount of space that
was reclaimed (in bytes). The attribute can be used, for example, to use as a
threshold for performing fstrim actions. Reclaimed space for `network` events
will always be 0, but the field is added to be consistent with prune events for
other resources.

To test this patch:

Create some resources:

    for i in foo bar baz; do \
        docker network create network_$i \
        && docker volume create volume_$i \
        && docker run -d --name container_$i -v volume_$i:/volume busybox sh -c 'truncate -s 5M somefile; truncate -s 5M /volume/file' \
        && docker tag busybox:latest image_$i; \
    done;

    docker pull alpine
    docker pull nginx:alpine

    echo -e "FROM busybox\nRUN truncate -s 50M bigfile" | DOCKER_BUILDKIT=1 docker build -

Start listening for "prune" events in another shell:

    docker events --filter event=prune

Prune containers, networks, volumes, and build-cache:

    docker system prune -af --volumes

See the events that are returned:

    docker events --filter event=prune
    2020-07-25T12:12:09.268491000Z container prune  (reclaimed=15728640)
    2020-07-25T12:12:09.447890400Z network prune  (reclaimed=0)
    2020-07-25T12:12:09.452323000Z volume prune  (reclaimed=15728640)
    2020-07-25T12:12:09.517236200Z image prune  (reclaimed=21568540)
    2020-07-25T12:12:09.566662600Z builder prune  (reclaimed=52428841)

Signed-off-by: Sebastiaan van Stijn <[email protected]>
@thaJeztah
Copy link
Copy Markdown
Member Author

All green now; PTAL

Copy link
Copy Markdown
Member

@tonistiigi tonistiigi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wouldn't catch the buildkit gc releases. Otherwise LGTM

@thaJeztah
Copy link
Copy Markdown
Member Author

thaJeztah commented Jul 28, 2020 via email

Copy link
Copy Markdown
Contributor

@tiborvass tiborvass left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do GC in a followup

@tiborvass tiborvass merged commit 663d143 into moby:master Jul 29, 2020
@thaJeztah thaJeztah deleted the add_prune_events branch July 29, 2020 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants