integration/container: add a base test for C/R by avagin · Pull Request #38452 · moby/moby

avagin · 2018-12-29T23:45:13Z

- What I did

I spent a Friday night to write this test.

- How I did it

Silently

- How to verify it

Just run it

- Description for the changelog

This test creates a container, checkpoints it twice, restore it once and checks that a content of a tmpfs mount is restored correctly.

- A picture of a cute animal (not mandatory but encouraged)

avagin · 2018-12-29T23:49:18Z

Cc: @kolyshkin @rst0git

codecov · 2018-12-30T06:19:24Z

Codecov Report

❗ No coverage uploaded for pull request base (master@9c83848). Click here to learn what that means.
The diff coverage is 0%.

@@            Coverage Diff            @@
##             master   #38452   +/-   ##
=========================================
  Coverage          ?   36.48%           
=========================================
  Files             ?      613           
  Lines             ?    45882           
  Branches          ?        0           
=========================================
  Hits              ?    16741           
  Misses            ?    26847           
  Partials          ?     2294

rst0git · 2018-12-30T08:02:37Z

daemon/checkpoint.go

@lifubang should we close #38028 in favour of this commit?

I don't know. If maintainers think it should close, close it.

I don't have any objections against config.json with some additional information.

rst0git · 2018-12-30T08:18:32Z

integration/container/checkpoint_test.go

criu check --all ?

I think that for this test just checking the required features is enough, checking all would require more effort like enabling uffd and specific tmpfs configurations.
I would say, let's have this and then we can add more specific test cases.

olljanat · 2019-01-01T14:36:08Z

@avagin CI fails which looks to be caused by this change.

Experimental and Janky fails to:

10:01:39 --- FAIL: TestRenameAnonymousContainer (1.72s)
10:01:39     rename_test.go:169: assertion failed: 0 (int) != 1 (inspect.State.ExitCode int): container 8fc6fb78a11c77d8af79ae9e7d73c44ab6c624c10b89c1681484a69ff2b4649e exited with the wrong exitcode: {ContainerJSONBase:0xc00080da20 Mounts:[] Config:0xc0004e03c0 NetworkSettings:0xc0007c8300}

PowerPC and Z fails to:

09:53:38 Running /go/src/github.com/docker/docker/integration/container
09:53:38 INFO: Testing against a local daemon
09:53:38 === RUN   TestCheckpoint
09:53:41 --- FAIL: TestCheckpoint (3.03s)
09:53:41     checkpoint_test.go:55: Error (criu/util.c:816): exited, status=3
09:53:41         Warn  (criu/net.c:2840): Unable to get socket network namespace
09:53:41         Warn  (criu/net.c:2840): Unable to get tun network namespace
09:53:41         Warn  (criu/sk-unix.c:229): unix: Unable to open a socket file: Bad address
09:53:41         Warn  (criu/net.c:2840): Unable to get socket network namespace
09:53:41         Warn  (criu/kerndat.c:881): Can't keep kdat cache on non-tempfs
09:53:41         Looks good.
09:53:41     daemon.go:296: [d572613c2a932] waiting for daemon to start
09:53:41     daemon.go:328: [d572613c2a932] daemon started
09:53:41     checkpoint_test.go:105: ++ type -P true
09:53:41         ++ type -P ip6tables-restore
09:53:41         + mount --bind /bin/true /sbin/ip6tables-restore
09:53:41         ++ type -P true
09:53:41         ++ type -P ip6tables-save
09:53:41         + mount --bind /bin/true /sbin/ip6tables-save
09:53:41     checkpoint_test.go:146: assertion failed: false (false bool) != true (inspect.State.Running bool)
09:53:41     checkpoint_test.go:165: assertion failed: error is not nil: Error response from daemon: cannot start a paused container, try unpause instead
09:53:41     daemon.go:283: [d572613c2a932] exiting daemon
0

fntlnz

LGTM

fntlnz · 2019-01-01T23:13:59Z

I've been able to reproduce this test on my machine following the process that ended with success.

INFO: Testing against a local daemon                                                                                                                      
=== RUN   TestCheckpoint                                                                                                                                  
--- PASS: TestCheckpoint (4.78s)                                                                                                                          
    checkpoint_test.go:55: Error (criu/uffd.c:264): uffd: Lazy pages are not available: Function not implemented                                          
        Warn  (criu/kerndat.c:881): Can't keep kdat cache on non-tempfs                                                                                   
        Looks good.                                                                                                                                       
    daemon.go:296: [d5c0d8008e63a] waiting for daemon to start                                                                                            
    daemon.go:328: [d5c0d8008e63a] daemon started                                                                                                         
    checkpoint_test.go:105: ++ type -P true                                                                                                               
        ++ type -P ip6tables-restore                                                                                                                      
        + mount --bind /bin/true /sbin/ip6tables-restore                                                                                                  
        ++ type -P true                                                                                                                                   
        ++ type -P ip6tables-save                                                                                                                         
        + mount --bind /bin/true /sbin/ip6tables-save                                                                                                     
    daemon.go:283: [d5c0d8008e63a] exiting daemon                                                                                                         
PASS

Not sure why it's complaining about pause,I ran multiple times and haven't experienced that, sending a rebuild to verify if it's flaky.

kolyshkin

Perhaps we can simplify the test by reusing some stuff from integration/internal/container?

integration/container/checkpoint_test.go

kolyshkin · 2019-01-03T20:50:04Z

CI failures

experimental

00:14:35.536 === RUN TestRenameAnonymousContainer
00:14:37.289 --- FAIL: TestRenameAnonymousContainer (1.75s)
00:14:37.289 rename_test.go:169: assertion failed: 0 (int) != 1 (inspect.State.ExitCode int): container baac251d5a1cb2221ffedf6f10acbad166b90e3549601e96d908e76762675a81 exited with the wrong exitcode: {ContainerJSONBase:0xc0007a4840 Mounts:[] Config:0xc000714500 NetworkSettings:0xc000235b00}

Haven't seen this one before, will take a look 🙄

janky

00:24:30.155 === RUN TestServiceUpdateSecrets
00:24:32.287 --- FAIL: TestServiceUpdateSecrets (2.13s)
00:24:32.287 daemon.go:296: [dd677d8ec1bb7] waiting for daemon to start
00:24:32.287 daemon.go:328: [dd677d8ec1bb7] daemon started
00:24:32.287 update_test.go:126: assertion failed: error is not nil: Error response from daemon: rpc error: code = Unknown desc = update out of sequence
00:24:32.287 daemon.go:283: [dd677d8ec1bb7] exiting daemon

Known flaky test; #37547

ppc

01:29:00.842 FAIL: docker_api_swarm_test.go:296: DockerSwarmSuite.TestAPISwarmLeaderElection

Known flaky test; #32673

olljanat · 2019-01-10T16:14:35Z

I've been able to reproduce this test on my machine following the process that ended with success.

@fntlnz how you run it?

I tested this with #38523 like this:

Cherry picked commit: f7c69d5
Renamed TestRenameAnonymousContainer --> TestRenameAnonymousContainer2 and commited it.
Enabled experimental mode with: export DOCKER_EXPERIMENTAL=true
Run flaky finder make test-integration-flaky

and result was that TestCheckpoint and TestRenameAnonymousContainer2 fails on every run. You can see log on: 38452_stress_test.log

kolyshkin · 2019-02-20T19:42:37Z

Derek add label: rebuild/*

kolyshkin · 2019-02-28T19:58:24Z

@avagin can you please squash all the commits dealing with the test case (i.e. everything except the second one) into one?

kolyshkin · 2019-02-28T20:01:48Z

The only CI failure is

01:27:19.604 FAIL: docker_cli_daemon_test.go:118: DockerDaemonSuite.TestDaemonRestartUnlessStopped

which should be fixed by a rebase since #38737 is merged.

Signed-off-by: Andrei Vagin <[email protected]>

A container checkpoint directory doesn't have config.json. Signed-off-by: Andrei Vagin <[email protected]>

kolyshkin · 2019-03-06T00:51:14Z

LGTM // cc @thaJeztah

thaJeztah

LGTM, thanks!

avagin requested a review from vdemeester as a code owner December 29, 2018 23:45

GordonTheTurtle added the status/0-triage label Dec 29, 2018

rst0git mentioned this pull request Dec 30, 2018

Enable checkpoint/restore of containers with TTY #38405

Merged

rst0git reviewed Dec 30, 2018

View reviewed changes

avagin force-pushed the cr-test branch 2 times, most recently from eaac6e8 to 3923a97 Compare December 30, 2018 18:36

derek bot added the rebuild/* label Dec 31, 2018

GordonTheTurtle removed the rebuild/* label Dec 31, 2018

derek bot added status/failing-ci Indicates that the PR in its current state fails the test suite area/testing labels Jan 1, 2019

fntlnz approved these changes Jan 1, 2019

View reviewed changes

fntlnz added rebuild/* area/daemon Core Engine and removed rebuild/* labels Jan 1, 2019

derek bot added rebuild/janky and removed status/failing-ci Indicates that the PR in its current state fails the test suite rebuild/janky labels Jan 2, 2019

GordonTheTurtle removed the rebuild/experimental label Jan 2, 2019

derek bot added the rebuild/z label Jan 2, 2019

GordonTheTurtle removed the rebuild/z label Jan 2, 2019

fntlnz added the rebuild/experimental label Jan 2, 2019

GordonTheTurtle removed the rebuild/experimental label Jan 2, 2019

kolyshkin requested changes Jan 2, 2019

View reviewed changes

integration/container/checkpoint_test.go Outdated Show resolved Hide resolved

integration/container/checkpoint_test.go Outdated Show resolved Hide resolved

integration/container/checkpoint_test.go Outdated Show resolved Hide resolved

avagin force-pushed the cr-test branch from 8611c9b to 45b822b Compare January 3, 2019 16:51

kolyshkin mentioned this pull request Jan 4, 2019

TestRenameAnonymousContainer: fix error msg #38490

Merged

vdemeester added the status/failing-ci Indicates that the PR in its current state fails the test suite label Jan 4, 2019

GordonTheTurtle assigned tiborvass Jan 13, 2019

avagin force-pushed the cr-test branch 2 times, most recently from 2a7a7b1 to 2c88895 Compare February 5, 2019 19:23

derek bot added the rebuild/* label Feb 20, 2019

thaJeztah added rebuild/* and removed status/failing-ci Indicates that the PR in its current state fails the test suite rebuild/* labels Feb 21, 2019

GordonTheTurtle removed the rebuild/* label Feb 21, 2019

avagin added 2 commits February 28, 2019 23:04

integration/container: add a base test for C/R

bd4fff3

Signed-off-by: Andrei Vagin <[email protected]>

Fix CheckpointList

0b96bf8

A container checkpoint directory doesn't have config.json. Signed-off-by: Andrei Vagin <[email protected]>

avagin force-pushed the cr-test branch from 2c88895 to 0b96bf8 Compare March 1, 2019 07:05

thaJeztah approved these changes Mar 7, 2019

View reviewed changes

thaJeztah merged commit 54dddad into moby:master Mar 7, 2019

kolyshkin mentioned this pull request Mar 21, 2019

[WIP] revive config.json for docker checkpoints #38028

Closed

thaJeztah mentioned this pull request Mar 28, 2019

Flaky test: TestCheckpoint #38963

Open

thaJeztah mentioned this pull request Oct 8, 2019

[18.09 backport] assorted test-fixes docker-archive/engine#401

Merged

thaJeztah added the process/cherry-picked label Oct 8, 2019

Conversation

avagin commented Dec 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

avagin commented Dec 29, 2018

Uh oh!

codecov bot commented Dec 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rst0git Dec 30, 2018

Choose a reason for hiding this comment

Uh oh!

lifubang Dec 31, 2018

Choose a reason for hiding this comment

Uh oh!

avagin Jan 1, 2019

Choose a reason for hiding this comment

Uh oh!

rst0git Dec 30, 2018

Choose a reason for hiding this comment

Uh oh!

fntlnz Jan 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olljanat commented Jan 1, 2019

Uh oh!

fntlnz left a comment

Choose a reason for hiding this comment

Uh oh!

fntlnz commented Jan 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kolyshkin commented Jan 3, 2019

experimental

janky

ppc

Uh oh!

olljanat commented Jan 10, 2019

Uh oh!

kolyshkin commented Feb 20, 2019

Uh oh!

kolyshkin commented Feb 28, 2019

Uh oh!

kolyshkin commented Feb 28, 2019

Uh oh!

kolyshkin commented Mar 6, 2019

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

avagin commented Dec 29, 2018 •

edited

Loading

codecov bot commented Dec 30, 2018 •

edited

Loading

fntlnz Jan 1, 2019 •

edited

Loading

fntlnz commented Jan 1, 2019 •

edited

Loading