Jenkinsfile: avoid errors from find by kolyshkin · Pull Request #39685 · moby/moby

kolyshkin · 2019-08-07T03:02:39Z

There are many errors like this one:

01:39:28.750 find: ‘bundles/test-integration/dbc77018d39a5/root/overlay2/f49953a883daceee60a481dd8e1e37b0f806d309258197d6ba0f6871236d3d47/work/work’: Permission denied

(probably caused by bad permissions)

These directories are not to be looked at when we search for logs, so
let's exclude them. While at it, let the find know we're only looking
for files.

kolyshkin · 2019-08-07T05:53:07Z

OK, I'm not sure why but the old Jenkinsfile is used. How do I test changes to Jenkinsfile? @andrewhsu maybe you know?

thaJeztah · 2019-08-07T16:35:09Z

@kolyshkin I think Jenkins is configured to accept jenkinsfile changes only from certain people/users

@psftw could you check?

andrewhsu · 2019-08-07T17:41:38Z

@kolyshkin the Jenkinsfile changes you made to this PR looks like it took effect on the new jenkins rig ci.docker.com:
https://ci.docker.com/public/job/moby/job/PR-39685/1/execution/node/230/log/

01:48:13.445  Creating janky-bundles.tar.gz
01:48:13.445  + find bundles -not -path */root/overlay2/* -type f -name *.log -o -name *.prof -o -name integration.test
01:48:13.445  + xargs tar -czf janky-bundles.tar.gz
01:48:13.445  find: ‘bundles/test-integration/DockerDaemonSuite.TestDaemonBridgeExternal/da5cc1e78ecfb/root/overlay2/c60bef58b7f8e84cf1edf4d3aa45d9ab5292e5b9bb4d147f256d6f3cb2974801/work/work’: Permission denied
01:48:13.445  find: ‘bundles/test-integration/DockerDaemonSuite.TestDaemonBridgeExternal/da5cc1e78ecfb/root/overlay2/c60bef58b7f8e84cf1edf4d3aa45d9ab5292e5b9bb4d147f256d6f3cb2974801-init/work/work’: Permission denied

For now, the old jenkins.dockerproject.org PR check job janky is still run which does not use Jenkinsfile. The old jenkins rig will be turned off in the future when the new Jenkinsfile is properly running a PR check jobs we care about from the legacy jobs.

tiborvass

your find assumes a certain working directory, which looks incorrect (it's one level deeper) and btw will most certainly fail on my PR (#39682), so you should probably make this more robust. Does ** work for arbitrarily nested ?

kolyshkin · 2019-08-07T18:27:37Z

My find was not doing what it should to begin with. Please see updated commit.

tiborvass · 2019-08-07T18:29:58Z

@kolyshkin no difference https://github.com/moby/moby/compare/a886facc963fe5103023b47886766e8992e42ce9..8a1acfcb4e0203e0fbe962d545c12de71fe2bdcc

kolyshkin · 2019-08-07T18:33:11Z

forgot -a to git commit --amend for the umpteenth time.

accidentally approved

kolyshkin · 2019-08-07T21:34:19Z

it works (almost... I had not anticipated root/*/overlay2 case)

[2019-08-07T20:20:20.475Z] + find bundles -path */root/overlay2 -prune -o -type f ( -name *.log -o -name *.prof -o -name integration.test ) -print

[2019-08-07T20:20:20.475Z] + xargs tar -czf janky-bundles.tar.gz

[2019-08-07T20:20:20.475Z] find: 'bundles/test-integration/DockerDaemonSuite.TestDaemonUserNamespaceRootSetting/d9ea96ae28da6/root/165536.165536/overlay2/91988bda06922abba2a956a5afe1db5713dce3f73fa9c00a601d272e49d16c7c-init/work/work': Permission denied

[2019-08-07T20:20:20.475Z] find: 'bundles/test-integration/DockerDaemonSuite.TestDaemonUserNamespaceRootSetting/d9ea96ae28da6/root/165536.165536/overlay2/9659dbdb8781fb1a2f4208303ee082e5fa7abf89c9151fd378f198e93475fedd/work/work': Permission denied

[2019-08-07T20:20:20.475Z] find: 'bundles/test-integration/DockerDaemonSuite.TestDaemonUserNamespaceRootSetting/d9ea96ae28da6/root/165536.165536/overlay2/91988bda06922abba2a956a5afe1db5713dce3f73fa9c00a601d272e49d16c7c/work/work': Permission denied

[2019-08-07T20:20:20.475Z] find: 'bundles/test-integration/DockerDaemonSuite.TestDaemonUserNamespaceRootSetting/d9ea96ae28da6/root/165536.165536/overlay2/9659dbdb8781fb1a2f4208303ee082e5fa7abf89c9151fd378f198e93475fedd-init/work/work': Permission denied

thaJeztah · 2019-08-07T22:16:36Z

Note that there are tests with subtests that have deeper directory nesting

kolyshkin · 2019-08-07T22:46:09Z

Note that there are tests with subtests that have deeper directory nesting

it does not matter for find -- it just traverses the whole directory tree (no matter how deep) looking for files to match the pattern. What I'm trying to do here is to ignore graphdriver directories (which are either */root/overlay2 or */root/*/overlay2, where * is any number of directories).

thaJeztah · 2019-08-07T23:04:03Z

Ah, okay; that looks good (thought I'd double-check)

kolyshkin · 2019-08-07T23:45:20Z

Ah, okay; that looks good (thought I'd double-check)

This commit is not changing anything wrt how deep find will go, except for it won't go into certain directories (whose full patch matches */root*overlay2).

thaJeztah · 2019-08-08T13:02:12Z

Not sure what went wrong on RS1 https://jenkins.dockerproject.org/job/Docker-PRs-WoW-RS1/26308/console

Triggering again

21:54:08 No such image: sha256:f6ae774a1c32467b422c8fb40728d051d433591035023c00b39123e197873cbf
21:54:08 
21:54:08 
21:54:08 ERROR: Failed 'ERROR: Failed to build image from Dockerfile.windows' at 08/07/2019 21:54:08
21:54:08 At C:\gopath\src\github.com\docker\docker\hack\ci\windows.ps1:448 char:12
21:54:08 +            Throw "ERROR: Failed to build image from Dockerfile.window ...
21:54:08 +            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
21:54:08

thaJeztah

LGTM

thaJeztah · 2019-08-09T00:50:17Z

@kolyshkin still WIP?

thaJeztah · 2019-08-09T22:13:47Z

needs a rebase now; probably should be copied to all stages?

There are many errors like this one: > 01:39:28.750 find: ‘bundles/test-integration/dbc77018d39a5/root/overlay2/f49953a883daceee60a481dd8e1e37b0f806d309258197d6ba0f6871236d3d47/work/work’: Permission denied (probably caused by bad permissions) These directories are not to be looked at when we search for logs, so let's exclude them. It's not super easy to do in find, here is some kind of an explanation for find arguments ``` PATTERN ACTION OR PATTERN ACTION -path X -prune -o -type f [AND] (-name A -o name B) -print ``` (here -o means OR, while AND is implicit) While at it, - let the find know we're only looking for files, not directories - remove a subshell and || true - remove `-name integration.test` (there are no such files) Signed-off-by: Kir Kolyshkin <[email protected]>

kolyshkin · 2019-08-09T22:19:37Z

rebased; should be good to go but let's wait for ci

thaJeztah · 2019-08-09T22:26:06Z

oh, nice one: https://jenkins.dockerproject.org/job/Docker-PRs/55373/console

22:20:38 #44 34.30 Err:1 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 liblzo2-2 amd64 2.08-1.2+b2
22:20:38 #44 34.30   Could not connect to cdn-fastly.deb.debian.org:80 (151.101.52.204). - connect (111: Connection refused) [IP: 151.101.52.204 80]
22:20:38 #44 34.30 Err:2 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 btrfs-progs amd64 4.7.3-1
22:20:38 #44 34.30   Unable to connect to cdn-fastly.deb.debian.org:http: [IP: 151.101.52.204 80]
22:20:38 #44 34.30 Err:3 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 btrfs-tools amd64 4.7.3-1
22:20:38 #44 34.30   Unable to connect to cdn-fastly.deb.debian.org:http: [IP: 151.101.52.204 80]
22:20:38 #44 34.31 E: Failed to fetch http://cdn-fastly.deb.debian.org/debian/pool/main/l/lzo2/liblzo2-2_2.08-1.2+b2_amd64.deb  Could not connect to cdn-fastly.deb.debian.org:80 (151.101.52.204). - connect (111: Connection refused) [IP: 151.101.52.204 80]
22:20:38 #44 34.31 E: Failed to fetch http://cdn-fastly.deb.debian.org/debian/pool/main/b/btrfs-progs/btrfs-progs_4.7.3-1_amd64.deb  Unable to connect to cdn-fastly.deb.debian.org:http: [IP: 151.101.52.204 80]
22:20:38 #44 34.31 E: Failed to fetch http://cdn-fastly.deb.debian.org/debian/pool/main/b/btrfs-progs/btrfs-tools_4.7.3-1_amd64.deb  Unable to connect to cdn-fastly.deb.debian.org:http: [IP: 151.101.52.204 80]
22:20:38 #44 34.31 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
22:20:38 #44 ERROR: executor failed running [/bin/sh -c apt-get update && apt-get install -y --no-install-recommends 	btrfs-tools 	&& rm -rf /var/lib/apt/lists/*]: exit code: 100

kolyshkin · 2019-08-09T22:48:07Z

exp CI restarted

kolyshkin · 2019-08-10T05:19:03Z

CI error in docker-py, not sure why but def unrelated.

[2019-08-09T22:30:40.095Z] =================================== FAILURES ===================================
[2019-08-09T22:30:40.095Z] ___________________ PullImageTest.test_pull_invalid_platform ___________________
[2019-08-09T22:30:40.095Z] tests/integration/api_image_test.py:72: in test_pull_invalid_platform
[2019-08-09T22:30:40.095Z]     assert excinfo.value.status_code == 500
[2019-08-09T22:30:40.095Z] E   AssertionError: assert 400 == 500
[2019-08-09T22:30:40.095Z] E    +  where 400 = APIError(HTTPError('400 Client Error: Bad Request for url: http+docker://localhost/v1.35/images/create?fromImage=hello-world&platform=foobar',),).status_code
[2019-08-09T22:30:40.095Z] E    +    where APIError(HTTPError('400 Client Error: Bad Request for url: http+docker://localhost/v1.35/images/create?fromImage=hello-world&platform=foobar',),) = <ExceptionInfo APIError tblen=4>.value
[2019-08-09T22:30:40.095Z] =========================== short test summary info ============================

kolyshkin · 2019-08-10T05:26:17Z

OK this one is ready, artifacts looks same as before i.e. seems to be no regression
@tiborvass PTAL

thaJeztah · 2019-08-10T10:52:29Z

opened a PR for the failing docker-py test (my bad); #39714

thaJeztah · 2019-08-10T11:00:35Z

Output looks good: no more errors/warnings:

[2019-08-10T00:07:54.519Z] + echo Creating janky-bundles.tar.gz
[2019-08-10T00:07:54.519Z] Creating janky-bundles.tar.gz
[2019-08-10T00:07:54.519Z] + find bundles -path */root/*overlay2 -prune -o -type f ( -name *.log -o -name *.prof ) -print
[2019-08-10T00:07:54.519Z] + xargs tar -czf janky-bundles.tar.gz

[2019-08-10T00:14:27.115Z] + echo Creating powerpc-bundles.tar.gz
[2019-08-10T00:14:27.115Z] Creating powerpc-bundles.tar.gz
[2019-08-10T00:14:27.115Z] + find bundles -path */root/*overlay2 -prune -o -type f ( -name *.log -o -name *.prof ) -print
[2019-08-10T00:14:27.115Z] + xargs tar -czf powerpc-bundles.tar.gz

[2019-08-10T00:17:35.281Z] + echo Creating s390x-bundles.tar.gz
[2019-08-10T00:17:35.281Z] Creating s390x-bundles.tar.gz
[2019-08-10T00:17:35.281Z] + xargs tar -czf s390x-bundles.tar.gz
[2019-08-10T00:17:35.281Z] + find bundles -path */root/*overlay2 -prune -o -type f ( -name *.log -o -name *.prof ) -print

Bundles look good;

Before this patch:

tree -L 6
.
└── test-integration
    ├── DockerDaemonSuite.TestBridgeIPIsExcludedFromAllocatorPool
    │   └── d40217620191d
    │       ├── docker.log
    │       └── root
    │           └── containers
    │               └── 10884243b8f8007d5b6c31fb9eac022f4862b496b5eb6a91c90af32bb0441b0f
    ├── DockerDaemonSuite.TestBuildOnDisabledBridgeNetworkDaemon
    │   └── d730bc40a1700
    │       └── docker.log
...


1157 directories, 432 files

After this patch applied:

tree -L 6
.
└── test-integration
    ├── DockerDaemonSuite.TestBridgeIPIsExcludedFromAllocatorPool
    │   └── d71777cf89af2
    │       ├── docker.log
    │       └── root
    │           └── containers
    │               └── a350ca31a47a27f279afeba23400c0a15d0f585edd7262242eacfe016084a246
    ├── DockerDaemonSuite.TestBuildOnDisabledBridgeNetworkDaemon
    │   └── d64248256e8ef
    │       └── docker.log

...


1157 directories, 432 files

thaJeztah · 2019-08-10T11:00:51Z

I'm merging this one 👍

GordonTheTurtle added the status/0-triage label Aug 7, 2019

thaJeztah added area/testing process/cherry-pick status/2-code-review and removed status/0-triage labels Aug 7, 2019

andrewhsu mentioned this pull request Aug 7, 2019

ppc64le PR checks fail because cannot allocate memory #39692

Closed

tiborvass approved these changes Aug 7, 2019

View reviewed changes

tiborvass requested changes Aug 7, 2019

View reviewed changes

kolyshkin force-pushed the jenkins-find branch from a886fac to 8a1acfc Compare August 7, 2019 18:14

kolyshkin force-pushed the jenkins-find branch from 8a1acfc to 0b340d8 Compare August 7, 2019 18:30

tiborvass previously approved these changes Aug 7, 2019

View reviewed changes

tiborvass approved these changes Aug 7, 2019

View reviewed changes

kolyshkin force-pushed the jenkins-find branch from 0b340d8 to 02ccebe Compare August 7, 2019 21:53

thaJeztah added rebuild/experimental labels Aug 8, 2019

GordonTheTurtle removed rebuild/experimental labels Aug 8, 2019

thaJeztah approved these changes Aug 9, 2019

View reviewed changes

kolyshkin changed the title ~~[WIP] Jenkinsfile: avoid errors from find~~ Jenkinsfile: avoid errors from find Aug 9, 2019

kolyshkin force-pushed the jenkins-find branch from 02ccebe to b283dff Compare August 9, 2019 22:19

thaJeztah added status/4-merge and removed status/2-code-review labels Aug 9, 2019

thaJeztah mentioned this pull request Aug 10, 2019

Jenkinsfile: remove "experimental" stage #39709

Merged

thaJeztah merged commit 901d30b into moby:master Aug 10, 2019

This was referenced Aug 19, 2019

[19.03 backport] Jenkinsfile and related test-changes docker-archive/engine#328

Merged

[18.09 backport] Jenkinsfile and related test-changes docker-archive/engine#329

Merged

thaJeztah added process/cherry-picked and removed process/cherry-pick labels Aug 21, 2019

thaJeztah added this to the 20.03.0 milestone Apr 2, 2020

Conversation

kolyshkin commented Aug 7, 2019

Uh oh!

kolyshkin commented Aug 7, 2019

Uh oh!

thaJeztah commented Aug 7, 2019

Uh oh!

andrewhsu commented Aug 7, 2019

Uh oh!

tiborvass left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolyshkin commented Aug 7, 2019

Uh oh!

tiborvass commented Aug 7, 2019

Uh oh!

kolyshkin commented Aug 7, 2019

Uh oh!

kolyshkin commented Aug 7, 2019

Uh oh!

thaJeztah commented Aug 7, 2019

Uh oh!

kolyshkin commented Aug 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Aug 7, 2019

Uh oh!

kolyshkin commented Aug 7, 2019

Uh oh!

thaJeztah commented Aug 8, 2019

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Aug 9, 2019

Uh oh!

thaJeztah commented Aug 9, 2019

Uh oh!

kolyshkin commented Aug 9, 2019

Uh oh!

thaJeztah commented Aug 9, 2019

Uh oh!

kolyshkin commented Aug 9, 2019

Uh oh!

kolyshkin commented Aug 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kolyshkin commented Aug 10, 2019

Uh oh!

thaJeztah commented Aug 10, 2019

Uh oh!

thaJeztah commented Aug 10, 2019

Uh oh!

thaJeztah commented Aug 10, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tiborvass left a comment •

edited

Loading

kolyshkin commented Aug 7, 2019 •

edited

Loading

kolyshkin commented Aug 10, 2019 •

edited

Loading