ARROW-17953: [Archery] Add archery docker info command by raulcd · Pull Request #14345 · apache/arrow

raulcd · 2022-10-07T16:02:46Z

Some usage examples:

$ archery docker info debian-go-cgo --show command
Service debian-go-cgo docker compose config:
 - command: /bin/bash -c "
  /arrow/ci/scripts/go_build.sh /arrow &&
  /arrow/ci/scripts/go_test.sh /arrow"

or

$ archery docker info debian-go-cgo
Service debian-go-cgo docker compose config:
 - image: ${REPO}:${ARCH}-debian-${DEBIAN}-go-${GO}-cgo
 - build
  - context: .
  - dockerfile: ci/docker/debian-go-cgo.dockerfile
  - cache_from: ['${REPO}:${ARCH}-debian-${DEBIAN}-go-${GO}-cgo']
  - args
   - base: ${REPO}:${ARCH}-debian-${DEBIAN}-go-${GO}
 - shm_size: 2G
 - volumes: ['.:/arrow:delegated', '${DOCKER_VOLUME_PREFIX}debian-ccache:/ccache:delegated']
 - environment
  - ARROW_GO_TESTCGO: 1
 - command: /bin/bash -c "
  /arrow/ci/scripts/go_build.sh /arrow &&
  /arrow/ci/scripts/go_test.sh /arrow"

or

$ archery docker info ubuntu-cuda-cpp --show environment
Service ubuntu-cuda-cpp docker compose config:
 - environment
  - ARROW_BUILD_STATIC: OFF
  - ARROW_CUDA: ON
  - ARROW_GANDIVA: OFF
  - ARROW_GCS: OFF
  - ARROW_ORC: OFF
  - ARROW_S3: OFF
  - ARROW_SUBSTRAIT: OFF
  - ARROW_WITH_OPENTELEMETRY: OFF
  - CCACHE_COMPILERCHECK: content
  - CCACHE_COMPRESS: 1
  - CCACHE_COMPRESSLEVEL: 6
  - CCACHE_MAXSIZE: 1G
  - CCACHE_DIR: /ccache

edited only the commands to reflect the change of the command --show

github-actions · 2022-10-07T16:03:26Z

https://issues.apache.org/jira/browse/ARROW-17953

assignUser

Cool idea! Looks good overall, As a minor thing I am not sure if I like the addition of - for structuring the output or if it just adds noise and only indentation would be cleaner?

dev/archery/archery/docker/cli.py

dev/archery/archery/docker/core.py

dev/archery/archery/docker/cli.py

Co-authored-by: Antoine Pitrou <[email protected]>

raulcd · 2022-10-13T16:24:23Z

@pitrou @assignUser this should be ready to review again. Thanks for the feedback!

kou

+1

assignUser

Tested it locally, looks good 👍

pitrou · 2022-10-17T15:30:46Z

I've noticed something:

$ archery docker info conda-cpp
Service conda-cpp docker-compose config:
  image: ${REPO}:${ARCH}-conda-cpp
  build
    context: .
    dockerfile: ci/docker/conda-cpp.dockerfile
    cache_from: ['${REPO}:${ARCH}-conda-cpp']
    args
      repo: ${REPO}
      arch: ${ARCH}
  shm_size: 2G
  ulimits
    core: ${ULIMIT_CORE}
  environment
    ARROW_BUILD_BENCHMARKS: ON
    ARROW_BUILD_EXAMPLES: ON
    ARROW_ENABLE_TIMING_TESTS: None
    ARROW_EXTRA_ERROR_CONTEXT: ON
    ARROW_MIMALLOC: ON
    ARROW_USE_LD_GOLD: ON
    ARROW_USE_PRECOMPILED_HEADERS: ON
    BUILD_DOCS_PYTHON: ON
    CCACHE_COMPILERCHECK: content
    CCACHE_COMPRESS: 1
    CCACHE_COMPRESSLEVEL: 6
    CCACHE_MAXSIZE: 1G
    CCACHE_DIR: /ccache
    AWS_ACCESS_KEY_ID: None
    AWS_SECRET_ACCESS_KEY: None
    SCCACHE_BUCKET: None
    SCCACHE_S3_KEY_PREFIX: ${SCCACHE_S3_KEY_PREFIX:-sccache}
  volumes: ['.:/arrow:delegated', '${DOCKER_VOLUME_PREFIX}conda-ccache:/ccache:delegated']
  command: [' /arrow/ci/scripts/cpp_build.sh /arrow /build && /arrow/ci/scripts/cpp_test.sh /arrow /build']

Printing ARROW_ENABLE_TIMING_TESTS: None is a bit unfortunate. What happens actually is that the value is inherited from the calling shell's environment. So perhaps print out <inherited> or something.

…r info command is None as it would be inherited from shell environment values

assignUser · 2022-10-18T14:30:28Z

+1

Service conda-cpp docker-compose config:
  image: ${REPO}:${ARCH}-conda-cpp
  build
    context: .
    dockerfile: ci/docker/conda-cpp.dockerfile
    cache_from: ['${REPO}:${ARCH}-conda-cpp']
    args
      repo: ${REPO}
      arch: ${ARCH}
  shm_size: 2G
  ulimits
    core: ${ULIMIT_CORE}
  environment
    ARROW_BUILD_BENCHMARKS: ON
    ARROW_BUILD_EXAMPLES: ON
    ARROW_ENABLE_TIMING_TESTS: <inherited>
    ARROW_EXTRA_ERROR_CONTEXT: ON
    ARROW_MIMALLOC: ON
    ARROW_USE_LD_GOLD: ON
    ARROW_USE_PRECOMPILED_HEADERS: ON
    BUILD_DOCS_PYTHON: ON
    CCACHE_COMPILERCHECK: content
    CCACHE_COMPRESS: 1
    CCACHE_COMPRESSLEVEL: 6
    CCACHE_MAXSIZE: 1G
    CCACHE_DIR: /ccache
    AWS_ACCESS_KEY_ID: <inherited>
    AWS_SECRET_ACCESS_KEY: <inherited>
    SCCACHE_BUCKET: <inherited>
    SCCACHE_S3_KEY_PREFIX: ${SCCACHE_S3_KEY_PREFIX:-sccache}
  volumes: ['.:/arrow:delegated', '${DOCKER_VOLUME_PREFIX}conda-ccache:/ccache:delegated']
  command: [' /arrow/ci/scripts/cpp_build.sh /arrow /build && /arrow/ci/scripts/cpp_test.sh /arrow /build']

pitrou

Thanks @raulcd !

raulcd added 3 commits October 7, 2022 12:44

ARROW-17953: [Archery] Add archery docker info command

e543e22

Add possibility of filtering parts of the docker-compose info

509dc5d

Move functionality to DockerCompose and add tests

e503561

assignUser approved these changes Oct 10, 2022

View reviewed changes

pitrou reviewed Oct 10, 2022

View reviewed changes

dev/archery/archery/docker/cli.py Outdated Show resolved Hide resolved

pitrou reviewed Oct 10, 2022

View reviewed changes

dev/archery/archery/docker/cli.py Outdated Show resolved Hide resolved

pitrou reviewed Oct 10, 2022

View reviewed changes

dev/archery/archery/docker/core.py Outdated Show resolved Hide resolved

pitrou reviewed Oct 10, 2022

View reviewed changes

dev/archery/archery/docker/cli.py Outdated Show resolved Hide resolved

raulcd and others added 3 commits October 13, 2022 13:04

Update dev/archery/archery/docker/cli.py

b8b4485

Co-authored-by: Antoine Pitrou <[email protected]>

Update dev/archery/archery/docker/cli.py

8fb2c33

Co-authored-by: Antoine Pitrou <[email protected]>

Apply code review suggestions to improve UX for command and output

9dde65d

kou approved these changes Oct 17, 2022

View reviewed changes

assignUser approved these changes Oct 17, 2022

View reviewed changes

ARROW-17953: [Archery] Use <inherited> keyword if the value for docke…

28e9408

…r info command is None as it would be inherited from shell environment values

pitrou approved these changes Oct 18, 2022

View reviewed changes

pitrou merged commit c33bb50 into apache:master Oct 18, 2022

asfimport mentioned this pull request Oct 18, 2022

[Archery] Add a docker info command #33161

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-17953: [Archery] Add archery docker info command#14345

ARROW-17953: [Archery] Add archery docker info command#14345
pitrou merged 7 commits intoapache:masterfrom
raulcd:ARROW-17953

raulcd commented Oct 7, 2022 •

edited

Loading

Uh oh!

github-actions bot commented Oct 7, 2022

Uh oh!

assignUser left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulcd commented Oct 13, 2022

Uh oh!

kou left a comment

Uh oh!

assignUser left a comment

Uh oh!

pitrou commented Oct 17, 2022

Uh oh!

assignUser commented Oct 18, 2022

Uh oh!

pitrou left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

raulcd commented Oct 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 7, 2022

Uh oh!

assignUser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulcd commented Oct 13, 2022

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

assignUser left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou commented Oct 17, 2022

Uh oh!

assignUser commented Oct 18, 2022

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

raulcd commented Oct 7, 2022 •

edited

Loading