refactor docker directory #5994

ammolitor · 2021-11-16T03:10:10Z

make release images ALL based from centos:7
keep eks images that are based from amazonlinux:2, but create strong alignment with release (centos7) images
use multi-stage Dockerfile(s)
have a single build-images script
remove per image directories (which would only contain scripts that are copied into images)
remove empty requirements.txt file for sidecar.py (the pip command, package and version are in the Dockerfile)

Put description here...

Code-Reviewer Section

The general guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or master if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

ammolitor · 2021-11-16T03:15:06Z

This will fail the linux CI build as the Dockerfile(s) were drastically altered. I'll update the CI builds after this is reviewed and merged.

foundationdb-ci · 2021-11-16T03:15:35Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: 4e3f1ef
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T03:25:45Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 4e3f1ef
Result: FAILED
Error: Error while executing command: docker build --build-arg FDB_WEBSITE=file:///mnt/website --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_ADDITIONAL_VERSIONS="${FDB_VERSION}" -t foundationdb/foundationdb:${TAG} -f release/Dockerfile .. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T03:59:27Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: 48bf6ea
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T04:06:35Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 48bf6ea
Result: FAILED
Error: Error while executing command: docker build --build-arg FDB_WEBSITE=file:///mnt/website --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_ADDITIONAL_VERSIONS="${FDB_VERSION}" -t foundationdb/foundationdb:${TAG} -f release/Dockerfile .. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T05:08:33Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: 230091d
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T05:14:33Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 230091d
Result: FAILED
Error: Error while executing command: cd /root/src/${BUILD_DIR}/packages/docker. Reason: exit status 1
Build Logs (available for 30 days)

ltsampros · 2021-11-16T10:18:48Z

packaging/docker/Dockerfile

+USER fdb
+VOLUME /var/input-files
+VOLUME /var/output-files
+ENV ADDITIONAL_ENV_FILE "/opt/rh/rh-python38/enable"


Why is this needed?

to enable python38 in a centos7 image.

Ack. thanks.

This will break when a user injects a different ADDITIONAL_ENV_FILE into that container. If that is required that should be part of the start script.

good call, just pushed a change for this.

foundationdb-ci · 2021-11-16T18:56:22Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 230091d
Result: FAILED
Error: Error while executing command: cmake -S . -B ${BUILD_OUTPUT} -D USE_CCACHE=ON -D USE_WERROR=ON -D RocksDB_ROOT=/opt/rocksdb-6.22.1 -D RUN_JUNIT_TESTS=ON -D RUN_JAVA_INTEGRATION_TESTS=ON -G Ninja. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T19:02:11Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 230091d
Result: FAILED
Error: Error while executing command: ninja -v -C ${BUILD_OUTPUT} -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T19:22:30Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 230091d
Result: FAILED
Error: Error while executing command: xsltproc /opt/ctest2junit.xsl Testing/**/Test.xml > /root/src/junit.xml. Reason: exit status 1
Build Logs (available for 30 days)

johscheuer

We should change the sidecar python env setup. Otherwise the changes LGTM.

packaging/docker/Dockerfile

johscheuer · 2021-11-16T19:26:19Z

packaging/docker/Dockerfile

+USER fdb
+VOLUME /var/input-files
+VOLUME /var/output-files
+ENV ADDITIONAL_ENV_FILE "/opt/rh/rh-python38/enable"


This will break when a user injects a different ADDITIONAL_ENV_FILE into that container. If that is required that should be part of the start script.

packaging/docker/Dockerfile.eks

packaging/docker/build-images.sh

foundationdb-ci · 2021-11-16T19:41:14Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 963e6f5
Result: FAILED
Error: Error while executing command: popd. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T19:49:02Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: e91549b
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T20:03:13Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 6da98dd
Result: FAILED
Error: Error while executing command: pushd ${BUILD_OUTPUT}/packages/docker. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T20:37:48Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 6da98dd
Result: FAILED
Error: Error while executing command: mvn -pl --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn site.ycsb:foundationdb-binding -am clean package. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T21:08:02Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 6da98dd
Result: FAILED
Error: Error while executing command: tar -czf "${CODEBUILD_SRC_DIR}/packaging/docker/website/downloads/${fdb_version}/linux/fdb_${FDB_VERSION}.tar.gz" --directory "${CODEBUILD_SRC_DIR}/packaging/docker/website/downloads/${fdb_version}/linux/bin" .. Reason: exit status 2
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T21:42:46Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 6da98dd
Result: FAILED
Error: Error while executing command: docker build --label "org.foundationdb.version=${FDB_VERSION}" --label "org.foundationdb.build_date=${BUILD_DATE}" --label "org.foundationdb.commit=${COMMIT_SHA}" --progress plain --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_LIBRARY_VERSIONS="${FDB_VERSION}" --build-arg FDB_WEBSITE="${FDB_WEBSITE}" --tag foundationdb/foundationdb-base:${FDB_VERSION}-${COMMIT_SHA}-debug --file Dockerfile.eks --target foundationdb-base .. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T22:14:58Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 6da98dd
Result: FAILED
Error: Error while executing command: docker build --label "org.foundationdb.version=${FDB_VERSION}" --label "org.foundationdb.build_date=${BUILD_DATE}" --label "org.foundationdb.commit=${COMMIT_SHA}" --progress plain --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_LIBRARY_VERSIONS="${FDB_VERSION}" --build-arg FDB_WEBSITE="${FDB_WEBSITE}" --tag foundationdb/foundationdb-base:${FDB_VERSION}-${COMMIT_SHA}-debug --file Dockerfile.eks --target foundationdb-base .. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-16T23:13:50Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 6da98dd
Result: FAILED
Error: Error while executing command: docker build --label "org.foundationdb.version=${FDB_VERSION}" --label "org.foundationdb.build_date=${BUILD_DATE}" --label "org.foundationdb.commit=${COMMIT_SHA}" --progress plain --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_LIBRARY_VERSIONS="${FDB_VERSION}" --build-arg FDB_WEBSITE="${FDB_WEBSITE}" --tag foundationdb/foundationdb-base:${FDB_VERSION}-${COMMIT_SHA}-debug --file Dockerfile.eks --target foundationdb-base .. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-17T00:23:27Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 6da98dd
Result: FAILED
Error: Error while executing command: cd /root/src/${BUILD_DIR}/packages/docker. Reason: exit status 1
Build Logs (available for 30 days)

johscheuer

LGTM 👍

brownleej

👍

- make release images ALL based from centos:7 - keep eks images that are based from amazonlinux:2, but create strong alignment with release (centos7) images - use multi-stage Dockerfile(s) - have a single build-images script - remove per image directories (which would only contain scripts that are copied into images) - remove empty requirements.txt file for sidecar.py (the pip command, package and version are in the Dockerfile) - don't copy docker into PROJECT_BINARY_DIR

ammolitor · 2021-11-17T23:24:36Z

CodeBuild has been update to build these changes correctly, merging.

foundationdb-ci · 2021-11-17T23:41:35Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: 9e59fda
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-18T00:09:22Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 9e59fda
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

fetch commit_sha from source_code_directory (don't assume we're in the source tree anymore), allow custom tag (if a parameter is passed in as $1) update README.md

copy packaging/docker to PROJECT_BINARY_DIR (undoing part of #5994), fetch commit_sha from source_code_directory (don't assume we're in the source tree anymore), allow custom tag (if a parameter is passed in as $1) update README.md

ammolitor requested review from ltsampros and sears November 16, 2021 03:10

ammolitor assigned brownleej and johscheuer and unassigned brownleej and johscheuer Nov 16, 2021

ammolitor requested review from brownleej, jleach4 and johscheuer November 16, 2021 03:10

ammolitor force-pushed the docker_refactor branch from 4e3f1ef to 48bf6ea Compare November 16, 2021 03:54

ammolitor force-pushed the docker_refactor branch from 48bf6ea to 230091d Compare November 16, 2021 05:03

ltsampros reviewed Nov 16, 2021

View reviewed changes

ammolitor force-pushed the docker_refactor branch from 230091d to 963e6f5 Compare November 16, 2021 19:26

johscheuer requested changes Nov 16, 2021

View reviewed changes

ammolitor force-pushed the docker_refactor branch from 963e6f5 to e91549b Compare November 16, 2021 19:40

ammolitor force-pushed the docker_refactor branch from e91549b to 6da98dd Compare November 16, 2021 19:44

ltsampros self-requested a review November 17, 2021 10:26

ltsampros previously approved these changes Nov 17, 2021

View reviewed changes

johscheuer previously approved these changes Nov 17, 2021

View reviewed changes

brownleej previously approved these changes Nov 17, 2021

View reviewed changes

ammolitor dismissed stale reviews from brownleej, johscheuer, and ltsampros via 9e59fda November 17, 2021 23:24

ammolitor force-pushed the docker_refactor branch from 6da98dd to 9e59fda Compare November 17, 2021 23:24

ammolitor merged commit 218a2ce into release-6.3 Nov 17, 2021

ammolitor deleted the docker_refactor branch November 17, 2021 23:25

ammolitor mentioned this pull request Dec 14, 2021

fix issues with build-images script #6145

Merged

5 tasks

This was referenced Feb 10, 2022

Add overwrite operation to mako sfc-gh-ezhang/foundationdb#1

Closed

Add overwrite operation to mako sfc-gh-sgwydir/foundationdb#1

Closed

refactor docker directory #5994

refactor docker directory #5994

Uh oh!

Conversation

ammolitor commented Nov 16, 2021

Code-Reviewer Section

For Release-Branches

Uh oh!

ammolitor commented Nov 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

ltsampros Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

ammolitor Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

ltsampros Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

johscheuer Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

ammolitor Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

johscheuer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

johscheuer Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 16, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Uh oh!

ammolitor commented Nov 16, 2021 •

edited

Loading