Remove git and python packages from the docker images used by Zip-Nuget-Java-Nodejs Packaging Pipeline#11651
Merged
Remove git and python packages from the docker images used by Zip-Nuget-Java-Nodejs Packaging Pipeline#11651
Conversation
snnn
changed the title
Contributor
Author
|
Need more work. |
snnn
force-pushed
the
snnn/update_git
branch
3 times, most recently
from
b075cb2 to
01aae17
Compare
snnn
changed the title
snnn
commented
Jun 4, 2022
| @@ -0,0 +1,71 @@ | |||
| #!/bin/bash | |||
Contributor
Author
There was a problem hiding this comment.
This file is identical with the one in aarch64 folder.
Contributor
There was a problem hiding this comment.
do they need to be kept consistent? is there a way to avoid duplication?
Contributor
Author
There was a problem hiding this comment.
Use a script to generate them.
| Context: tools/ci_build/github/linux/docker | ||
| DockerBuildArgs: "--network=host --build-arg POLICY=manylinux2014 --build-arg PLATFORM=x86_64 --build-arg BASEIMAGE=nvidia/cuda:11.4.0-cudnn8-devel-centos7 --build-arg DEVTOOLSET_ROOTPATH=/opt/rh/devtoolset-10/root --build-arg PREPEND_PATH=/opt/rh/devtoolset-10/root/usr/bin: --build-arg LD_LIBRARY_PATH_ARG=/opt/rh/devtoolset-10/root/usr/lib64:/opt/rh/devtoolset-10/root/usr/lib:/opt/rh/devtoolset-10/root/usr/lib64/dyninst:/opt/rh/devtoolset-10/root/usr/lib/dyninst:/usr/local/lib64 --build-arg BUILD_UID=$( id -u )" | ||
| Repository: onnxruntimecuda11build | ||
| Dockerfile: tools/ci_build/github/linux/docker/inference/x64/default/cpu/Dockerfile |
Contributor
Author
There was a problem hiding this comment.
This change is to remove manylinux related things from this C API pipeline, because we don't need these python stuffs here.
edgchen1
reviewed
Jun 4, 2022
tools/ci_build/build.py
Outdated
| else: | ||
| my_env["LD_LIBRARY_PATH"] = dll_path | ||
| if cuda_home: | ||
| my_env["PATH"] += os.pathsep + os.path.join(cuda_home, "bin") + os.pathsep + my_env["PATH"] |
Contributor
There was a problem hiding this comment.
so the final value is <original PATH>:<cuda_home>/bin:<original PATH>?
| --volume /data/models:/build/models:ro --volume $HOME/.onnx:/home/onnxruntimedev/.onnx -e NIGHTLY_BUILD onnxruntimecuda11build \ | ||
| /opt/python/cp37-cp37m/bin/python3 /onnxruntime_src/tools/ci_build/build.py --build_java --build_dir /build --config Release \ | ||
| --skip_submodule_sync --parallel --build_shared_lib --use_cuda --cuda_version=$(CUDA_VERSION) --cuda_home=/usr/local/cuda-$(CUDA_VERSION) --cudnn_home=/usr/local/cuda-$(CUDA_VERSION) --cmake_extra_defines CMAKE_CUDA_HOST_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/cc 'CMAKE_CUDA_ARCHITECTURES=37;50;52;60;61;70;75;80' | ||
| docker run --gpus all -e CFLAGS="-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection -fcf-protection -O3 -Wl,--strip-all" -e CXXFLAGS="-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection -fcf-protection -O3 -Wl,--strip-all" -e NVIDIA_VISIBLE_DEVICES=all --rm --volume /data/onnx:/data/onnx:ro --volume $(Build.SourcesDirectory):/onnxruntime_src --volume $(Build.BinariesDirectory):/build \ |
Contributor
There was a problem hiding this comment.
nit: can we make the lines shorter for readability?
edgchen1
reviewed
Jun 4, 2022
| my_env["LD_LIBRARY_PATH"] = dll_path | ||
| # Add nvcc's folder to PATH env so that our cmake file can find nvcc | ||
| if cuda_home: | ||
| my_env["PATH"] = os.path.join(cuda_home, "bin") + os.pathsep + my_env["PATH"] |
Contributor
There was a problem hiding this comment.
should we also check if "PATH" in my_env as is done for dll_path?
Contributor
Author
There was a problem hiding this comment.
PATH is always in env vars.
edgchen1
approved these changes
Jun 4, 2022
Craigacp
pushed a commit
to Craigacp/onnxruntime
that referenced
this pull request
Aug 8, 2022
* adding conditional variable again * Adding split test cases in python * Adding python cases for split * Enable s8s8 split * Optimize input * Revert "Remove git and python packages from the docker images used by Zip-Nuget-Java-Nodejs Packaging Pipeline (microsoft#11651)" This reverts commit d5e34ac * Revert "Revert "Remove git and python packages from the docker images used by Zip-Nuget-Java-Nodejs Packaging Pipeline (microsoft#11651)"" This reverts commit 3c1a330. * format file * Update c-api-linux-cpu.yml * Update c-api-linux-cpu.yml * Update c-api-linux-cpu.yml * Reformat file * Reformat file * format file * Optimize input * Remove unused import * Remove useless init * Format split.py with black
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Remove git from the docker images used by "Zip-Nuget-Java-Nodejs Packaging Pipeline". For unknown reason, the git224 package is gone. While I can upgrade it to git236, I think a better choice is to just remove it.
Remove python packages also. To reduce the number of OSS components we need to maintain for compliance purpose.
The other changes in this PR are just code refactoring.
Motivation and Context
To resolve the build error in https://aiinfra.visualstudio.com/Lotus/_build/results?buildId=209965&view=logs&j=714209ae-2e3b-5c94-7779-6bb56a181c3b&t=4dc3fda7-6978-52ea-4a7e-758a98a77dd1