Skip to content

[docker][android] use aws instead of google repository #42393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
IvanKobzarev wants to merge 5 commits into from
Closed

[docker][android] use aws instead of google repository #42393

IvanKobzarev wants to merge 5 commits into from

Conversation

@IvanKobzarev
Copy link
Contributor

@IvanKobzarev IvanKobzarev commented Jul 31, 2020 •
edited
Loading

Stack from ghstack:

@IvanKobzarev IvanKobzarev mentioned this pull request Jul 31, 2020
Closed
@dr-ci
Copy link

dr-ci bot commented Jul 31, 2020 •
edited
Loading

💊 CI failures summary and remediations

As of commit 458b857 (more details on the Dr. CI page):

  • 4/4 failures possibly* introduced in this PR
    • 1/4 non-CircleCI failure(s)

🕵️ 3 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_macos_10_13_py3_test (1/3)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

Aug 03 13:04:02 clang: error: linker command failed with exit code 1 (use -v to see invocation) 
Aug 03 13:04:02      
Aug 03 13:04:02     creating build/temp.macosx-10.9-x86_64-3.8/numba/cext 
Aug 03 13:04:02     compile options: '-I/Users/distiller/workspace/miniconda3/lib/python3.8/site-packages/numpy/core/include -I/Users/distiller/workspace/miniconda3/include/python3.8 -c' 
Aug 03 13:04:02     extra options: '-g' 
Aug 03 13:04:02     clang: numba/_helpermod.cclang: numba/cext/utils.c 
Aug 03 13:04:02      
Aug 03 13:04:02     clang: numba/cext/dictobject.c 
Aug 03 13:04:02     clang: numba/cext/listobject.c 
arch x86_64 -L/Users/distiller/workspace/miniconda3/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.9-x86_64-3.8/numba/_helpermod.o build/temp.macosx-10.9-x86_64-3.8/numba/cext/utils.o build/temp.macosx-10.9-x86_64-3.8/numba/cext/dictobject.o build/temp.macosx-10.9-x86_64-3.8/numba/cext/listobject.o -L/Users/distiller/workspace/miniconda3/lib/python3.8/site-packages/numpy/core/lib -lnpymath -o build/lib.macosx-10.9-x86_64-3.8/numba/_helperlib.cpython-38-darwin.so -headerpad_max_install_names 
Aug 03 13:04:02     ld: in /Users/distiller/workspace/miniconda3/lib/python3.8/site-packages/numpy/core/lib/libnpymath.a(npy_math.o), could not parse object file /Users/distiller/workspace/miniconda3/lib/python3.8/site-packages/numpy/core/lib/libnpymath.a(npy_math.o): 'Unknown attribute kind (61) (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_902.0.39.2_0')', using libLTO version 'LLVM version 9.1.0, (clang-902.0.39.2)' for architecture x86_64 
Aug 03 13:04:02     clang: error: linker command failed with exit code 1 (use -v to see invocation) 
ler/workspace/miniconda3/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.9-x86_64-3.8/numba/_helpermod.o build/temp.macosx-10.9-x86_64-3.8/numba/cext/utils.o build/temp.macosx-10.9-x86_64-3.8/numba/cext/dictobject.o build/temp.macosx-10.9-x86_64-3.8/numba/cext/listobject.o -L/Users/distiller/workspace/miniconda3/lib/python3.8/site-packages/numpy/core/lib -lnpymath -o build/lib.macosx-10.9-x86_64-3.8/numba/_helperlib.cpython-38-darwin.so -headerpad_max_install_names" failed with exit status 1 
Aug 03 13:04:02     ---------------------------------------- 
d3j9r52rv80000gn/T/pip-install-8poblff0/numba/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/gk/lkr9pm5x039fx6d3j9r52rv80000gn/T/pip-record-ggskkxkx/install-record.txt --single-version-externally-managed --compile --install-headers /Users/distiller/workspace/miniconda3/include/python3.8/numba Check the logs for full command output. 
Aug 03 13:04:03 + cleanup 
Aug 03 13:04:03 + retcode=1 
Aug 03 13:04:03 + set +x

See CircleCI build pytorch_xla_linux_bionic_py3_6_clang9_build (2/3)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result. 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/bazel_definitions.py 
Auto-merging .circleci/cimodel/data/simple/bazel_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/android_definitions.py 
Auto-merging .circleci/cimodel/data/simple/android_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_definitions.py 
Auto-merging .circleci/cimodel/data/pytorch_build_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_data.py 
Auto-merging .circleci/cimodel/data/pytorch_build_data.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/binary_build_data.py 
Auto-merging .circleci/cimodel/data/binary_build_data.py 
Automatic merge failed; fix conflicts and then commit the result.

See CircleCI build pytorch_linux_xenial_py3_6_gcc5_4_build (3/3)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result. 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/bazel_definitions.py 
Auto-merging .circleci/cimodel/data/simple/bazel_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/android_definitions.py 
Auto-merging .circleci/cimodel/data/simple/android_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_definitions.py 
Auto-merging .circleci/cimodel/data/pytorch_build_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_data.py 
Auto-merging .circleci/cimodel/data/pytorch_build_data.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/binary_build_data.py 
Auto-merging .circleci/cimodel/data/binary_build_data.py 
Automatic merge failed; fix conflicts and then commit the result.

ci.pytorch.org: 1 failed

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 24 times.

IvanKobzarev added a commit that referenced this pull request Aug 1, 2020
@IvanKobzarev
[docker][android] use aws instead of google repository
d20ba6f 
ghstack-source-id: 918e326
Pull Request resolved: #42393
seemethere
seemethere requested changes Aug 3, 2020
View reviewed changes
.circleci/docker/common/install_android.sh Outdated
curl --silent --show-error --location --fail --retry 3 --output /tmp/$_sdk_version https://dl.google.com/android/repository/$_sdk_version
sudo unzip -q /tmp/$_sdk_version -d $_android_home
rm /tmp/$_sdk_version
curl --silent --show-error --location --fail --retry 3 --output /tmp/android-sdk-linux.zip $_https_amazon_aws/android-sdk-linux.zip
Copy link
Member

@seemethere seemethere Aug 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be

Suggested change
curl --silent --show-error --location --fail --retry 3 --output /tmp/android-sdk-linux.zip $_https_amazon_aws/android-sdk-linux.zip
curl --silent --show-error --location --fail --retry 3 --output /tmp/android-sdk-linux.zip $_https_amazon_aws/sdk-tools-linux-3859397.zip

Copy link
Contributor Author

@IvanKobzarev IvanKobzarev Aug 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I did a change how we install it,

it was hard to just with unpacking archives to replicate all sdk state that should be equivalent to:

unzip sdk-tools-linux-3859397.zip
sdkmanager \
  "tools" \
  "platform-tools" \
  "emulator"

sdkmanager \
  "build-tools;28.0.3" \
  "build-tools;29.0.2"

sdkmanager \
  "platforms;android-28" \
  "platforms;android-29"

I zipped that state from my devserver to android-sdk-linux.zip
If we want to support naming how it was packed, probably it should be something like
android-sdk-linux-tools3859397-build-tools2803-2902-platforms28-29.zip

IvanKobzarev added a commit that referenced this pull request Aug 3, 2020
@IvanKobzarev
[docker][android] use aws instead of google repository
52ddc00 
ghstack-source-id: 079d031
Pull Request resolved: #42393
seemethere pushed a commit that referenced this pull request Aug 3, 2020
@IvanKobzarev @seemethere
[docker][android] use aws instead of google repository
bcc449f 
ghstack-source-id: 079d031
Pull Request resolved: #42393
@IvanKobzarev
Copy link
Contributor Author

IvanKobzarev commented Aug 3, 2020

Closing as change was merged inside original PR: #40950

@IvanKobzarev IvanKobzarev mentioned this pull request Aug 3, 2020
Closed
@facebook-github-bot facebook-github-bot deleted the gh/IvanKobzarev/69/head branch September 3, 2020 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@seemethere seemethere seemethere approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@IvanKobzarev @seemethere