Skip to content

ARROW-17435: [CI][Python][CUDA] Install Numba for CUDA interop tests - #13899

Merged
pitrou merged 2 commits into
apache:masterfrom
pitrou:ARROW-17435-numba-ci
Aug 17, 2022
Merged

pitrou merged 2 commits into
apache:masterfrom
pitrou:ARROW-17435-numba-ci

Conversation

@pitrou

@pitrou pitrou commented Aug 16, 2022 •

Copy link
Copy Markdown
Member

Checked locally on a Ubuntu 20.04 host with:

archery docker run ubuntu-cuda-python

@pitrou

pitrou commented Aug 16, 2022

Copy link
Copy Markdown
Member Author

This is based on PR #13896.

cc @assignUser @kszucs

Comment thread ci/docker/linux-apt-python-3.dockerfile Outdated

@pitrou pitrou Aug 16, 2022 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this is ideal, but It wasn't able to create a child Dockerfile with this file as FROM (I may be misunderstanding how these images are managed).

@kszucs

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A good example on child images/services is the ubuntu-r one, which builds on the ubuntu-cpp dockerfile in it's own linux-apt-r.dockerfile
https://github.com/apache/arrow/blob/master/docker-compose.yml#L1234-L1242

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... can you suggest a patch? I already tried to follow this pattern and didn't succeed. I got this error:

DEBUG:archery:Executing `['docker-compose', '--file', '/home/antoine/arrow/dev/docker-compose.yml', 'build', '--build-arg', 'BUILDKIT_INLINE_CACHE=1', 'ubuntu-cuda-python']`
Building ubuntu-cuda-python
[+] Building 0.4s (3/3) FINISHED                                                                                                                                     
 => [internal] load build definition from linux-apt-python-numba.dockerfile                                                                                     0.0s
 => => transferring dockerfile: 1.04kB                                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                               0.0s
 => => transferring context: 35B                                                                                                                                0.0s
 => ERROR [internal] load metadata for docker.io/apache/arrow-dev:amd64-linux-apt-python-3                                                                      0.4s
------
 > [internal] load metadata for docker.io/apache/arrow-dev:amd64-linux-apt-python-3:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: docker.io/apache/arrow-dev:amd64-linux-apt-python-3: not found
ERROR: Service 'ubuntu-cuda-python' failed to build : Build failed
Error: `docker-compose --file /home/antoine/arrow/dev/docker-compose.yml build --build-arg BUILDKIT_INLINE_CACHE=1 ubuntu-cuda-python` exited with a non-zero exit code 1, see the process log above.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(side note: it's not help ful that browsing to docker.io/apache/arrow-dev does not let me view the available images/tags)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it doesn't take much time to install numba then it's fine have it here.

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@pitrou
pitrou force-pushed the ARROW-17435-numba-ci branch from 870f049 to 2652163 Compare August 17, 2022 09:07
@pitrou
pitrou marked this pull request as ready for review August 17, 2022 09:07
@pitrou
pitrou requested a review from kszucs August 17, 2022 09:07
@kszucs

kszucs commented Aug 17, 2022

Copy link
Copy Markdown
Member

Can you post a link from the logs showing that the numba tests are running?

@pitrou

pitrou commented Aug 17, 2022

Copy link
Copy Markdown
Member Author

Can you post a link from the logs showing that the numba tests are running?

Here is the output of archery --debug docker run ubuntu-cuda-python here:
https://gist.github.com/pitrou/53ae68d6720fe790ca648de05974e7e5

@kszucs

kszucs commented Aug 17, 2022

Copy link
Copy Markdown
Member

Can you post a link from the logs showing that the numba tests are running?

Here is the output of archery --debug docker run ubuntu-cuda-python here: https://gist.github.com/pitrou/53ae68d6720fe790ca648de05974e7e5

Thanks! Feel free to merge it on green.

@pitrou
pitrou merged commit b0422e5 into apache:master Aug 17, 2022
@pitrou
pitrou deleted the ARROW-17435-numba-ci branch August 17, 2022 13:38
@ursabot

ursabot commented Aug 17, 2022

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 42ed37e and contender = b0422e5. b0422e5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️1.36% ⬆️1.57%] test-mac-arm
[Failed ⬇️1.1% ⬆️0.27%] ursa-i9-9960x
[Finished ⬇️2.7% ⬆️1.39%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] b0422e5f ec2-t3-xlarge-us-east-2
[Failed] b0422e5f test-mac-arm
[Failed] b0422e5f ursa-i9-9960x
[Finished] b0422e5f ursa-thinkcentre-m75q
[Finished] 42ed37e3 ec2-t3-xlarge-us-east-2
[Finished] 42ed37e3 test-mac-arm
[Failed] 42ed37e3 ursa-i9-9960x
[Finished] 42ed37e3 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@ursabot

ursabot commented Aug 17, 2022

Copy link
Copy Markdown

['Python', 'R'] benchmarks have high level of regressions.
test-mac-arm
ursa-i9-9960x

zagto pushed a commit to zagto/arrow that referenced this pull request Oct 7, 2022
…pache#13899)

Checked locally on a Ubuntu 20.04 host with:
```
archery docker run ubuntu-cuda-python
```

Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants