Skip to content

ARROW-12958: [CI][Developer] Build + host the docs for PR branches - #13913

Merged
rok merged 14 commits into
apache:masterfrom
assignUser:docs-preview
Aug 19, 2022
Merged

rok merged 14 commits into
apache:masterfrom
assignUser:docs-preview

Conversation

@assignUser

Copy link
Copy Markdown
Member

No description provided.

@github-actions

Copy link
Copy Markdown

@assignUser

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -p pr_number=13913 preview-docs

@github-actions

Copy link
Copy Markdown
Failed to render template `docs/github.linux.yml` with UndefinedError: 'pr_number' is undefined
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/2882330760

@assignUser

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -p pr_number=13913 preview-docs

@github-actions

Copy link
Copy Markdown

Revision: 8d77f242f2716f6de93773ff0fac9c2b62f3deb0

Submitted crossbow builds: ursacomputing/crossbow @ actions-76bab72639

Task Status
preview-docs Github Actions

@assignUser

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -p pr_number=13913 preview-docs

@github-actions

Copy link
Copy Markdown

Revision: 03a8233

Submitted crossbow builds: ursacomputing/crossbow @ actions-cff97e19a8

Task Status
preview-docs Github Actions

@assignUser
assignUser marked this pull request as ready for review August 19, 2022 12:12
@assignUser

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -p pr_number=13913 preview-docs

@github-actions

Copy link
Copy Markdown

Revision: 4d0d4ee

Submitted crossbow builds: ursacomputing/crossbow @ actions-6e06515170

Task Status
preview-docs Github Actions

Comment thread dev/tasks/tasks.yml
ci: github
template: r/github.packages.yml
params:
custom_version: Unset

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a minor fix for an issue I noticed in this PR, namely that for params to work they have to be listed in tasks.yml. This also necessitates the replace( thing in the templates themselves as even empty params in task.yml are None and default does not trigger on None.

Comment thread dev/tasks/tasks.yml
pr_number: Unset
artifacts: "build/docs.tar.gz"
flags: "-v $PWD/build/:/build/"
image: ubuntu-docs No newline at end of file

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
image: ubuntu-docs
image: ubuntu-docs

Comment thread java/pom.xml
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
<version>2.2.2</version>
<version>4.0.5</version>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This plugin was causing problems with shallow copies, it should have been able to handle it but it looks like our version was too old. The new version introduces the cli switch I added to java_build.sh. This is now the newest possible version we can use as 5.0.0 does not support java < 11.

@assignUser

Copy link
Copy Markdown
Member Author

We will of course also need to add a "clean up"-job on crossbow that removes previews older than n-days (We could use the api to check if the pr is merged/closed but we have an issue with the cb gh-api limit already so I'd rather not spend it on this).

When using archery crossbow submit you have to include the pr number as a parameter -p pr_number=xxx. For submissions via the bot this will be passed through automatically (but the change will only take affect once this is merged, so I had to use -p to test).

@wjones127 @jorisvandenbossche review please?

@assignUser

Copy link
Copy Markdown
Member Author

🥳 https://crossbow.voltrondata.com/pr/13913/

@rok rok left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great @assignUser! I can't wait to use it.
Perhaps a follow up task (outside of this PR) could be also generating Go's docs.

@rok

rok commented Aug 19, 2022

Copy link
Copy Markdown
Member

We should probably document this in https://arrow.apache.org/docs/developers/documentation.html ?

A nit about the URL: perhaps /docs_preview/# or /pr_docs/# would be more idiomatic than /pr/#?

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome! I'm impressed you got this done this quickly.

The only thing I think needs improving in some reminder in the CI where the output of the docs is living.

Also, I guess clean up will be a follow up for later?

git config --local user.email "[email protected]"
git add docs/pr/{{ pr_number }}
git commit -m "Add docs preview for PR {{ pr_number }}"
git push

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we maybe echo the URL where the docs are hosted here? Just thinking it would be nice to have a reminder where they appear in the CI output.

@assignUser

assignUser commented Aug 19, 2022 •

Copy link
Copy Markdown
Member Author

@wjones127 How about something like this (open to text suggestions :D): https://github.com/assignUser/test-repo-a/actions/runs/2890950254
image

@assignUser

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -p pr_number=13913 preview-docs

@github-actions

Copy link
Copy Markdown

Revision: 5aaa225

Submitted crossbow builds: ursacomputing/crossbow @ actions-0d5323889b

Task Status
preview-docs Github Actions

@assignUser

Copy link
Copy Markdown
Member Author

@rok
rok merged commit 28cf3f9 into apache:master Aug 19, 2022
@ursabot

ursabot commented Aug 20, 2022

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 4af3df0 and contender = 28cf3f9. 28cf3f9 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.1% ⬆️0.0%] test-mac-arm
[Failed ⬇️2.19% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.28% ⬆️0.14%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 28cf3f9f ec2-t3-xlarge-us-east-2
[Failed] 28cf3f9f test-mac-arm
[Failed] 28cf3f9f ursa-i9-9960x
[Finished] 28cf3f9f ursa-thinkcentre-m75q
[Finished] 4af3df0c ec2-t3-xlarge-us-east-2
[Failed] 4af3df0c test-mac-arm
[Failed] 4af3df0c ursa-i9-9960x
[Finished] 4af3df0c ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

zagto pushed a commit to zagto/arrow that referenced this pull request Oct 7, 2022
pribor pushed a commit to GlobalWebIndex/arrow that referenced this pull request Oct 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants