Various updates to CI #331

BradyBonnette · 2023-06-09T18:57:52Z

The changes included here, at a 10,000 foot view:

An overhaul of how tests are ran on Amazon Linux 2 on aarch64 platforms. An adjacent git repo (not shown here) also includes a lot of work to make this happen. Basically, we relied on a "pool" of CI runners to sit and wait idle until a new job came through. Now, CI runners are launched ad-hoc, for as many runners that are needed with or without a matrix. This also allows for the re-run of individual failed tasks if required, whereas before this simply would not work
Re-introduces sccache back into the runs. Previously, PL/Rust and some required dependencies flat out not work with sccache. However, those issues have been addressed
Introduces a nightly run setup, which for now does two things: 1.) Tests to make sure that the Dockerfile.try setup works as per the documentation, and 2.) Tests that the Debian packages can be both built and installed. These will be ran nightly against the develop branch, and we can certainly add more to it in the future

NOTE: The nightly CI runs will not happen until these changes eventually land in main on next release.

Update from upstream

Upstream develop

BradyBonnette · 2023-06-09T18:58:44Z

.github/scripts/load_cache.sh

-  if aws s3api head-object --profile $ARTIFACT_USER_AWS_PROFILE --bucket $AWS_CACHE_BUCKET --key $cache_key &> /dev/null; then
+  if aws s3api head-object --bucket $AWS_CACHE_BUCKET --key $cache_key &> /dev/null; then


Now the instances have direct access to the buckets they require.

BradyBonnette · 2023-06-09T18:59:34Z

.github/workflows/ci.yml

+  # If something is missing, it can be added here or baked into a new image outside of Github Actions.
+  plrust_arm64_amzn2:
+    name: aarch64 tests (amzn2)
+    runs-on: [self-hosted, linux, ARM64, launch_template_id__lt-0013395587950cb5d]


launch_template_id__lt-xxxxxxxxxx is important for the new CI setup. The private ops repo has more explanation on what this means.

BradyBonnette · 2023-06-09T19:00:06Z

.github/workflows/ci.yml

        cd $WORK_DIR
        shopt -s globstar
-        checksum=$(cat **/Cargo.lock .github/workflows/ci.yml | sha256sum | awk '{print $1}')
+        checksum=$(cat **/Cargo.lock **/rust-toolchain.toml .github/workflows/ci.yml | sha256sum | awk '{print $1}')


Adds any rust-toolchain.toml file as a cache key calculation

BradyBonnette · 2023-06-09T19:00:40Z

.github/workflows/ci.yml

+        sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' && \
+        wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/apt.postgresql.org.gpg >/dev/null && \


The old way of installing keys was deprecated by Ubuntu. This is the new way.

BradyBonnette · 2023-06-09T19:01:38Z

.github/workflows/nightly.yml

+# CODE REVIEW: Ensure this event setup is used and not on push/PR below!
+# CODE REVIEW: Ensure this event setup is used and not on push/PR below!
+# CODE REVIEW: Ensure this event setup is used and not on push/PR below!
+# on:
+#   schedule:
+#     - cron: '0 7 * * *'
+#   workflow_dispatch:


I promise Ill change this before merging in. I am leaving it like this until it's been reviewed because any subsequent CI run that's needed in the future (pre-merging) should also re-run the nightly things to be sure.

BradyBonnette · 2023-06-09T19:02:16Z

.github/workflows/release_artifacts.yml

-    name: Spin up aarch64 runner instances
+  arm64_deb_artifacts:
+    name: Build aarch64 Debian artifacts
+    runs-on: [self-hosted, linux, ARM64, launch_template_id__lt-0bad2911d6aad1b0d]


See above regarding launch_template_id__lt-xxxxxxxxxxxx

BradyBonnette · 2023-06-09T19:02:49Z

.github/workflows/release_artifacts.yml

-    # The following permissions are required to instruct GHA that it is allowed
-    # to communicate to another service via OIDC
    permissions:
-      id-token: write # Required for requesting OIDC JWTs
-      contents: read  # This is required for actions/checkout
-
-    runs-on: ubuntu-latest
-
-    # In order to run this against aarch64 instances in AWS, GHA needs to authenticate
-    # against AWS via OIDC so it can directly communicate and launch AWS instances as
-    # needed. All the infrastructure and machinery is set up outside of Github Actions.
-    # Using OIDC is an alternative to setting up hard-coded AWS CLI credentials somewhere
-    # on a Github Action runner.
-    steps:
-    - name: Configure AWS Credentials
-      uses: aws-actions/configure-aws-credentials@v2
-      with:
-        aws-region: us-east-2
-        role-to-assume: arn:aws:iam::950481341027:role/github_oidc_iam_role
-        role-session-name: GithubPlrustSession
-
-    # Launch as many instances as needed, dictated by strategy.matrix.pg_version above.
-    # LaunchTemplateId is something known ahead of time, and is created elsewhere outside
-    # of Github Actions.
-    - name: Launch runner instance
-      run: aws ec2 run-instances --launch-template LaunchTemplateId=lt-0bad2911d6aad1b0d
-
-  build_deb_artifacts:
-    name: Build aarch64 Debian artifacts
-    needs: spin_up_aarch64_instances # Dont run this until 'spin_up_aarch64_instances' has completed
-    runs-on: [self-hosted, linux, ARM64, plrust_artifacts]
-
-    strategy:
-      matrix:
-        pg_version: [pg13, pg14, pg15] # See MATRIX NOTES above
+      contents: write


OIDC is no longer needed as per the new CI launcher setup.

BradyBonnette · 2023-06-09T19:03:54Z

Cargo.toml

 opt-level = 3
 lto = "fat"
 codegen-units = 1
+strip = "debuginfo"


BradyBonnette · 2023-06-09T19:04:27Z

plrustc/plrustc/Cargo.toml


 [dev-dependencies]
-compiletest_rs = "0.10"
+compiletest_rs = "0.10.2"


An update to compiletest_rs was needed to make it compatible with sccache.

eeeebbbbrrrr

I'm going to accept this on faith. I don't have enough XP to validate every line of these .yml and .sh files.

Just make sure to hit the scheduling section you already noted in your own review.

And thanks! Excited to get this merged -- it'll be great!

BradyBonnette · 2023-06-09T19:22:59Z

@eeeebbbbrrrr

That's understandable, and also why I put a bunch of discussion comments above as to maybe take away some of the mystery. Also, we can use the actual CI runs/checks here as a guide as to whether or not this works properly.

BradyBonnette added 30 commits May 18, 2023 12:16

Try out new way of running aarch64 instances for release

540654a

Trying out new launch template workflow for CI

3936095

simulate failure

5197f92

Try out full aarch64 fun

94be45e

Fixes S3 connectivity issues

68c51ef

Trying out sccache again

cba74fa

Ugh extra double quote

aaaa00b

Trying out sccache on x86_64 builds

4c2eb44

fix quotes

b0027bd

Use only pg13, also various sccache printouts

11562b2

Trying it out without sccache

f49e2ef

adding sccache back as a test

eae763c

Trying out HOST env var setup

f95edcc

Trying another forceful tactic

a7522e3

Trying out thomcc patch

30541e4

Towards a small overhaul

8ff6c73

correct ubuntu version

8778f94

fix postgresql key

208aa0f

fix llvm ver

380dd08

check out sccache directory

8f18faa

add some times

1752c2d

Inspect sccache dir

d1de8f4

Try out new official version of compiletest_rs

0d4da0a

Try out newest version of sccache

cf6c4da

Try out aarch64 amzn tests again

a318aca

Output some stats

99b2a9d

Try out new setup

1660f48

Trying again

836a82d

Properly use PG_VER

35a1c3b

Use correct path

d40acb4

BradyBonnette and others added 15 commits June 7, 2023 14:28

Maybe new version of docker is messing with things?

163868f

Maybe a sleep was needed?

3e48185

try with new version of docker

1f6c513

try them all

69520da

Try out everything

34e23fd

Merge branch 'main' of github.com:tcdi/plrust into update-from-upstream

8d5bbaa

Merge pull request #1 from BradyBonnette/update-from-upstream

3ec05dc

Update from upstream

Dockerfile.try installs toolchain from rust-toolchain.toml file

8be7751

some cleanup and notes

5dd6a20

Merge branch 'develop' of github.com:tcdi/plrust into upstream-develop

6268a82

bump version of pgrx in plrust/build script

dc41b4b

Merge branch 'develop' of github.com:tcdi/plrust into upstream-develop

aad3297

Merge branch 'develop' of github.com:tcdi/plrust into upstream-develop

318df6d

Merge pull request #2 from BradyBonnette/upstream-develop

0ab53b8

Upstream develop

adds pg13 test runs on aarch64 AL2

f697115

BradyBonnette commented Jun 9, 2023

View reviewed changes

Cargo.toml

opt-level = 3

lto = "fat"

codegen-units = 1

strip = "debuginfo"

Copy link

Contributor Author

BradyBonnette Jun 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See: #305

BradyBonnette commented Jun 9, 2023

View reviewed changes

BradyBonnette requested a review from eeeebbbbrrrr June 9, 2023 19:06

eeeebbbbrrrr approved these changes Jun 9, 2023

View reviewed changes

set up nightly on schedule instead of on push/PR

cd4da95

BradyBonnette merged commit 6481666 into tcdi:develop Jun 9, 2023

eeeebbbbrrrr mentioned this pull request Jun 14, 2023

prepare-v1.2.0 #336

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Various updates to CI #331

Various updates to CI #331

Uh oh!

BradyBonnette commented Jun 9, 2023 •

edited

Loading

Uh oh!

BradyBonnette Jun 9, 2023

Uh oh!

BradyBonnette Jun 9, 2023

Uh oh!

BradyBonnette Jun 9, 2023

Uh oh!

BradyBonnette Jun 9, 2023

Uh oh!

BradyBonnette Jun 9, 2023

Uh oh!

BradyBonnette Jun 9, 2023

Uh oh!

BradyBonnette Jun 9, 2023

Uh oh!

BradyBonnette Jun 9, 2023

Uh oh!

BradyBonnette Jun 9, 2023

Uh oh!

eeeebbbbrrrr left a comment

Uh oh!

BradyBonnette commented Jun 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		if aws s3api head-object --profile $ARTIFACT_USER_AWS_PROFILE --bucket $AWS_CACHE_BUCKET --key $cache_key &> /dev/null; then
		if aws s3api head-object --bucket $AWS_CACHE_BUCKET --key $cache_key &> /dev/null; then

		sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' && \
		wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc \| gpg --dearmor \| sudo tee /etc/apt/trusted.gpg.d/apt.postgresql.org.gpg >/dev/null && \

Various updates to CI #331

Various updates to CI #331

Uh oh!

Conversation

BradyBonnette commented Jun 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eeeebbbbrrrr left a comment

Choose a reason for hiding this comment

Uh oh!

BradyBonnette commented Jun 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BradyBonnette commented Jun 9, 2023 •

edited

Loading