Skip to content

Commit fdcb08f

Browse files
andreydevyatkinandreydevyatkin
authored
Migrate "beam_PostCommit_Python_ValidatesRunner_*" Jenkins jobs to GitHub Actions (#28439)
* added beam_PostCommit_Python_ValidatesRunner jobs to GitHub Actions * added beam_PostCommit_Python_ValidatesRunner jobs to GitHub Actions * added beam_PostCommit_Python_ValidatesRunner jobs to GitHub Actions
1 parent 1e60aee commit fdcb08f

7 files changed

Lines changed: 427 additions & 4 deletions

‎.github/workflows/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,10 @@ Please note that jobs with matrix need to have matrix element in the comment. Ex
207207
| [ PostCommit Python Examples Direct ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Direct.yml) | N/A |`Run Python Examples_Direct`| [![PostCommit Python Examples Direct](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Direct.yml) |
208208
| [ PostCommit Python Examples Flink ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Flink.yml) | N/A |`Run Python Examples_Flink`| [![PostCommit Python Examples Flink](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Flink.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Flink.yml) |
209209
| [ PostCommit Python Examples Spark ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Spark.yml) | N/A |`Run Python Examples_Spark`| [![PostCommit Python Examples Spark](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Spark.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Spark.yml) |
210+
| [ PostCommit Python ValidatesRunner Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml) | ['3.8','3.11'] |`Run Python Dataflow ValidatesRunner (matrix_element)`| [![PostCommit Python ValidatesRunner Dataflow](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml) |
211+
| [ PostCommit Python ValidatesRunner Flink ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml) | ['3.8','3.11'] |`Run Python Flink ValidatesRunner (matrix_element)`| [![PostCommit Python ValidatesRunner Flink](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml) |
212+
| [ PostCommit Python ValidatesRunner Samza ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Samza.yml) | ['3.8','3.11'] |`Run Python Samza ValidatesRunner (matrix_element)`| [![PostCommit Python ValidatesRunner Samza](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Samza.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Samza.yml) |
213+
| [ PostCommit Python ValidatesRunner Spark ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml) | ['3.8','3.9','3.11'] |`Run Python Spark ValidatesRunner (matrix_element)`| [![PostCommit Python ValidatesRunner Spark](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml) |
210214
| [ PostCommit Website Publish](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Website_Publish.yml) | N/A | N/A | [![.github/workflows/beam_PostCommit_Website_Publish](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Website_Publish.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Website_Publish.yml) |
211215
| [ PostCommit XVR GoUsingJava Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_GoUsingJava_Dataflow.yml) | N/A |`Run XVR_GoUsingJava_Dataflow PostCommit`| [![.github/workflows/beam_PostCommit_XVR_GoUsingJava_Dataflow](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_GoUsingJava_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_GoUsingJava_Dataflow.yml) |
212216
| [ PostCommit Sickbay Python ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Sickbay_Python.yml) | ['3.8','3.9','3.10','3.11'] |`Run Python PostCommit Sickbay tests (matrix_element)Upda`| [![PostCommit Sickbay Python ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Sickbay_Python.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Sickbay_Python.yml) |

‎.github/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
name: PostCommit Python ValidatesRunner Dataflow
17+
18+
on:
19+
issue_comment:
20+
types: [created]
21+
schedule:
22+
- cron: '0 */6 * * *'
23+
workflow_dispatch:
24+
25+
#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
26+
permissions:
27+
actions: write
28+
pull-requests: read
29+
checks: read
30+
contents: read
31+
deployments: read
32+
id-token: none
33+
issues: read
34+
discussions: read
35+
packages: read
36+
pages: read
37+
repository-projects: read
38+
security-events: read
39+
statuses: read
40+
41+
# This allows a subsequently queued workflow run to interrupt previous runs
42+
concurrency:
43+
group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.comment.body || github.event.sender.login}}'
44+
cancel-in-progress: true
45+
46+
env:
47+
GRADLE_ENTERPRISE_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}
48+
GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
49+
GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
50+
51+
jobs:
52+
beam_PostCommit_Python_ValidatesRunner_Dataflow:
53+
if: |
54+
github.event_name == 'schedule' ||
55+
github.event_name == 'workflow_dispatch' ||
56+
startsWith(github.event.comment.body, 'Run Python Dataflow ValidatesRunner')
57+
runs-on: [self-hosted, ubuntu-20.04, main]
58+
timeout-minutes: 200
59+
name: ${{ matrix.job_name }} (${{ matrix.job_phrase }} ${{ matrix.python_version }})
60+
strategy:
61+
fail-fast: false
62+
matrix:
63+
job_name: ["beam_PostCommit_Python_ValidatesRunner_Dataflow"]
64+
job_phrase: ["Run Python Dataflow ValidatesRunner"]
65+
python_version: ['3.8', '3.11']
66+
steps:
67+
- uses: actions/checkout@v3
68+
- name: Setup repository
69+
uses: ./.github/actions/setup-action
70+
with:
71+
comment_phrase: ${{ matrix.job_phrase }} ${{ matrix.python_version }}
72+
github_token: ${{ secrets.GITHUB_TOKEN }}
73+
github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }} ${{ matrix.python_version }})
74+
- name: Install Python
75+
uses: actions/setup-python@v4
76+
with:
77+
python-version: ${{ matrix.python_version }}
78+
- name: Install Java
79+
uses: actions/setup-java@v3
80+
with:
81+
distribution: 'zulu'
82+
java-version: '8'
83+
- name: Setup Gradle
84+
uses: gradle/gradle-build-action@v2
85+
with:
86+
cache-read-only: false
87+
- name: Set PY_VER_CLEAN
88+
id: set_py_ver_clean
89+
run: |
90+
PY_VER=${{ matrix.python_version }}
91+
PY_VER_CLEAN=${PY_VER//.}
92+
echo "py_ver_clean=$PY_VER_CLEAN" >> $GITHUB_OUTPUT
93+
- name: Run validatesRunnerBatchTests script
94+
uses: ./.github/actions/gradle-command-self-hosted-action
95+
with:
96+
gradle-command: :sdks:python:test-suites:dataflow:py${{steps.set_py_ver_clean.outputs.py_ver_clean}}:validatesRunnerBatchTests
97+
arguments: |
98+
-PuseWheelDistribution \
99+
-PpythonVersion=${{ matrix.python_version }} \
100+
- name: Run validatesRunnerStreamingTests script
101+
uses: ./.github/actions/gradle-command-self-hosted-action
102+
with:
103+
gradle-command: :sdks:python:test-suites:dataflow:py${{steps.set_py_ver_clean.outputs.py_ver_clean}}:validatesRunnerStreamingTests
104+
arguments: |
105+
-PuseWheelDistribution \
106+
-PpythonVersion=${{ matrix.python_version }} \
107+
- name: Archive code coverage results
108+
uses: actions/upload-artifact@v3
109+
with:
110+
name: python-code-coverage-report
111+
path: "**/pytest*.xml"

‎.github/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
name: PostCommit Python ValidatesRunner Flink
17+
18+
on:
19+
issue_comment:
20+
types: [created]
21+
schedule:
22+
- cron: '0 */6 * * *'
23+
workflow_dispatch:
24+
25+
#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
26+
permissions:
27+
actions: write
28+
pull-requests: read
29+
checks: read
30+
contents: read
31+
deployments: read
32+
id-token: none
33+
issues: read
34+
discussions: read
35+
packages: read
36+
pages: read
37+
repository-projects: read
38+
security-events: read
39+
statuses: read
40+
41+
# This allows a subsequently queued workflow run to interrupt previous runs
42+
concurrency:
43+
group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.comment.body || github.event.sender.login}}'
44+
cancel-in-progress: true
45+
46+
env:
47+
GRADLE_ENTERPRISE_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}
48+
GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
49+
GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
50+
51+
jobs:
52+
beam_PostCommit_Python_ValidatesRunner_Flink:
53+
if: |
54+
github.event_name == 'schedule' ||
55+
github.event_name == 'workflow_dispatch' ||
56+
startsWith(github.event.comment.body, 'Run Python Flink ValidatesRunner')
57+
runs-on: [self-hosted, ubuntu-20.04, main]
58+
timeout-minutes: 100
59+
name: ${{ matrix.job_name }} (${{ matrix.job_phrase }} ${{ matrix.python_version }})
60+
strategy:
61+
fail-fast: false
62+
matrix:
63+
job_name: ["beam_PostCommit_Python_ValidatesRunner_Flink"]
64+
job_phrase: ["Run Python Flink ValidatesRunner"]
65+
python_version: ['3.8', '3.11']
66+
steps:
67+
- uses: actions/checkout@v3
68+
- name: Setup repository
69+
uses: ./.github/actions/setup-action
70+
with:
71+
comment_phrase: ${{ matrix.job_phrase }} ${{ matrix.python_version }}
72+
github_token: ${{ secrets.GITHUB_TOKEN }}
73+
github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }} ${{ matrix.python_version }})
74+
- name: Install Python
75+
uses: actions/setup-python@v4
76+
with:
77+
python-version: ${{ matrix.python_version }}
78+
- name: Install Java
79+
uses: actions/setup-java@v3
80+
with:
81+
distribution: 'zulu'
82+
java-version: '8'
83+
- name: Setup Gradle
84+
uses: gradle/gradle-build-action@v2
85+
with:
86+
cache-read-only: false
87+
- name: Set PY_VER_CLEAN
88+
id: set_py_ver_clean
89+
run: |
90+
PY_VER=${{ matrix.python_version }}
91+
PY_VER_CLEAN=${PY_VER//.}
92+
echo "py_ver_clean=$PY_VER_CLEAN" >> $GITHUB_OUTPUT
93+
- name: Run flinkValidatesRunner script
94+
env:
95+
CLOUDSDK_CONFIG: ${{ env.KUBELET_GCLOUD_CONFIG_PATH}}
96+
uses: ./.github/actions/gradle-command-self-hosted-action
97+
with:
98+
gradle-command: :sdks:python:test-suites:portable:py${{steps.set_py_ver_clean.outputs.py_ver_clean}}:flinkValidatesRunner
99+
arguments: |
100+
-PpythonVersion=${{ matrix.python_version }} \
101+
- name: Archive code coverage results
102+
uses: actions/upload-artifact@v3
103+
with:
104+
name: python-code-coverage-report
105+
path: "**/pytest*.xml"

‎.github/workflows/beam_PostCommit_Python_ValidatesRunner_Samza.yml‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
name: PostCommit Python ValidatesRunner Samza
17+
18+
on:
19+
issue_comment:
20+
types: [created]
21+
schedule:
22+
- cron: '0 */6 * * *'
23+
workflow_dispatch:
24+
25+
#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
26+
permissions:
27+
actions: write
28+
pull-requests: read
29+
checks: read
30+
contents: read
31+
deployments: read
32+
id-token: none
33+
issues: read
34+
discussions: read
35+
packages: read
36+
pages: read
37+
repository-projects: read
38+
security-events: read
39+
statuses: read
40+
41+
# This allows a subsequently queued workflow run to interrupt previous runs
42+
concurrency:
43+
group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.comment.body || github.event.sender.login}}'
44+
cancel-in-progress: true
45+
46+
env:
47+
GRADLE_ENTERPRISE_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}
48+
GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
49+
GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
50+
51+
jobs:
52+
beam_PostCommit_Python_ValidatesRunner_Samza:
53+
if: |
54+
github.event_name == 'schedule' ||
55+
github.event_name == 'workflow_dispatch' ||
56+
startsWith(github.event.comment.body, 'Run Python Samza ValidatesRunner')
57+
runs-on: [self-hosted, ubuntu-20.04, main]
58+
timeout-minutes: 100
59+
name: ${{ matrix.job_name }} (${{ matrix.job_phrase }} ${{ matrix.python_version }})
60+
strategy:
61+
fail-fast: false
62+
matrix:
63+
job_name: ["beam_PostCommit_Python_ValidatesRunner_Samza"]
64+
job_phrase: ["Run Python Samza ValidatesRunner"]
65+
python_version: ['3.8', '3.11']
66+
steps:
67+
- uses: actions/checkout@v3
68+
- name: Setup repository
69+
uses: ./.github/actions/setup-action
70+
with:
71+
comment_phrase: ${{ matrix.job_phrase }} ${{ matrix.python_version }}
72+
github_token: ${{ secrets.GITHUB_TOKEN }}
73+
github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }} ${{ matrix.python_version }})
74+
- name: Install Python
75+
uses: actions/setup-python@v4
76+
with:
77+
python-version: ${{ matrix.python_version }}
78+
- name: Install Java
79+
uses: actions/setup-java@v3
80+
with:
81+
distribution: 'zulu'
82+
java-version: '8'
83+
- name: Setup Gradle
84+
uses: gradle/gradle-build-action@v2
85+
with:
86+
cache-read-only: false
87+
- name: Set PY_VER_CLEAN
88+
id: set_py_ver_clean
89+
run: |
90+
PY_VER=${{ matrix.python_version }}
91+
PY_VER_CLEAN=${PY_VER//.}
92+
echo "py_ver_clean=$PY_VER_CLEAN" >> $GITHUB_OUTPUT
93+
- name: Run samzaValidatesRunner script
94+
uses: ./.github/actions/gradle-command-self-hosted-action
95+
with:
96+
gradle-command: :sdks:python:test-suites:portable:py${{steps.set_py_ver_clean.outputs.py_ver_clean}}:samzaValidatesRunner
97+
arguments: |
98+
-PpythonVersion=${{ matrix.python_version }} \
99+
- name: Archive code coverage results
100+
uses: actions/upload-artifact@v3
101+
with:
102+
name: python-code-coverage-report
103+
path: "**/pytest*.xml"

0 commit comments

Comments
 (0)