Skip to content

Commit de7500d

Browse files
potiukpotiuk
authored
CI Images are now pre-build and stored in registry (#10368)
* CI Images are now pre-build and stored in registry With this change we utilise the latest pull_request_target event type from Github Actions and we are building the CI image only once (per version) for the entire run. This safes from 2 to 10 minutes per job (!) depending on how much of the Docker image needs to be rebuilt. It works in the way that the image is built only in the build-or-wait step. In case of direct push run or scheduled runs, the build-or-wait step builds and pushes to the GitHub registry the CI image. In case of the pull_request runs, the build-and-wait step waits until separate build-ci-image.yml workflow builds and pushes the image and it will only move forward once the image is ready. This has numerous advantages: 1) Each job that requires CI image is much faster because instead of pulling + rebuilding the image it only pulls the image that was build once. This saves around 2 minutes per job in regular builds but in case of python patch level updates, or adding new requirements it can save up to 10 minutes per job (!) 2) While the images are buing rebuilt we only block one job waiting for all the images. The tests will start running in parallell only when all images are ready, so we are not blocking other runs from running. 3) Whole run uses THE SAME image. Previously we could have some variations because the images were built at different times and potentially releases of dependencies in-between several jobs could make different jobs in the same run use slightly different image. This is not happening any more. 4) Also when we push image to github or dockerhub we push the very same image that was built and tested. Previously it could happen that the image pushed was slightly different than the one that was used for testing (for the same reason) 5) Similar case is with the production images. We are now building and pushing consistently the same images accross the board. 6) Documentation building is split into two parallel jobs docs building and spell checking - decreases elapsed time for the docs build. 7) Last but not least - we keep the history of al the images - those images contain SHA of the commit. This means that we can simply download and run the image locally to reproduce any problem that anyone had in their PR (!). This is super useful to be able to help others to test their problems. * fixup! CI Images are now pre-build and stored in registry * fixup! fixup! CI Images are now pre-build and stored in registry * fixup! fixup! fixup! CI Images are now pre-build and stored in registry * fixup! fixup! fixup! CI Images are now pre-build and stored in registry
1 parent 5739ba2 commit de7500d

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+1422
-833
lines changed

‎.github/workflows/build-images-workflow-run.yml‎

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
#
18+
---
19+
name: "Build Images"
20+
on:
21+
workflow_run:
22+
workflows: ["CI Build"]
23+
types: ['requested']
24+
25+
env:
26+
MOUNT_LOCAL_SOURCES: "false"
27+
MOUNT_FILES: "true"
28+
FORCE_ANSWER_TO_QUESTIONS: "yes"
29+
FORCE_PULL_IMAGES: "true"
30+
CHECK_IMAGE_FOR_REBUILD: "true"
31+
SKIP_CHECK_REMOTE_IMAGE: "true"
32+
DB_RESET: "true"
33+
VERBOSE: "true"
34+
UPGRADE_TO_LATEST_CONSTRAINTS: false
35+
PYTHON_MAJOR_MINOR_VERSION: 3.6
36+
USE_GITHUB_REGISTRY: "true"
37+
GITHUB_REPOSITORY: ${{ github.repository }}
38+
GITHUB_USERNAME: ${{ github.actor }}
39+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
40+
GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
41+
GITHUB_REGISTRY_WAIT_FOR_IMAGE: "false"
42+
43+
jobs:
44+
45+
cancel-workflow-runs:
46+
timeout-minutes: 10
47+
name: "Cancel workflow runs"
48+
runs-on: ubuntu-latest
49+
outputs:
50+
sourceHeadRepo: ${{ steps.cancel.outputs.sourceHeadRepo }}
51+
sourceHeadBranch: ${{ steps.cancel.outputs.sourceHeadBranch }}
52+
sourceHeadSha: ${{ steps.cancel.outputs.sourceHeadSha }}
53+
sourceEvent: ${{ steps.cancel.outputs.sourceEvent }}
54+
cacheDirective: ${{ steps.cache-directive.outputs.docker-cache }}
55+
steps:
56+
- name: "Cancel duplicated 'CI Build' runs"
57+
uses: potiuk/cancel-workflow-runs@master
58+
id: cancel
59+
with:
60+
token: ${{ secrets.GITHUB_TOKEN }}
61+
cancelMode: duplicates
62+
sourceRunId: ${{ github.event.workflow_run.id }}
63+
- name: "Cancel duplicated 'Build Image' runs"
64+
65+
# We find duplicates of our own "Build Image" runs - due to a missing feature
66+
# in GitHub Actions, we have to use Job names to match Event/Repo/Branch from the
67+
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
68+
69+
uses: potiuk/cancel-workflow-runs@master
70+
with:
71+
cancelMode: namedJobs
72+
token: ${{ secrets.GITHUB_TOKEN }}
73+
jobNameRegexps: >
74+
[".*Event: ${{ steps.cancel.outputs.sourceEvent }}
75+
Repo: ${{ steps.cancel.outputs.sourceHeadRepo }}
76+
Branch: ${{ steps.cancel.outputs.sourceHeadBranch }}.*"]
77+
- name: "Cancel all 'CI Build' runs where some jobs failed"
78+
79+
# We find any of the "CI Build" workflow runs, where any of the important jobs
80+
# failed. The important jobs are selected by the regexp array below.
81+
# We also produce list of canceled "CI Build' runs as output, so that we
82+
# can cancel all the matching "Build Images" workflow runs in the two following steps.
83+
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
84+
85+
uses: potiuk/cancel-workflow-runs@master
86+
id: cancel-failed
87+
with:
88+
token: ${{ secrets.GITHUB_TOKEN }}
89+
cancelMode: failedJobs
90+
sourceRunId: ${{ github.event.workflow_run.id }}
91+
jobNameRegexps: >
92+
["^Static checks.*", "^Build docs$", "^Spell check docs$", "^Backport packages$",
93+
"^Checks: Helm tests$", "^Test OpenAPI*"]
94+
- name: "Extract canceled failed runs"
95+
96+
# We use this step to build regexp that will be used to match the Source Run id in
97+
# the build-info job below. If we cancelled some "CI Build" runs in the "cancel-failed' step
98+
# above - we want to cancel also the corresponding "Build Images" runs. Again we have
99+
# to match the jobs using job name rather than use proper API because that feature
100+
# is currently missing in GitHub Actions ¯\_(ツ)_/¯.
101+
102+
id: extract-cancelled-failed-runs
103+
if: steps.cancel-failed.outputs.cancelledRuns != '[]'
104+
run: |
105+
REGEXP="Source Run id: "
106+
SEPARATOR=""
107+
for run_id in $(echo "${{ steps.cancel-failed.outputs.cancelledRuns }}" | jq '.[]')
108+
do
109+
REGEXP="${REGEXP}${SEPARATOR}(${run_id})"
110+
SEPARATOR="|"
111+
done
112+
echo "::set-output name=matching-regexp::[\"${REGEXP}\"]"
113+
- name: "Cancel triggered 'Build Images' runs for the cancelled failed runs"
114+
115+
# In case we do have some cancelled jobs in the "cancel-failed" step above
116+
# We take the extracted regexp array prepared in the previous step and we use
117+
# it to cancel any jobs that have matching names containing Source Run Id:
118+
# followed by one of the run ids. Yes I know it's super complex ¯\_(ツ)_/¯.
119+
120+
if: steps.cancel-failed.outputs.cancelledRuns != '[]'
121+
uses: potiuk/cancel-workflow-runs@master
122+
with:
123+
cancelMode: namedJobs
124+
token: ${{ secrets.GITHUB_TOKEN }}
125+
jobNameRegexps: ${{ steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
126+
- name: "Set Docker Cache Directive"
127+
id: cache-directive
128+
run: |
129+
if [[ ${{ steps.cancel.outputs.sourceEvent }} == "schedule" ]]; then
130+
echo "::set-output name=docker-cache::disabled"
131+
else
132+
echo "::set-output name=docker-cache::pulled"
133+
fi
134+
135+
build-info:
136+
# The name is such long because we are using it to cancel duplicated 'Build Images' runs
137+
# by matching Event/Repo/Branch. This is a workaround for a missing feature of GitHub
138+
# Actions to link the source workflow run and the triggered workflow_run one.
139+
# We are also cancelling SourceRunId in case we determine that we should cancel the source
140+
# Run because of some failing jobs in the source run. Again ¯\_(ツ)_/¯.
141+
name: >
142+
Event: ${{ needs.cancel-workflow-runs.outputs.sourceEvent }}
143+
Repo: ${{ needs.cancel-workflow-runs.outputs.sourceHeadRepo }}
144+
Branch: ${{ needs.cancel-workflow-runs.outputs.sourceHeadBranch }}
145+
Run id: ${{ github.run_id }}
146+
Source Run id: ${{ github.event.workflow_run.id }}
147+
Sha: ${{ github.sha }}
148+
Source Sha: ${{ needs.cancel-workflow-runs.outputs.sourceHeadSha }}
149+
runs-on: ubuntu-latest
150+
needs: [cancel-workflow-runs]
151+
env:
152+
GITHUB_CONTEXT: ${{ toJson(github) }}
153+
steps:
154+
- name: >
155+
Event: ${{ needs.cancel-workflow-runs.outputs.sourceEvent }}
156+
Repo: ${{ needs.cancel-workflow-runs.outputs.sourceHeadRepo }}
157+
Branch: ${{ needs.cancel-workflow-runs.outputs.sourceHeadBranch }}
158+
Run id: ${{ github.run_id }}
159+
Source Run id: ${{ github.event.workflow_run.id }}
160+
Sha: ${{ github.sha }}
161+
Source Sha: ${{ needs.cancel-workflow-runs.outputs.sourceHeadSha }}
162+
run: |
163+
printenv
164+
165+
build-images:
166+
timeout-minutes: 80
167+
name: "Build ${{matrix.image-type}} images ${{matrix.python-version}}"
168+
runs-on: ubuntu-latest
169+
needs: [cancel-workflow-runs]
170+
strategy:
171+
matrix:
172+
python-version: [3.6, 3.7, 3.8]
173+
image-type: [CI, PROD]
174+
fail-fast: true
175+
env:
176+
BACKEND: postgres
177+
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
178+
GITHUB_REGISTRY_PUSH_IMAGE_TAG: ${{ github.event.workflow_run.id }}
179+
UPGRADE_TO_LATEST_CONSTRAINTS: >
180+
${{ needs.cancel-workflow-runs.outputs.sourceEvent == 'push' ||
181+
needs.cancel-workflow-runs.outputs.sourceEvent == 'scheduled' }}
182+
DOCKER_CACHE: ${{ needs.cancel-workflow-runs.outputs.cacheDirective }}
183+
steps:
184+
- name: >
185+
Checkout [${{ needs.cancel-workflow-runs.outputs.sourceEvent }}]
186+
Event: ${{ needs.cancel-workflow-runs.outputs.sourceEvent }}
187+
Repo: ${{ needs.cancel-workflow-runs.outputs.sourceHeadRepo }}
188+
Branch: ${{ needs.cancel-workflow-runs.outputs.sourceHeadBranch }}
189+
Run id: ${{ github.run_id }}
190+
Source Run id: ${{ github.event.workflow_run.id }}
191+
Sha: ${{ github.sha }}
192+
Source Sha: ${{ needs.cancel-workflow-runs.outputs.sourceHeadSha }}
193+
uses: actions/checkout@v2
194+
with:
195+
ref: ${{ needs.cancel-workflow-runs.outputs.sourceHeadSha }}
196+
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} ) to 'main-airflow' to use main scripts"
197+
uses: actions/checkout@v2
198+
with:
199+
path: "main-airflow"
200+
- name: "Setup python"
201+
uses: actions/setup-python@v2
202+
with:
203+
python-version: 3.6
204+
- name: "Override 'scripts' with the ${{ github.ref }} version so that the PR cannot override it."
205+
run: |
206+
rm -rf "scripts"
207+
mv "main-airflow/scripts" .
208+
- name: "Free space"
209+
run: ./scripts/ci/tools/ci_free_space_on_ci.sh
210+
- name: "Build CI images ${{ matrix.python-version }}:${{ github.event.workflow_run.id }}"
211+
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
212+
if: matrix.image-type == 'CI'
213+
- name: "Push CI images ${{ matrix.python-version }}:${{ github.event.workflow_run.id }}"
214+
run: ./scripts/ci/images/ci_push_ci_images.sh
215+
if: matrix.image-type == 'CI'
216+
- name: "Build PROD images ${{ matrix.python-version }}:${{ github.event.workflow_run.id }}"
217+
run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
218+
if: matrix.image-type == 'PROD'
219+
- name: "Push PROD images ${{ matrix.python-version }}:${{ github.event.workflow_run.id }}"
220+
run: ./scripts/ci/images/ci_push_production_images.sh
221+
if: matrix.image-type == 'PROD'
222+
- name: "Canceling the CI Build source workflow in case of failure!"
223+
if: cancelled() || failure()
224+
uses: potiuk/cancel-workflow-runs@master
225+
with:
226+
token: ${{ secrets.GITHUB_TOKEN }}
227+
cancelMode: self
228+
sourceRunId: ${{ github.event.workflow_run.id }}

‎.github/workflows/cancel_other_workflow_runs.yml‎

Lines changed: 0 additions & 36 deletions
This file was deleted.

0 commit comments

Comments
 (0)