Speed up image building in CI by exporting and importing mount cache#45314
Merged
potiuk merged 1 commit intoapache:mainfrom Jan 1, 2025
Merged
Conversation
potiuk
requested review from
eladkal,
ephraimbuddy,
gopidesupavan,
hussein-awala and
kaxil
Member
Author
|
And here is caching of the "cache-mount" as well cc: @gopidesupavan |
Member
Author
|
First tests show that we can go down to < 2 minutes from ~6 minutes to build the image |
potiuk
force-pushed
the
export-import-mount-cache-to-speed-up-builds
branch
from
7b1ebeb to
d6c758a
Compare
potiuk
force-pushed
the
export-import-mount-cache-to-speed-up-builds
branch
from
d6c758a to
96ca5fa
Compare
potiuk
force-pushed
the
export-import-mount-cache-to-speed-up-builds
branch
from
96ca5fa to
1417d17
Compare
gopidesupavan
approved these changes
Dec 31, 2024
Member
gopidesupavan
left a comment
gopidesupavan
left a comment
There was a problem hiding this comment.
LGTM , useful with breeze commands
Member
Nice improvement :) |
potiuk
force-pushed
the
export-import-mount-cache-to-speed-up-builds
branch
13 times, most recently
from
7504d5f to
f4cd2ba
Compare
potiuk
force-pushed
the
export-import-mount-cache-to-speed-up-builds
branch
from
f4cd2ba to
e7987e1
Compare
During the build, cache of ``uv`` and ``pip`` is stored in a separate "cache mount" volum that is mounted during the build. This cache mount volume is preserved between builds and can be exported and imported to speed up the build process in CI - where cache is stored as artifact and can be imported in the next build. This PR implements it: * export and import commands are added to breeze to export/import cache mount content * the cache mount content is stashed as artifact in the build after image is built and it is restored before the image is built
potiuk
force-pushed
the
export-import-mount-cache-to-speed-up-builds
branch
from
e7987e1 to
7d411b1
Compare
Member
Author
|
Finally: 3m30 s. to build the image instead of 5m47. Not as fast as the fastest "registry" build we had (1m20s or so ) but "fast enough" |
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Jan 2, 2025
The apache#45314 introduced the bug where image artifacts were not uploaded alongside cache mount artifacts. It caused all PRs to be tested with "main" version of code.
potiuk
added a commit
that referenced
this pull request
Jan 2, 2025
The #45314 introduced the bug where image artifacts were not uploaded alongside cache mount artifacts. It caused all PRs to be tested with "main" version of code.
HariGS-DB
pushed a commit
to HariGS-DB/airflow
that referenced
this pull request
Jan 16, 2025
…pache#45314) During the build, cache of ``uv`` and ``pip`` is stored in a separate "cache mount" volum that is mounted during the build. This cache mount volume is preserved between builds and can be exported and imported to speed up the build process in CI - where cache is stored as artifact and can be imported in the next build. This PR implements it: * export and import commands are added to breeze to export/import cache mount content * the cache mount content is stashed as artifact in the build after image is built and it is restored before the image is built
HariGS-DB
pushed a commit
to HariGS-DB/airflow
that referenced
this pull request
Jan 16, 2025
The apache#45314 introduced the bug where image artifacts were not uploaded alongside cache mount artifacts. It caused all PRs to be tested with "main" version of code.
got686-yandex
pushed a commit
to got686-yandex/airflow
that referenced
this pull request
Jan 30, 2025
…pache#45314) During the build, cache of ``uv`` and ``pip`` is stored in a separate "cache mount" volum that is mounted during the build. This cache mount volume is preserved between builds and can be exported and imported to speed up the build process in CI - where cache is stored as artifact and can be imported in the next build. This PR implements it: * export and import commands are added to breeze to export/import cache mount content * the cache mount content is stashed as artifact in the build after image is built and it is restored before the image is built
got686-yandex
pushed a commit
to got686-yandex/airflow
that referenced
this pull request
Jan 30, 2025
The apache#45314 introduced the bug where image artifacts were not uploaded alongside cache mount artifacts. It caused all PRs to be tested with "main" version of code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During the build, cache of
uvandpipis stored in a separate "cache mount" volum that is mounted during the build. This cache mount volume is preserved between builds and can be exported and imported to speed up the build process in CI - where cache is stored as artifact and can be imported in the next build.This PR implements it:
export and import commands are added to breeze to export/import cache mount content
the cache mount content is stashed as artifact in the build after image is built and it is restored before the image is built
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.