Adds Podman Container Runtime by schnie · Pull Request #1750 · astronomer/astro-cli

schnie · 2024-11-26T17:49:48Z

Description

This PR adds changes specific to podman container runtime and podman machine management. These changes will allow users to create astro local project without needing to manually manage podman machines and also removes some prerequisites required to setup astro cli therefore reducing friction during installation of CLI.

Notable code changes:

Separated out the container "Runtime" from the container "Engine". This allows us to unit test the logic of the runtime, while mocking out the output of the underlying docker and podman commands. We don't actually run the commands during the unit tests. We should test the concrete implementations with some e2e tests in the future though. "Runtimes" contain the high-level logic around managing the specific container runtimes. "Engines" are responsible for actually executing commands and returning the output.
New and existing interface mocks in the runtimes package have been replaced with mockery-generated mocks for unit tests.

🎟 Issue(s)

Related to https://github.com/astronomer/astro/issues/24344

🧪 Functional Testing

Tested Podman specific changes by spinning up local projects on Mac and Windows:

Mac:

astro dev init -t=etl
Initializing Astro project
Pulling Airflow development files from Astro Runtime 12.5.0
Initialized empty Astro project in /Users/pritt/Astronomer/plg/astro-cli/test

astro dev start                                     
⠙ Astro uses container technology to run your Airflow project. Please wait while we get things started…

[build --pull -t etl_698e5c/airflow:latest -f Dockerfile .]
STEP 1/1: FROM quay.io/astronomer/astro-runtime:12.5.0
Trying to pull quay.io/astronomer/astro-runtime:12.5.0...
Getting image source signatures
Copying blob sha256:48eae20780410ddf42e9eb7c7e9a2c0e0a53507b929716514bff24d3f8838e2e
Copying blob sha256:75401debba30b9b283a9036c6488069d4dcc3fdb4bd71d84fdd8cdfbe8f01dd2
Copying blob sha256:850900bac3cbca3b3c7086a7815ed91ea5d82b6c9ef9be2cd830c49f2a7b0333
Copying blob sha256:8332e5d9a406fe441dfd17139d4e8f83dd05860577530424c0701de3e5914787
Copying blob sha256:38543f9d811a7dae3fd758e0323bd403c35fcaa400495502684cddfab4a32502
Copying blob sha256:f066e6e2b5ee556c5a460ff5698cd85140b4098d2a2bfe3f5b29183d2c5e6e80
Copying blob sha256:2b9acf93ba85c4fa29c2daef7d57eba520df8e6c1771cdefe4e22fffe0406565
Copying blob sha256:a68a9747f3b0f9d06ce17904eb7f6506c3c4e6ee155958c7ad1297a42c6da519
Copying blob sha256:1df980c6e89be5e4c0b582113f11750f63943ca648145414526129e62839ca1d
Copying blob sha256:ab27309f25c2418727fc073d10d1dfad142263aa87069dec0639931ad9a3d395
Copying blob sha256:f82b5718b95c8e931a76a2e62731aa3387f75b0d7d4ead7fac2d330ef5333c32
Copying blob sha256:efcb71ea9f6dd860e169fe945b6444f452430c989360a7cb106e98f1f29107f1
Copying blob sha256:a19adae7a57cdd32f4bf9890f7f82ebbb533144977071e1d3716e5b0f4690b2a
Copying blob sha256:c0d0e2de59623e332d0cc37a0bed3761e2dcb2b8910a1642dedd3b37e100c181
Copying blob sha256:ba70da4c13274f8e0f2e56ee7c24ede97380cca179a89e01a76dfa6c2a58a5e5
Copying blob sha256:0b92b2a9b0ecd08e7f90ee9a1786432612b4dee2eb8c6dd67d0968407d778a70
Copying blob sha256:c0d0e2de59623e332d0cc37a0bed3761e2dcb2b8910a1642dedd3b37e100c181
Copying blob sha256:b2d112fab93ede576f32507e883231a9243b05ff203349e94542d9c85cbee3cf
Copying blob sha256:dd6cf619f12d40d8b27123256fd864917471f977663ac634e89c370a29d21863
Copying blob sha256:c012c3d3ffb561a2c683a5828e471855c67584ac92ccda0a5a33019c93289466
Copying blob sha256:afb02ac0943f56a28ac9c91d6129bec9f872f52055c57fb1e0ada7d38ca574eb
Copying blob sha256:17818f9b2bca6da7809cef5e5e0df82627095ff3131960855f6afd16d8f5181a
Copying blob sha256:81d94ecf766d4989c0f639800c744d2c6841273496a16cfebe0eb18343266117
Copying blob sha256:40b62ff8e2d36c7033c72f4121a0b9d987f2250c4568f6b5fc1fdb58f252fd3b
Copying blob sha256:10d622bca2d6e616f3a71801938f9adf8b77bd5e6f25a745dd735753ec4c8a80
Copying blob sha256:d1fe7dfff25d762dd6f4df13c58c718e46dc329e03da9e9ea6bdf89dde0a5ded
Copying blob sha256:2d3c2cad3b405661f1e22f3757ae6ecd2429faf9a3784d08695829b14d7c77c7
Copying blob sha256:c98c8473fc6297e2892ede8a3c8bf51583557c36d92c4ff535b4fbb3cffdac20
Copying blob sha256:6ca380c163afeddedffd5efda4f6305f03064055a94ade942ce659ac54a8ec6b
Copying blob sha256:fbbd55ef6877a63b9787d44b4de8ecfea3a6d63aaab5e064939a11b4482e017a
Copying blob sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3
Copying blob sha256:ecd031a502287a2cf6a743949b89e749fbda6b734eac80f9194d33920616dc92
Copying blob sha256:da694ee10f7eb862c67ad2720a9ead1365791850227935b7d776021a79778200
Copying blob sha256:3494ec024966474897dc3c59693ffabc1523bb31655548a47fec0a281deba767
Copying blob sha256:3d7e444fcf7fce125a572b02a29ab2bbd523f1ddcbfab5c9ea31b01497f5af2e
Copying blob sha256:b16580d55d6e1ac68f5e2084c4ac3f67f1035789b35088ad97895b6f08f3eede
Copying config sha256:24e85647f55a1205778e4b5ea03e62e6bc1cd7e94ce51d4ef7afdb8d19ea6dea
Writing manifest to image destination
STEP 2/8: COPY packages.txt .
time="2024-12-05T23:39:27+05:30" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> 61877e314d98
STEP 3/8: USER root
time="2024-12-05T23:39:27+05:30" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> e2e6f067f5d6
STEP 4/8: RUN /usr/local/bin/install-system-packages
time="2024-12-05T23:39:27+05:30" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> 076c67873082
STEP 5/8: COPY requirements.txt .
time="2024-12-05T23:39:27+05:30" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> a3521f39224c
STEP 6/8: RUN /usr/local/bin/install-python-dependencies
Looking in indexes: https://pip.astronomer.io/v2/
< gathering and Installing python dependencies >
Installing collected packages: shellingham, python-dotenv, duckdb, typer, astro-run-dag
Successfully installed astro-run-dag-0.2.7 duckdb-0.10.3 python-dotenv-1.0.1 shellingham-1.5.4 typer-0.15.1
time="2024-12-05T23:39:31+05:30" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> e81a565a1e30
STEP 7/8: USER astro
time="2024-12-05T23:39:31+05:30" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> 2cb9f19bd0cb
STEP 8/8: COPY --chown=astro:0 . .
COMMIT etl_698e5c/airflow:latest
time="2024-12-05T23:39:31+05:30" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> e2b5262ddf52
Successfully tagged localhost/etl_698e5c/airflow:latest
e2b5262ddf527345d73fb6004c82d1664c1de73b7873d977d0f4235980406ceb
Env file ".env" not found. Skipping...
[+] Running 1/15
 ⠿ postgres Pulled                                                                                                                                                   7.5s
   ⠋ e4b026cbbe96 Download complete                                                                                                                                  4.0s
   ⠇ fcad0c936ea5 Download complete                                                                                                                                  3.8s
   ⠼ d7ae7da7379c Download complete                                                                                                                                  3.4s
   ⠼ 74e46d49d057 Download complete                                                                                                                                  3.4s
   ⠸ 26859bddbb05 Download complete                                                                                                                                  3.3s
   ⠸ 82b3c6f95338 Download complete                                                                                                                                  3.3s
   ⠋ 10d1b949111f Download complete                                                                                                                                  3.0s
   ⠋ c84155d87589 Download complete                                                                                                                                  3.0s
   ⠏ 82414ff401d5 Download complete                                                                                                                                  2.9s
   ⠙ 79a2a10fcdff Download complete                                                                                                                                  2.1s
   ⠙ 7818d8e22ce6 Download complete                                                                                                                                  2.1s
   ⠋ 7dcc24de3f20 Download complete                                                                                                                                  2.0s
   ⠧ f0c0ca6a3d23 Download complete                                                                                                                                  1.7s
   ⠧ 6fa5e08fd59f Download complete                                                                                                                                  1.7s
   ⠋ 9691f14013a7 Download complete                                                                                                                                  0.0s
[+] Running 8/8
 ⠿ Network etl_698e5c_airflow         Created                                                                                                                        0.0s
 ⠿ Network etl_698e5c_default         Created                                                                                                                        0.0s
 ⠿ Volume "etl_698e5c_postgres_data"  Created                                                                                                                        0.0s
 ⠿ Volume "etl_698e5c_airflow_logs"   Created                                                                                                                        0.0s
 ⠿ Container etl_698e5c-postgres-1    Started                                                                                                                        0.3s
 ⠿ Container etl_698e5c-scheduler-1   Started                                                                                                                        0.8s
 ⠿ Container etl_698e5c-triggerer-1   Started                                                                                                                        0.8s
 ⠿ Container etl_698e5c-webserver-1   Started                                                                                                                        1.3s


Airflow is starting up!

Project is running! All components are now available.

Airflow Webserver: http://localhost:8080
Postgres Database: localhost:5432/postgres
The default Airflow UI credentials are: admin:admin
The default Postgres DB credentials are: postgres:postgres

astro dev ps                             
Name                    State           Ports
etl_698e5c-postgres-1   running         5432
etl_698e5c-triggerer-1  running
etl_698e5c-scheduler-1  running
etl_698e5c-webserver-1  running         8080

astro dev run dags list                               
Running: airflow dags list
[2024-12-05T18:15:20.172+0000] {plugin.py:32} WARNING - Astro managed secrets backend is disabled
dag_id               | fileloc                                         | owners | is_paused
=====================+=================================================+========+==========
example_etl_galaxies | /usr/local/airflow/dags/example_etl_galaxies.py | Astro  | False

Windows:

PS C:\Users\prittastro> winget install --manifest C:\Users\prittastro\astrocli
Found Astro CLI [Astronomer.Astro] Version 1.30.0
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
This package requires the following dependencies:
  - Packages
      Redhat.Podman
(1/1) Found Podman [RedHat.Podman] Version 5.3.1
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Downloading https://github.com/containers/podman/releases/download/v5.3.1/podman-5.3.1-setup.exe
  ██████████████████████████████  30.5 MB / 30.5 MB
Successfully verified installer hash
Starting package install...
Successfully installed

Downloading https://github.com/astronomer/astro-cli/releases/download/v1.30.0/astro_1.30.0_windows_amd64.exe
  ██████████████████████████████  30.7 MB / 30.7 MB
Successfully verified installer hash
Starting package install...
Command line alias added: "astro"
Path environment variable modified; restart your shell to use the new value.
Successfully installed

PS C:\Users\prittastro\test> C:\Users\prittastro\astroexe\astrowin.exe dev init -t=etl
Initializing Astro project
Pulling Airflow development files from Astro Runtime 12.5.0
Initialized empty Astro project in C:\Users\prittastro\test

PS C:\Users\prittastro\test> C:\Users\prittastro\astroexe\astrowin.exe dev start
⠹ Astro uses container technology to run your Airflow project. Please wait while we get things started…

PS C:\Users\prittastro\test> C:\Users\prittastro\astroexe\astrowin.exe dev start
[build --pull -t etl_add225/airflow:latest -f Dockerfile .]
STEP 1/1: FROM quay.io/astronomer/astro-runtime:12.5.0
Trying to pull quay.io/astronomer/astro-runtime:12.5.0...
Getting image source signatures
Copying blob sha256:845ed6b9a8eb7dc4dedadec9697465a4aa25b32e8c1c4be222449e23416cb9e7
Copying blob sha256:b7f1a5b763f46a0d128f7cd3f9f905ef260f8f28e566b3c395c82a614c898b6e
Copying blob sha256:5ecba4535d069f85ee470516ae6c11eaf7aba9439ef20be97df79b7603a79dde
Copying blob sha256:8d5e3089e6667d0cb1f83885b6dd5d9202de765bb653ba8cd09b2d4d55648759
Copying blob sha256:33cb52c61a55413a3e23f9a1fecdd09f1ce4e6f987bd73f994dbe063a2d693d4
Copying blob sha256:c7dffbbae38f7e0dc83c97fc2ab1024840d7ac169dc59f54f358ae73e801de07
Copying blob sha256:b743cee17d6d64bc211f7a190800f8908dba69972a9312603af28ef53d60a177
Copying blob sha256:94b7f478381ad53c1ecdf14184c0fcdb991b16cf7e5ca0f69da55f81afdba268
Copying blob sha256:dd6f839df82db2d3d51b77a8a0dbd870fd15e18781167a1624dbf4a033765829
Copying blob sha256:1601db97fd348a105a6fc3b8151997a628d908c1e8c19a68d56d05f8c83f4ad6
Copying blob sha256:15d587065c5268c8f3692737dc8923e4be770853928d015a8ad5c66d1eecb76f
Copying blob sha256:f49dea80626c7053b87ef8e52e91afc8fb9854f034be53151da29d9135d842ea
Copying blob sha256:c5d2b9689a78d39bbf79cdb568e23e16085cbb5cae9464091fff0891ce680504
Copying blob sha256:969f1c389d26a27d69e0edbc714588f8aa871e5a7acd4977e5f601a54e8915ec
Copying blob sha256:1f7358cbed987de06fc685a96473a69147b0ed50363b7f503940c5fc2a2fc4c6
Copying blob sha256:a4c12ec40bd9e6b3771a9e16335855f78a644778bee9ff6faca02e82d8fdbc67
Copying blob sha256:969f1c389d26a27d69e0edbc714588f8aa871e5a7acd4977e5f601a54e8915ec
Copying blob sha256:48d16b45b6b16713516a58ad6eb32fdb750ded8c79e6c079b49f2164d9e803aa
Copying blob sha256:ac11ce3523eda408d6a2127067772e7383a94dde3506b2d50f702125ba3e4e55
Copying blob sha256:622c4130dbe56f3c767d71573aafe48dabafa4f74d10f03bb084be12668e8c24
Copying blob sha256:db34e084d3e4d8250c6991ae007af3a1e5dc1afb74b4c0d31f20b2b5e08f1be1
Copying blob sha256:d897d616e060e775f2834b6a1bcc3ac25f8e0438c04f79663ce1dcc9831443b8
Copying blob sha256:4e89d9e1fd21499373cc2c86d075832a82df0014e9afc332845ab7ba890e0e0d
Copying blob sha256:90583ab9f8cfac6cf7f7831c8e6e1a9203f4241661dcd883ea963f62293621c4
Copying blob sha256:7496bdf7d6fed1e9ecd35c88e0eba4730675e61bd262552d57052ce508afb153
Copying blob sha256:6599e23523bfe3fd95092988523a452f3cdb3bcecf2d8a8a409999eb7d6f829e
Copying blob sha256:56821375c475b1add6ebcfa7ef60630e42e96e239cc1fb92ead37afc1576bbbf
Copying blob sha256:343640c29acf5a4c7d26bd96ac05179e6688df572a9a7198d30eeaecd1f6cd88
Copying blob sha256:91bd247e53f06e85ac86534724d4fc9fb03654c8dfd73a57e4f59c4441c8bd8f
Copying blob sha256:0f6cd58cd43aad46de7b0c45caf93ad8c6736e7ea2be5760dadcf01585439777
Copying blob sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3
Copying blob sha256:71f2864d29c043adce08acaef19d29527722c07a0b4c837bed98597fa3fdf52e
Copying blob sha256:5a2d24094312246b20ae2b4d45062f98ee7db4b1f1ded7435fbdc457b1acb4bc
Copying blob sha256:93b4547b6836c0033557f6e56e40c527ece06f1b14a42c5840bcd082475bbc10
Copying blob sha256:d7ad2313dfce2086a398350e06dcb7a09cc9a626b2021c55cf39a6775188188c
Copying blob sha256:f2b28d755446cfee050dab02119b7912dd666ce017fc09d624ae1140b4d3d0a3
Copying config sha256:8d7ad292b8868e9ca828eb9e762315ec4196d03ec12e020bcb49cc8cec413b79
Writing manifest to image destination
STEP 2/8: COPY packages.txt .
time="2024-12-05T18:33:49Z" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> ad765363975d
STEP 3/8: USER root
time="2024-12-05T18:33:49Z" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> 048e04ce9608
STEP 4/8: RUN /usr/local/bin/install-system-packages
time="2024-12-05T18:33:50Z" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> a9c85e3624a6
STEP 5/8: COPY requirements.txt .
time="2024-12-05T18:33:50Z" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> 3a1c724a53ca
STEP 6/8: RUN /usr/local/bin/install-python-dependencies
Looking in indexes: https://pip.astronomer.io/v2/

< Gathering and Installing python dependencies >

Successfully installed astro-run-dag-0.2.7 duckdb-0.10.3 python-dotenv-1.0.1 shellingham-1.5.4 typer-0.15.1
time="2024-12-05T18:33:55Z" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> 5d982daae84f
STEP 7/8: USER astro
time="2024-12-05T18:33:56Z" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> 9a1f3fabd328
STEP 8/8: COPY --chown=astro:0 . .
COMMIT etl_add225/airflow:latest
time="2024-12-05T18:33:56Z" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> 1e72861287bb
Successfully tagged localhost/etl_add225/airflow:latest
1e72861287bb2524cad88370b33a136ed8c60f6e9b198995b29bf68c015bd657
Env file ".env" not found. Skipping...
[+] Running 1/16
 - postgres Pulled                                                                                                                                                                                          5.6s
   - d3ee8e88c67c Download complete                                                                                                                                                                         4.9s
   - 69692152171a Download complete                                                                                                                                                                         4.9s
   - 435dd99ceb68 Download complete                                                                                                                                                                         4.8s
   - f65921886500 Download complete                                                                                                                                                                         4.7s
   - 84b08674f942 Download complete                                                                                                                                                                         4.7s
   - b9c1a94e4ca8 Download complete                                                                                                                                                                         4.7s
   - 7d358e850d3e Download complete                                                                                                                                                                         4.5s
   - a0daffc2a7a1 Download complete                                                                                                                                                                         4.4s
   - 53a7ff438b59 Download complete                                                                                                                                                                         4.3s
   - 44248978f8ed Download complete                                                                                                                                                                         4.2s
   - c74c1c4f5ca3 Download complete                                                                                                                                                                         4.1s
   - 83addcc511dd Download complete                                                                                                                                                                         4.0s
   - a31b993d5cc6 Download complete                                                                                                                                                                         3.9s
   - 478b899f4ef8 Download complete                                                                                                                                                                         3.9s
   - 145bb83bd157 Download complete                                                                                                                                                                         0.1s
[+] Running 8/8
 - Network etl_add225_default         Created                                                                                                                                                               0.0s
 - Network etl_add225_airflow         Created                                                                                                                                                               0.0s
 - Volume "etl_add225_airflow_logs"   Created                                                                                                                                                               0.0s
 - Volume "etl_add225_postgres_data"  Created                                                                                                                                                               0.0s
 - Container etl_add225-postgres-1    Started                                                                                                                                                               0.7s
 - Container etl_add225-triggerer-1   Started                                                                                                                                                               1.7s
 - Container etl_add225-scheduler-1   Started                                                                                                                                                               1.9s
 - Container etl_add225-webserver-1   Started                                                                                                                                                               2.4s


Airflow is starting up!

Project is running! All components are now available.

Airflow Webserver: http://localhost:8080
Postgres Database: localhost:5432/postgres
The default Airflow UI credentials are: admin:admin
The default Postgres DB credentials are: postgres:postgres

PS C:\Users\prittastro\test> C:\Users\prittastro\astroexe\astrowin.exe dev ps
Name                    State           Ports
etl_add225-postgres-1   running         5432
etl_add225-scheduler-1  running
etl_add225-triggerer-1  running
etl_add225-webserver-1  running         8080

PS C:\Users\prittastro\test> C:\Users\prittastro\astroexe\astrowin.exe dev run dags list
Running: airflow dags list
[2024-12-05T18:40:14.019+0000] {plugin.py:32} WARNING - Astro managed secrets backend is disabled
dag_id               | fileloc                                         | owners | is_paused
=====================+=================================================+========+==========
example_etl_galaxies | /usr/local/airflow/dags/example_etl_galaxies.py | Astro  | False

📸 Screenshots

Add screenshots to illustrate the validity of these changes.

📋 Checklist

Rebased from the main (or release if patching) branch (before testing)
Ran make test before taking out of draft
Ran make lint before taking out of draft
Added/updated applicable tests
Tested against Astro-API (if necessary).
Tested against Houston-API and Astronomer (if necessary).
Communicated to/tagged owners of respective clients potentially impacted by these changes.
Updated any related documentation

cmd/airflow_test.go

schnie · 2024-12-06T00:15:36Z

airflow/runtimes/container_runtime.go

+	Configure() error
+	ConfigureOrKill() error
+	Kill() error


Adding 3 new lifecycle methods here. The airflow hooks call these methods depending on the situation.

airflow/runtimes/podman_runtime.go

schnie · 2024-12-06T00:37:02Z

airflow/runtimes/utils.go

-// is running on Apple macOS.
-func isMac() bool {
-	return runtime.GOOS == "darwin"
-}


Refactored to os_checker

for more information, see https://pre-commit.ci

schnie · 2024-12-06T03:05:36Z

cmd/airflow_hooks.go

+	// Check if the OS is Windows and create the plugins project directory if it doesn't exist.
+	// In Windows, the compose project will fail if the plugins directory doesn't exist, due
+	// to the volume mounts we specify.
+	osChecker := new(runtimes.DefaultOSChecker)
+	if osChecker.IsWindows() {
+		pluginsDir := filepath.Join(config.WorkingPath, "plugins")
+		if err := os.MkdirAll(pluginsDir, os.ModePerm); err != nil && !os.IsExist(err) {
+			return fmt.Errorf(failedToCreatePluginsDir, err)
+		}
+	}


@pritt20 I attempted to simplify the nesting here a little. If you have your VM running would you mind testing this again on windows? And is the comment I added here accurate?

Thanks @schnie !

I tested this scenario on windows machine and it is working as expected.

neel-astro

The code was well commented making the review much easier, thanks for that ❤️.
Left some minor comments, but overall lgtm.

airflow/runtimes/docker_engine.go

airflow/runtimes/docker_runtime.go

airflow/runtimes/file_checker.go

airflow/runtimes/podman_engine.go

airflow/runtimes/podman_runtime.go

Co-authored-by: Neel Dalsania <[email protected]>

neel-astro

LGTM

Co-authored-by: Pritesh Arora <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Neel Dalsania <[email protected]>

schnie force-pushed the podman-container-runtime branch 2 times, most recently from 120d170 to 8fda48e Compare November 26, 2024 17:58

Adds podman container runtime support

24eb943

schnie force-pushed the podman-container-runtime branch from 8fda48e to 24eb943 Compare November 26, 2024 19:23

Pritesh Arora added 3 commits November 29, 2024 19:49

windows specific changes and tests fix

b3db819

fix linting

9c5acbb

fix plugins directory issue with windows

5dc16a6

schnie commented Dec 3, 2024

View reviewed changes

cmd/airflow_test.go Show resolved Hide resolved

Refactoring for unit tests

2cec150

schnie force-pushed the podman-container-runtime branch from b2c1aa7 to 2cec150 Compare December 4, 2024 01:56

Pritesh Arora and others added 3 commits December 4, 2024 21:17

add config for podman resources

7787031

Runtime and Engine Refactor for Testing

d6043d0

fix linting

6c4dcfd

schnie force-pushed the podman-container-runtime branch 2 times, most recently from 9d8943c to 8c353ec Compare December 6, 2024 00:14

schnie commented Dec 6, 2024

View reviewed changes

schnie force-pushed the podman-container-runtime branch from bea8e70 to c86a6da Compare December 6, 2024 00:33

schnie commented Dec 6, 2024

View reviewed changes

airflow/runtimes/podman_runtime.go Outdated Show resolved Hide resolved

schnie commented Dec 6, 2024

View reviewed changes

schnie force-pushed the podman-container-runtime branch 3 times, most recently from 20cc12b to 8e95465 Compare December 6, 2024 02:44

Major test overhaul for runtimes and podman

0c585f5

schnie force-pushed the podman-container-runtime branch from 349b073 to 0c585f5 Compare December 6, 2024 02:45

pre-commit-ci bot and others added 2 commits December 6, 2024 02:45

[pre-commit.ci] auto fixes from pre-commit.com hooks

c4824e9

for more information, see https://pre-commit.ci

Merge branch 'main' into podman-container-runtime

4123a38

schnie marked this pull request as ready for review December 6, 2024 02:49

schnie requested review from jeremybeard, kushalmalani and neel-astro as code owners December 6, 2024 02:49

schnie requested a review from lzdanski as a code owner December 6, 2024 02:49

Clean up hooks

f0a520e

schnie commented Dec 6, 2024

View reviewed changes

Pritesh Arora and others added 3 commits December 6, 2024 23:55

add docker host env logic for windows

98cd8c0

Fix tests from windows fix

e8255f6

Cleanup

2dc433b

neel-astro reviewed Dec 9, 2024

View reviewed changes

schnie and others added 4 commits December 9, 2024 09:26

Update airflow/runtimes/podman_runtime.go

af6ab5d

Co-authored-by: Neel Dalsania <[email protected]>

Merge branch 'main' into podman-container-runtime

0388ab4

Bump default memory

ae3fb8d

Rename Default implementations to lowercase

92b929c

schnie force-pushed the podman-container-runtime branch from f0b5a09 to 92b929c Compare December 10, 2024 19:14

neel-astro approved these changes Dec 11, 2024

View reviewed changes

Merge branch 'main' into podman-container-runtime

0c8795a

schnie merged commit 19672e5 into main Dec 11, 2024

schnie deleted the podman-container-runtime branch December 11, 2024 15:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Adds Podman Container Runtime#1750

Adds Podman Container Runtime#1750
schnie merged 20 commits intomainfrom
podman-container-runtime

schnie commented Nov 26, 2024 •

edited

Loading

Uh oh!

Uh oh!

schnie Dec 6, 2024

Uh oh!

Uh oh!

schnie Dec 6, 2024

Uh oh!

schnie Dec 6, 2024 •

edited

Loading

Uh oh!

pritt20 Dec 6, 2024

Uh oh!

neel-astro left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

neel-astro left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

schnie commented Nov 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

🎟 Issue(s)

🧪 Functional Testing

Mac:

Windows:

📸 Screenshots

📋 Checklist

Uh oh!

Uh oh!

schnie Dec 6, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

schnie Dec 6, 2024

Choose a reason for hiding this comment

Uh oh!

schnie Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pritt20 Dec 6, 2024

Choose a reason for hiding this comment

Uh oh!

neel-astro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

neel-astro left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

schnie commented Nov 26, 2024 •

edited

Loading

schnie Dec 6, 2024 •

edited

Loading