Upgrade azure blob to v12 #12188

ephraimbuddy · 2020-11-09T01:52:53Z

This PR attempts to upgrade azure storage blob to version 12
Closes: #11968

Manually tested with this dag:

from airflow.utils.dates import days_ago
from airflow.models import DAG
from airflow.providers.microsoft.azure.transfers.file_to_wasb import FileToWasbOperator
from airflow.providers.microsoft.azure.operators.wasb_delete_blob import WasbDeleteBlobOperator


with DAG(
    "file_to_wasb",
    schedule_interval="@once",
    start_date=days_ago(2)
) as dag:
    move = FileToWasbOperator(
        task_id="upload_file",
        file_path="/files/dags/example_sql_sensor.py",
        container_name="mycontainer",
        blob_name='myblob'
    )
    delete = WasbDeleteBlobOperator(
        task_id="delete_file",
        container_name="mycontainer",
        blob_name="myblob"
    )
    move >> delete

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ephraimbuddy · 2020-11-21T13:58:10Z

@mik-laj @turbaszek @feluelle, can you please review, thanks

airflow/providers/microsoft/azure/hooks/wasb.py

docs/spelling_wordlist.txt

mik-laj · 2020-11-21T14:07:09Z

Manually tested with this dag:

Can you add an automatic test? It shouldn't be difficult, and it will allow us to test this integration more easily.

airflow/providers/microsoft/azure/hooks/wasb.py

ephraimbuddy · 2020-11-24T10:33:15Z

Manually tested with this dag:

Can you add an automatic test? It shouldn't be difficult, and it will allow us to test this integration more easily.

I'll add this test latest this weekend

potiuk · 2020-11-27T16:41:52Z

Hey @ephraimbuddy - This is the last blocker for the #12636 , but for now I am going to add some conditional imports and make the relevant tssts as xfail, but once this on is ready to merge, I will remove those!

ephraimbuddy · 2020-11-27T16:44:29Z

Hey @ephraimbuddy - This is the last blocker for the #12636 , but for now I am going to add some conditional imports and make the relevant tssts as xfail, but once this on is ready to merge, I will remove those!

Ok. I will work on it tonight

ephraimbuddy · 2020-11-27T23:01:58Z

Depends on #12673
When #12673 is merged, I will rebase and run the system tests

ephraimbuddy · 2020-11-28T08:37:44Z

HI @potiuk , I'm not sure why the azure-blob-storage is not getting installed, despite that I added it to setup.py

potiuk · 2020-11-28T09:47:05Z

HI @potiuk , I'm not sure why the azure-blob-storage is not getting installed, despite that I added it to setup.py

In this case you need "upgrade to latest dependencies" label and rebase it . I just added the label. please rebase to the the latest master :)

potiuk · 2020-11-28T09:47:20Z

This is a new feature added last week.

ephraimbuddy · 2020-11-28T09:59:24Z

This is a new feature added last week.

Oh, cool!

ashb · 2020-11-28T10:31:27Z

HI @potiuk , I'm not sure why the azure-blob-storage is not getting installed, despite that I added it to setup.py

In this case you need "upgrade to latest dependencies" label and rebase it . I just added the label. please rebase to the the latest master :)

Shouldn't a change to setup.py/setup.cfg be all that is needed @potiuk ?

Let me be clearer: why do we need this label? Setup.py changed. We should respect it automatically. Not doing so is a bug, not a feature.

Since #12188 was merged I don't think we need this steps. This step also caused the docker build step for 2.0.1rc2 to fail Co-authored-by: Jarek Potiuk <[email protected]> (cherry picked from commit 3ffd217)

Since apache#12188 was merged I don't think we need this steps. This step also caused the docker build step for 2.0.1rc2 to fail Co-authored-by: Jarek Potiuk <[email protected]> (cherry picked from commit 3ffd217)

Since apache/airflow#12188 was merged I don't think we need this steps. This step also caused the docker build step for 2.0.1rc2 to fail Co-authored-by: Jarek Potiuk <[email protected]> (cherry picked from commit 3ffd21745d25e6239254fe3f5688b34f5f6f77e8) GitOrigin-RevId: 1b29db9148df815eec046532efb6c46ead8ddbbe

Since apache/airflow#12188 was merged I don't think we need this steps. This step also caused the docker build step for 2.0.1rc2 to fail Co-authored-by: Jarek Potiuk <[email protected]> GitOrigin-RevId: 3ffd21745d25e6239254fe3f5688b34f5f6f77e8

boring-cyborg bot added the provider:microsoft-azure Azure-related issues label Nov 9, 2020

ephraimbuddy changed the title ~~Upgrade azure blob and fileshare to v12~~ Upgrade azure blob to v12 Nov 16, 2020

ephraimbuddy marked this pull request as ready for review November 16, 2020 03:30

mik-laj reviewed Nov 21, 2020

View reviewed changes