Hey @kaxil @ash - as mentioned on slack - I need to get that one merged in order to solve the Docker building problem that is now causing Kubernetes failures. Simply - when installing airflow[EXTRAS] I need to install also all the providers. This has not been possible yet because we do not have those packages in PyPI. This one is needed to fix #11490

This PR will likely fail in a few cases because Airflow will not find the right packages in PyPI to install:

• when installing Docker (Make sure Docker works with installing core + providers. #11490)
• when installing Airflow locally from sources (Make sure that local installation works with installing airflow + all providers. #11489) with editable (likely not) or just regular local install with 'pip install .'

Now the question is how we approach it:

1. We can publish both airflow and provider's packages in PyPI. Though the result will be that the providers will be installed in the version from PyPI.

2. We can try to implements some workaround to be able to install provider packages in Docker without having PyPI. This is possible, I can prepare packages before building docker, place them in 'docker-context-files' and install them from there using ``--find-links` option of PyPI which will install the packages from there. This is what I am currently leaning towards - in which case I will not need to have anything in PyPI (which for alphas might be a good idea).

3. We could also just publish the providers in PyPI and do not install Airflow.

Another point is to solve the local installation (#11489) but I think this one will have to be installed differently because you for sure want to have local providers installed. This I am going to likely solve by a custom approach to not exclude packages in setup.py when installing locally (providing that I will be able to reliably detect the case that I am installing airflow from sources and not building the package which should be possible).

But I have to double check that one.

The Workflow run is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.

The Workflow run is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.

This one should work in principle now. I had hard time getting the version retrieval work, even if I manipulated the path, so if someone can make the check_extras_provider_packages work without the exception on reading version - I'd appreciate it , either I am to tired, or there is something fishy going on with the way how setup imports version.py which I do not understand.

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

Hey @kaxil @ash - as mentioned on slack - I need to get that one merged in order to solve the Docker building problem that is now causing Kubernetes failures. Simply - when installing airflow[EXTRAS] I need to install also all the providers. This has not been possible yet because we do not have those packages in PyPI. This one is needed to fix #11490

This PR will likely fail in a few cases because Airflow will not find the right packages in PyPI to install:

• when installing Docker (Make sure Docker works with installing core + providers. #11490)
• when installing Airflow locally from sources (Make sure that local installation works with installing airflow + all providers. #11489) with editable (likely not) or just regular local install with 'pip install .'

Now the question is how we approach it:

1. We can publish both airflow and provider's packages in PyPI. Though the result will be that the providers will be installed in the version from PyPI.
2. We can try to implements some workaround to be able to install provider packages in Docker without having PyPI. This is possible, I can prepare packages before building docker, place them in 'docker-context-files' and install them from there using ``--find-links` option of PyPI which will install the packages from there. This is what I am currently leaning towards - in which case I will not need to have anything in PyPI (which for alphas might be a good idea).
3. We could also just publish the providers in PyPI and do not install Airflow.

Another point is to solve the local installation (#11489) but I think this one will have to be installed differently because you for sure want to have local providers installed. This I am going to likely solve by a custom approach to not exclude packages in setup.py when installing locally (providing that I will be able to reliably detect the case that I am installing airflow from sources and not building the package which should be possible).

But I have to double check that one.

I like (2)

I like (2)

Yep. That's what I am doing :)

I think we can get away without any

Would love to hear how :).

I thought the same, but I am not so sure any more. I actually tested it and if you do pip install . now, the providers are not installed and if you are not in the "airflow" folder (, in pyhton 3.6+ is by default on python path) you won't be able to import providers.

Also when we add dependencies in extras, as you will see now - if you try to install airflow[google] - it will fail now because it won't find the apache-airflow-google-package. So I wonder how you would like to avoid those two problems.

We can of course do a "proper" sub-provider setup.py and keep separate setup.py per provider and split it and then have dependent package structure from airflow, but that would be a lot of maintenance, and especially if you want to manage common set of dependencies for the whole airflow, that woudl be quite complex. But If you would like to solve that this way - feel free :).

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

All is ready now for merging - I think I managed to get it working in all cases as expected.

Converted it to Draft as fhis one should be merged only after we release providers to PyPI.

We should merge this just before we release our PyPI package for airflow beta 1

Can you please rebase your PR on latest Master since we have applied Black and PyUpgrade on Master.

It will help if your squash your commits into single commit first so that there are less conflicts.

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

Looks that this one is good to go as well (there are some random cancels happening but I think they are because of GA problems not our own cancelling.)