Skip to content

Separate out and clarify policies for providers#30657

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:separate-out-and-clarify-approach-for-providers
Apr 18, 2023
Merged

Separate out and clarify policies for providers#30657
potiuk merged 1 commit into
apache:mainfrom
potiuk:separate-out-and-clarify-approach-for-providers

Conversation

@potiuk

@potiuk potiuk commented Apr 15, 2023

Copy link
Copy Markdown
Member

This change separates out the policies we have for providers to a separate PROVIERS.rst file. It also documents clearly the process and policy we have for accepting new community-managed providers, explaining the conditions that have to be fulfilled and stating a very strong preference of keeping providers maintained by the 3rd-party providers when there are 3rd-party teams that manage the providers.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Comment thread PROVIDERS.rst Outdated
Comment on lines 78 to 85

@eladkal eladkal Apr 15, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have a loop hole here.
If I take for example the google case. The current state allows to add any new google integration without passing trough this procedure because everything is packed under 1 provider.

I think we should also specify handling case of adding new SDK under existed provider that extend the provider to a new service that wasn't part of it to begin with. If we take for example when we first added Tableau to the existed provider of Salesforce following the current policy states that there is no need for a mailing list thread which seems wrong?

@potiuk potiuk Apr 15, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure we have to go that deep. I am more concerned about requests to add a new provider/integration thn a new "dependency" to an existing provider. We already get "historically" some providers that we manage, and that's sometihing that is both - the strength of Airflow and weakeness (dependency). And in general, any committer might block merging a new dependency being added when we think it is a problem. We do not have specifically explain that as a process (I think). Or when we see that a new operator does not fit. And we can always start a discussion at the devlist.

The process for adding a new provider is slightly different, I think - this is mostly about new integration, potentially new team of people from a service interacting with us, and a huge risk that they want to "drop" maintenance on us and won't be contributing new things, hoping that people in the community will and they can just walk away.

I think when getting new PR in an existing provider we have, we have much better knowledge - will there be maintenance, will there be support in case of problem, what happens when there is a need to upgrade or when the service gets disabled. We are going through that with Google team and while a bit late, the make a huge effort to make their provider updated now. This is a good lesson for us.

So I think while we could describe it in more detail, I tihnk this is really more of an explanation for anyone "new" (like Huawei Cloud service people) players that need to understand what it means to add a new provider.

Does it make sense?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also this is more a "default" behaviour. At any point, anyone can call for VOTE, LAZY CONSENSUS on just about anything, this is more like "what to expect" than "strict rules".

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think when getting new PR in an existing provider we have, we have much better knowledge - will there be maintenance, will there be support in case of problem, what happens when there is a need to upgrade or when the service gets disabled. We are going through that with Google team and while a bit late, the make a huge effort to make their provider updated now. This is a good lesson for us.

I'm not referring to the case that we have now.
I'm referring for a case that adding a new extension for a service that is not part of the original provider scope. If tomorrow Google acquire a company technically it allows them to add a new integration for it without violating this policy. I don't mean to cases where under google.cloud we add more services and improve the support there i'm referring to cases of adding a whole new service branch like: leveldb, looker etc...

This change separates out the policies we have for providers to
a separate PROVIERS.rst file. It also documents clearly the process
and policy we have for accepting new community-managed providers,
explaining the conditions that have to be fulfilled and stating
a very strong preference of keeping providers maintained by the
3rd-party providers when there are 3rd-party teams that manage
the providers.
@potiuk
potiuk force-pushed the separate-out-and-clarify-approach-for-providers branch from efaa0bf to d901c31 Compare April 15, 2023 11:25

@eladkal eladkal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for drafting this one!
I think we need also to clean up
https://airflow.apache.org/docs/apache-airflow-providers/index.html

Can I contribute my own provider to Apache Airflow?

this policy covers some of the content that we have in Airflow docs so maybe we just need to refer to this doc

@potiuk

potiuk commented Apr 15, 2023

Copy link
Copy Markdown
Member Author

Thank you for drafting this one!
I think we need also to clean up

Good point . Will take a look

@potiuk
potiuk merged commit 372a088 into apache:main Apr 18, 2023
@potiuk
potiuk deleted the separate-out-and-clarify-approach-for-providers branch April 18, 2023 19:49
wookiist pushed a commit to wookiist/airflow that referenced this pull request Apr 19, 2023
This change separates out the policies we have for providers to
a separate PROVIERS.rst file. It also documents clearly the process
and policy we have for accepting new community-managed providers,
explaining the conditions that have to be fulfilled and stating
a very strong preference of keeping providers maintained by the
3rd-party providers when there are 3rd-party teams that manage
the providers.
potiuk added a commit to potiuk/airflow that referenced this pull request Apr 22, 2023
We've recently clarified and described our policies for accepting
providers to be maintained by the community (apache#30657) - this was
directed towards the Airflow developers and contributors. This PR
reviews user-facing part of the documentation for providers by
removing some obsolete/not very useful documentation and pointing
to the new policy where appropriate.
@potiuk

potiuk commented Apr 22, 2023

Copy link
Copy Markdown
Member Author

this policy covers some of the content that we have in Airflow docs so maybe we just need to refer to this doc

PR here #30816

potiuk added a commit that referenced this pull request Apr 22, 2023
We've recently clarified and described our policies for accepting
providers to be maintained by the community (#30657) - this was
directed towards the Airflow developers and contributors. This PR
reviews user-facing part of the documentation for providers by
removing some obsolete/not very useful documentation and pointing
to the new policy where appropriate.
@ephraimbuddy ephraimbuddy added the type:doc-only Changelog: Doc Only label May 8, 2023
@ephraimbuddy ephraimbuddy added this to the Airflow 2.6.1 milestone May 8, 2023
ephraimbuddy pushed a commit that referenced this pull request May 8, 2023
We've recently clarified and described our policies for accepting
providers to be maintained by the community (#30657) - this was
directed towards the Airflow developers and contributors. This PR
reviews user-facing part of the documentation for providers by
removing some obsolete/not very useful documentation and pointing
to the new policy where appropriate.

(cherry picked from commit 76ebc9b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:doc-only Changelog: Doc Only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants