Skip to content

Difference of extras Airflow 2.0 vs. Airflow 1.10 #12744

Description

@potiuk

Description

When airflow 2.0 is installed from PyPI, providers are not installed by default. In order to install them, you should add an appropriate extra. While this behavior is identical in Airflow 1.10 for those "providers" that required additional packages, there were a few "providers" that did not require any extras to function (example http, ftp) - we have "http", "ftp" extras for them now, but maybe some of those are popular enough to be included by default?.

We have to make a decision now:

  • should all of them (or some of them) be included by default when you install Airflow?
  • if we decide to exclude only some (or none), we should add them in UPGRADING_to_2_0 and in UPDATING documentation.

Use case / motivation

We want people to get a familiar experience when installing airflow. Why we provide familiar mechanism (with extras) and people will expect a slightly different configurations, installation and we can describe the differences, maybe some of those providers are so popular that we should include them by default?

Related Issues

#12685 - where we discuss which of the extras should be included in the Production Image of 2.0.

Additional info

Here is the list of all "providers" that were present in 1.10 and had no additional dependencies - so basically they woudl work out-fhe-box in 1.10, but they need appropriate "extra" in 2.0.

  • "apache.pig": [],
  • "apache.sqoop": [],
  • "dingding": [],
  • "discord": [],
  • "ftp": [],
  • "http": [],
  • "imap": [],
  • "openfaas": [],
  • "opsgenie": [],
  • "sqlite": [],

Also here I appeal to the wisdom of crowd: @ashb, @dimberman @kaxil, @turbaszek, @mik-laj. @XD-DENG, @feluelle, @eladkal, @ryw, @vikramkoka, @KevinYang21 - let me know WDYT before I bring it to devlist?

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions