@kaxil - I missed this at the beginning of the call - this PR had "beta1" but I do not think it is actualy needed in beta1 (and this is a POC so I am going to re-implement it anyway in smaller pieces). I planned the work there for different betas to add those incrementally and split it into separate issues (all in https://github.com/apache/airflow/projects/5)

@potiuk I've been having a quick look at using https://pluggy.readthedocs.io/en/latest/ (the plugin system from pytest) for this -- I'll let you know my thoughts

@potiuk I've been having a quick look at using https://pluggy.readthedocs.io/en/latest/ (the plugin system from pytest) for this -- I'll let you know my thoughts

Interesting. I have not yet come back to it - busy with CI optimisations, but will come back to it this week, so happy to hear your thoughts. this was just POC to see that it is possible but if there are ready-to-use solutions, I am happy to use them instead

The main problem we we've run in to with the existing plugin mechanism (we ship some custom plugins in our Astronomer images) is that any conflicting deps means plugins are unloadable:

Traceback (most recent call last):
  File "/home/ash/code/airflow/airflow/airflow/plugins_manager.py", line 187, in load_entrypoint_plugins
    plugin_class = entry_point.load()
  File "/home/ash/.virtualenvs/airflow-with-plugins/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2442, in load
    self.require(*args, **kwargs)
  File "/home/ash/.virtualenvs/airflow-with-plugins/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2465, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/home/ash/.virtualenvs/airflow-with-plugins/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (lazy-object-proxy 1.2.2 (/home/ash/.virtualenvs/airflow-with-plugins/lib/python3.7/site-packages), Requirement.parse('lazy_object_proxy~=1.3'))

Given the odds of some version conflict in a non-trival airflow install is quite high, using our existing plugin mechanism for more doesn't seem like a great idea.

I haven't looked in detail yet at Pluggy, but somehow it doesn't suffer this same problem, even though it supports entrypoints.

Also at first though I quite like the idea of a richer plugin mechanism in Airflow.