The test_celery_executor test fails transiently in CI #8908
Description
Apache Airflow version:
Master - 2.0.0dev
Environment:
GitHub CI jobs.
What happened:
The Celery Executor test is transiently failing. I think this started to appear recently (so likely it is one of the optimizations implemented recently)
You can see an example of such failure here but there are many more similar ones)
I think there were recently some optimizations done in this area and I believe it might be the root cause of this problem. Can you please help to solve that problem? I do not want to put that problem to Quarantine, as it looks like an awfully "real" problem.
/usr/local/lib/python3.7/site-packages/celery/result.py:333: in maybe_throw
self.throw(value, self._to_remote_traceback(tb))
/usr/local/lib/python3.7/site-packages/celery/result.py:326: in throw
self.on_ready.throw(*args, **kwargs)
/usr/local/lib/python3.7/site-packages/vine/promises.py:244: in throw
reraise(type(exc), exc, tb)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tp = <class 'Exception'>
value = Exception('<class \'sqlalchemy.exc.IntegrityError\'>((\'(psycopg2.errors.UniqueViolation) duplicate key value violates..."pg_type_typname_nsp_index"\\nDETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.\\n\',))')
tb = None
def reraise(tp, value, tb=None):
"""Reraise exception."""
if value.__traceback__ is not tb:
raise value.with_traceback(tb)
> raise value
E Exception: <class 'sqlalchemy.exc.IntegrityError'>(('(psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "pg_type_typname_nsp_index"\nDETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.\n',))
What you expected to happen:
The test succeeds consistently.
How to reproduce it:
Happens practically with every build on CI recently - randomly at one of several jobs that run the test. Some of the jobs succeed (with different backends), some of them fail. This happens often but it's not fully repeatable.
Additional info
Discussion in devlist: https://lists.apache.org/thread.html/r2b531ef5c4a31746843127fc313c939296f3957eceafe4c1753b0f6d%40%3Cdev.airflow.apache.org%3E