Skip to content

MSSQL backend broken when DAG serialization enabled #11493

@Kaash

Description

@Kaash

Version is 1.10.12

Back-end is MSSQL 12

The following SQL query is broken:

result = session.query(cls.rendered_fields).filter(
cls.dag_id == ti.dag_id,
cls.task_id == ti.task_id,
cls.execution_date == ti.execution_date
).one_or_none()

This query fails as the rendered_task_instance_fields table in the database stores execution date as TIMESTAMP but other tables have updated execution date for MSSQL to DATETIME2

Here's the stack trace:

[2020-10-12 14:23:49,025] {taskinstance.py:1150} ERROR - (pyodbc.DataError) ('22018', '[22018] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Operand type clash: datetime2 is incompatible with timestamp (206) (SQLExecDirectW)')
[SQL: SELECT rendered_task_instance_fields.dag_id AS rendered_task_instance_fields_dag_id, rendered_task_instance_fields.task_id AS rendered_task_instance_fields_task_id, rendered_task_instance_fields.execution_date AS rendered_task_instance_fields_execution_date, rendered_task_instance_fields.rendered_fields AS rendered_task_instance_fields_rendered_fields
FROM rendered_task_instance_fields
WHERE rendered_task_instance_fields.dag_id = ? AND rendered_task_instance_fields.task_id = ? AND rendered_task_instance_fields.execution_date = ?]
[parameters: ('xxx-20200901', 'DATA_FEED_AVAILABLE_CHECK', <Pendulum [2020-10-09T00:00:00+00:00]>)]
(Background on this error at: http://sqlalche.me/e/13/9h9h)
Traceback (most recent call last):
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 596, in do_execute
cursor.execute(statement, parameters)
pyodbc.DataError: ('22018', '[22018] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Operand type clash: datetime2 is incompatible with timestamp (206) (SQLExecDirectW)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 967, in _run_raw_task
RTIF.write(RTIF(ti=self, render_templates=False), session=session)
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper
return func(*args, **kwargs)
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/airflow/models/renderedtifields.py", line 94, in write
session.merge(self)
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2155, in merge
_resolve_conflict_map=_resolve_conflict_map,
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2228, in merge
merged = self.query(mapper.class
).get(key[1])
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 1018, in get
return self._get_impl(ident, loading.load_on_pk_identity)
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 1135, in _get_impl
return db_load_fn(self, primary_key_identity)
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 286, in load_on_pk_identity
return q.one()
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3463, in one
ret = self.one_or_none()
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3432, in one_or_none
ret = list(self)
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3508, in iter
return self._execute_and_instances(context)
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3533, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/local/scratch/khkaas/conda/envs/airflow-final/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind:bugThis is a clearly a bug

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions