Repository navigation
Commit 1193158
committed
fix: skip scheduling instead of bulk-failing when serialized DAG is transiently missing
When the scheduler cannot find a DAG in the serialized_dag table while checking
task concurrency limits, it previously set all SCHEDULED task instances for that
DAG to FAILED via a bulk UPDATE. This caused intermittent mass task failures on
multi-scheduler setups (e.g. MWAA) where the serialized DAG may be transiently
absent during a DAG file parse cycle.
Instead, treat the missing serialized DAG as a transient condition: log a warning,
add the dag_id to starved_dags so subsequent tasks for the same DAG are also
skipped, and let the scheduler retry on the next iteration.
The existing test 'test_queued_task_instances_fails_with_missing_dag' has been
updated to reflect the new expected behavior (tasks remain SCHEDULED). A new
regression test 'test_missing_serialized_dag_does_not_bulk_fail_tasks' explicitly
verifies no bulk-failure occurs.
Closes apache#620501 parent dba4727 commit 1193158
2 files changed
Lines changed: 59 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
752 | 752 | | |
753 | 753 | | |
754 | 754 | | |
755 | | - | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
756 | 758 | | |
757 | | - | |
758 | | - | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
759 | 762 | | |
760 | 763 | | |
761 | 764 | | |
762 | | - | |
763 | | - | |
764 | | - | |
765 | | - | |
766 | | - | |
767 | | - | |
| 765 | + | |
768 | 766 | | |
769 | 767 | | |
770 | 768 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1800 | 1800 | | |
1801 | 1801 | | |
1802 | 1802 | | |
1803 | | - | |
1804 | | - | |
| 1803 | + | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
| 1808 | + | |
| 1809 | + | |
| 1810 | + | |
1805 | 1811 | | |
1806 | 1812 | | |
1807 | 1813 | | |
| |||
1825 | 1831 | | |
1826 | 1832 | | |
1827 | 1833 | | |
| 1834 | + | |
1828 | 1835 | | |
| 1836 | + | |
1829 | 1837 | | |
1830 | 1838 | | |
1831 | | - | |
| 1839 | + | |
| 1840 | + | |
| 1841 | + | |
| 1842 | + | |
| 1843 | + | |
| 1844 | + | |
| 1845 | + | |
| 1846 | + | |
| 1847 | + | |
| 1848 | + | |
| 1849 | + | |
| 1850 | + | |
| 1851 | + | |
| 1852 | + | |
| 1853 | + | |
| 1854 | + | |
| 1855 | + | |
| 1856 | + | |
| 1857 | + | |
| 1858 | + | |
| 1859 | + | |
| 1860 | + | |
| 1861 | + | |
| 1862 | + | |
| 1863 | + | |
| 1864 | + | |
| 1865 | + | |
| 1866 | + | |
| 1867 | + | |
| 1868 | + | |
| 1869 | + | |
| 1870 | + | |
| 1871 | + | |
| 1872 | + | |
| 1873 | + | |
| 1874 | + | |
| 1875 | + | |
| 1876 | + | |
| 1877 | + | |
| 1878 | + | |
| 1879 | + | |
| 1880 | + | |
1832 | 1881 | | |
1833 | 1882 | | |
1834 | 1883 | | |
| |||
0 commit comments