Repository navigation
Commit 5ddfc95
committed
fix: skip scheduling instead of bulk-failing when serialized DAG is transiently missing
When the scheduler cannot find a DAG in the serialized_dag table while checking
task concurrency limits, it previously set all SCHEDULED task instances for that
DAG to FAILED via a bulk UPDATE. This caused intermittent mass task failures on
multi-scheduler setups (e.g. MWAA) where the serialized DAG may be transiently
absent during a DAG file parse cycle.
Instead, treat the missing serialized DAG as a transient condition: log a warning,
add the dag_id to starved_dags so subsequent tasks for the same DAG are also
skipped, and let the scheduler retry on the next iteration.
The existing test 'test_queued_task_instances_fails_with_missing_dag' has been
updated to reflect the new expected behavior (tasks remain SCHEDULED). A new
regression test 'test_missing_serialized_dag_does_not_bulk_fail_tasks' explicitly
verifies no bulk-failure occurs.
Closes apache#620501 parent 6a616c5 commit 5ddfc95
2 files changed
Lines changed: 59 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
759 | 759 | | |
760 | 760 | | |
761 | 761 | | |
762 | | - | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
763 | 765 | | |
764 | | - | |
765 | | - | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
766 | 769 | | |
767 | 770 | | |
768 | 771 | | |
769 | | - | |
770 | | - | |
771 | | - | |
772 | | - | |
773 | | - | |
774 | | - | |
| 772 | + | |
775 | 773 | | |
776 | 774 | | |
777 | 775 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1797 | 1797 | | |
1798 | 1798 | | |
1799 | 1799 | | |
1800 | | - | |
1801 | | - | |
| 1800 | + | |
| 1801 | + | |
| 1802 | + | |
| 1803 | + | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
1802 | 1808 | | |
1803 | 1809 | | |
1804 | 1810 | | |
| |||
1822 | 1828 | | |
1823 | 1829 | | |
1824 | 1830 | | |
| 1831 | + | |
1825 | 1832 | | |
| 1833 | + | |
1826 | 1834 | | |
1827 | 1835 | | |
1828 | | - | |
| 1836 | + | |
| 1837 | + | |
| 1838 | + | |
| 1839 | + | |
| 1840 | + | |
| 1841 | + | |
| 1842 | + | |
| 1843 | + | |
| 1844 | + | |
| 1845 | + | |
| 1846 | + | |
| 1847 | + | |
| 1848 | + | |
| 1849 | + | |
| 1850 | + | |
| 1851 | + | |
| 1852 | + | |
| 1853 | + | |
| 1854 | + | |
| 1855 | + | |
| 1856 | + | |
| 1857 | + | |
| 1858 | + | |
| 1859 | + | |
| 1860 | + | |
| 1861 | + | |
| 1862 | + | |
| 1863 | + | |
| 1864 | + | |
| 1865 | + | |
| 1866 | + | |
| 1867 | + | |
| 1868 | + | |
| 1869 | + | |
| 1870 | + | |
| 1871 | + | |
| 1872 | + | |
| 1873 | + | |
| 1874 | + | |
| 1875 | + | |
| 1876 | + | |
| 1877 | + | |
1829 | 1878 | | |
1830 | 1879 | | |
1831 | 1880 | | |
| |||
0 commit comments