Repository navigation
Commit ce01a73
authored
Fix scheduler memory spike when a Dag run has a large conf (apache#73907)
* Add tests for the run conf loaded per task instance when scheduling
TaskInstance.dag_run is a joined eager load, so a task-instance query
that also selects dag_run.conf returns and decodes one copy of the conf
per row. A scheduling pass does that twice: fetch_task_instances reads
every task instance of the run, and _get_ready_tis re-reads the ones
still waiting on their upstreams. A large conf on a run with many
mapped task instances multiplies scheduler memory this way.
* Fix scheduler memory spike when a Dag run has a large conf
TaskInstance.dag_run is a joined eager load, so every row of a
task-instance query also carries the run's conf. The scheduler fetches
all task instances of each running Dag run on every loop, and re-reads
the ones still waiting on their upstreams, so a large conf was decoded
once per task instance: a 1 MB conf on a run with 500 mapped task
instances took the scheduler from 150 MB to over 3 GB.
Neither query needs the conf up front: the few callers that read it
can load it on access, and the re-check only compares states.1 parent d88ca31 commit ce01a73
2 files changed
Lines changed: 58 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| 66 | + | |
66 | 67 | | |
67 | 68 | | |
68 | 69 | | |
| |||
1064 | 1065 | | |
1065 | 1066 | | |
1066 | 1067 | | |
| 1068 | + | |
| 1069 | + | |
1067 | 1070 | | |
1068 | | - | |
| 1071 | + | |
1069 | 1072 | | |
1070 | 1073 | | |
1071 | 1074 | | |
| |||
1975 | 1978 | | |
1976 | 1979 | | |
1977 | 1980 | | |
1978 | | - | |
| 1981 | + | |
| 1982 | + | |
| 1983 | + | |
| 1984 | + | |
1979 | 1985 | | |
1980 | 1986 | | |
1981 | 1987 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
116 | 117 | | |
117 | 118 | | |
118 | 119 | | |
119 | | - | |
| 120 | + | |
120 | 121 | | |
121 | 122 | | |
122 | 123 | | |
| |||
134 | 135 | | |
135 | 136 | | |
136 | 137 | | |
| 138 | + | |
| 139 | + | |
137 | 140 | | |
138 | 141 | | |
139 | 142 | | |
| |||
1132 | 1135 | | |
1133 | 1136 | | |
1134 | 1137 | | |
| 1138 | + | |
| 1139 | + | |
| 1140 | + | |
| 1141 | + | |
| 1142 | + | |
| 1143 | + | |
| 1144 | + | |
| 1145 | + | |
| 1146 | + | |
| 1147 | + | |
| 1148 | + | |
| 1149 | + | |
| 1150 | + | |
| 1151 | + | |
| 1152 | + | |
| 1153 | + | |
| 1154 | + | |
| 1155 | + | |
| 1156 | + | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
| 1162 | + | |
| 1163 | + | |
| 1164 | + | |
| 1165 | + | |
| 1166 | + | |
| 1167 | + | |
| 1168 | + | |
| 1169 | + | |
| 1170 | + | |
| 1171 | + | |
| 1172 | + | |
| 1173 | + | |
| 1174 | + | |
| 1175 | + | |
| 1176 | + | |
| 1177 | + | |
| 1178 | + | |
| 1179 | + | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
1135 | 1184 | | |
1136 | 1185 | | |
1137 | 1186 | | |
| |||
0 commit comments