Calendar view in Airflow 3.0? #50907
-
TLDR;Airflow 3.0 appears to have removed the calendar view from the UI, which was an extremely useful feature in previous versions of Airflow. Are there plans to reintroduce this feature? I'm very hyped for 3.0 (DAG versioning!?!), but without the calendar view it's a bit of a game breaker for us I'm afraid... How we use the calendar viewWith our existing Airflow deployment, we use the calendar view as a quick way of assessing data quality. For example, for DAGs which pull in data from an external source, we use the DAG run's logical date to determine which data to pull from source. Our DAG contains tasks which check and verify the quality of the data returned from the external source, and verify that the data was successfully loaded into our database, failing the DAG run if there are any issues. This way, the calendar view shows us if we have issues for specific data ranges, which we can then address. This view seems to be entirely absent from Airflow 3.0 (unless it's simply moved in the UI and I just haven't found it!). For us, this is a bit of a game-breaker, and I'm not sure we'd be able to upgrade to 3.0 without this view.
With 3.0, it generally seems quite hard to find historic DAG runs. I can sort DAG runs by start/end date, but if for example I have 1,000 runs, I don't see a straightforward way to find a DAG run at a specific date in the past?
Similarly, Airflow 2.x had a fully fleshed out Task Instances view which allowed fine grained searching for specific task instances, and the taking action on a group of tasks at once. For example, this was very useful for re-running a group of tasks between a specific date range.
|
Beta Was this translation helpful? Give feedback.
Replies: 9 comments 11 replies
-
|
Up |
Beta Was this translation helpful? Give feedback.
-
|
Up |
Beta Was this translation helpful? Give feedback.
-
Down. @Djoren How much time of yours and money have you spend during those 5 years to make the UI better? I have not seen a single PR or other contribution from you. It's super easy to complain about things, but reporting that things are "barel better" in a software that you get for absolutely 0 (of any currency) and where you presnet your own personal view - without any attempt to improve it beforre or have a single comment when the calendar view was removed in previuous PR (you can search for it before) is like somoene getting a great meak for free and complaining to the chef that it did niot deliver it on exactly the kind of plate you wanted (but you have not spend any time or effort to explain what kind of plate you wanted). I suggest you spend the 5 years on being part of the community and even if you do not have skills to contribute, following what's going on and commenting when changes are proposed is a good idea. Highly recommended. |
Beta Was this translation helpful? Give feedback.
-
|
And before you have any response to it - please make sure to show evidences of your earlier contributions you made and comments you made before when those changes were implemented. Without evidences - it's just a very mean (and not realy respecting the time and effort of people who work on airflow) way of expressing your frustrations without making any effort to help in the project where people are contributing their own time - weekends and nights often to release the absolutely free software for people like you. I hope you will consider that perspective next time when you make similar comment @Djoren |
Beta Was this translation helpful? Give feedback.
-
|
All fair criticism. You're right and I agree, my comment was not in good manner. I will remove my complaint. |
Beta Was this translation helpful? Give feedback.
-
|
UI plugins aren't supported in Airflow 3 yet, but the new API looks a lot more flexible. Seems like this kind of thing should be doable via plugins later on. |
Beta Was this translation helpful? Give feedback.
-
|
Agree, would be great to see a plugin ecosystem grow up around Airflow. Possibilities seem endless! |
Beta Was this translation helpful? Give feedback.
-
|
@potiuk Thanks for looking at this question. However the question still stands:
At the London Astronomer Roadshow for 3.0 last month I was speaking with some of the contributors (@ashb and @BasPH I believe) and I got the impression that the calendar view might have been either overlooked, or dropped from the initial 3.0 release in the interest of priorities (both very understandable reasons). I'm curious whether it's expected that calendar view (or similar functionality) will make its way into 3.0 in the coming months, or whether it's been dropped entirely? As I said in the original question, without this functionality, 3.0 is unfortunately a no-go for us. Whilst I'd be more than happy to contribute towards the effort, my TypeScript / React skillset is essentially none-existent. If there are other ways I could assist, I'd be very happy to do so! |
Beta Was this translation helpful? Give feedback.
-
|
@CEardley That sounds like something I said for sure 😁 If you grab @pierrejeambrun or @bbovenzi somehow and discuss it with them then a PR to add a UI-only(? Maybe it could be part of the public API too) endpoint to feed the data to the eventual UI view might be helpful if you are keen to get involved. |
Beta Was this translation helpful? Give feedback.
-
|
I will let Brent confirm this but I believe that the calendar view followed the same path as the Gantt view. Related issue for the Gantt view here #44672. TLDR: It would be nice to have these features but considering priorities both were excluded from initial 3.0 scope. If you are willing to get involved that would be a great contribution. As mentioned by Ash, a UI-only endpoint + new tab in the UI Dag details page could be a good start. |
Beta Was this translation helpful? Give feedback.
-
|
This new issue mentions both gantt and calendar view too: |
Beta Was this translation helpful? Give feedback.
-
Yes I hope someone can answer this question. While I can easily comment on someone just "whining", I am not able to comment on decisions on the UI :). |
Beta Was this translation helpful? Give feedback.
-
|
Hi @CEardley, we ran out of time to re-implement the calendar view for 3.0. I would like to bring it back and also add more search+filters for the dag runs and task instance table views. I wrote about some options to add such a view onto the Dag or Task Overview tab. We can use the existing time range selector on that view and depending on the range one has selected we can change the calendar view so that it can be useful at 24 hours, one week, one month or multiple years: #44259 (comment) Would love to work with you on it if you have some spare time! |
Beta Was this translation helpful? Give feedback.
-
|
There is also an open PR to add multi-selection to the Task Instances table view: #51564 |
Beta Was this translation helpful? Give feedback.
-
|
Hello. My PR with calendar view endpoint has been merged into the main. Here in the PR you can find request/response example. Basically response/request I got from the airflow 2 and repeated the same, but added a few additional features like datetime filtration and granularity choosing. |
Beta Was this translation helpful? Give feedback.
-
|
If you have any question or want to change something feel free to DM in the airflow slack |
Beta Was this translation helpful? Give feedback.
-
|
I guess we are 'only' missing the front-end implementation part. |
Beta Was this translation helpful? Give feedback.
-
|
Love the hourly granularity! |
Beta Was this translation helpful? Give feedback.
-
|
(I really need to change my GitHub notification settings so it's not months between me checking this thread!) Thanks for all the useful info all. I'm going to mark this as resolved as sounds like it's sufficiently on the radar. Seems that @SavingFrame has already added the UI-API for this, which I suspect covers about as much as I could have hoped of contributing! If there's anything else (BE related probably) which I can help with, please let me know. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @CEardley. I created this tracking issue for the frontend implementation. Tagged it for Closing this discussion, feel free to reopen if needed. :) |
Beta Was this translation helpful? Give feedback.
I will let Brent confirm this but I believe that the calendar view followed the same path as the Gantt view. Related issue for the Gantt view here #44672.
TLDR: It would be nice to have these features but considering priorities both were excluded from initial 3.0 scope.
If you are willing to get involved that would be a great contribution. As mentioned by Ash, a UI-only endpoint + new tab in the UI Dag details page could be a good start.