Skip to content

[AIRFLOW-3515] Remove the run_duration option#4320

Merged
kaxil merged 1 commit intoapache:masterfrom
Fokko:fd-remove-run-duration
Jan 8, 2019
Merged

[AIRFLOW-3515] Remove the run_duration option#4320
kaxil merged 1 commit intoapache:masterfrom
Fokko:fd-remove-run-duration

Conversation

@Fokko
Copy link
Copy Markdown
Contributor

@Fokko Fokko commented Dec 14, 2018

We should not use the run_duration option anymore.

This used to be for restarting the scheduler from time to time,
but right now the scheduler is getting more stable and therefore
using this setting is considered bad and might cause an inconsistent
state,

Make sure you have checked all steps below.

Jira

Description

  • Here are some details about my PR, including screenshots of any UI changes:

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
    • All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

  • Passes flake8

@Fokko Fokko requested a review from bolkedebruin December 14, 2018 12:06
@Fokko Fokko force-pushed the fd-remove-run-duration branch from 7f054f8 to e4559e0 Compare December 14, 2018 12:23
@codecov-io
Copy link
Copy Markdown

codecov-io commented Dec 14, 2018 •
edited
Loading

Codecov Report

Merging #4320 into master will decrease coverage by 0.47%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4320      +/-   ##
==========================================
- Coverage   78.59%   78.12%   -0.48%     
==========================================
  Files         204      202       -2     
  Lines       16453    16479      +26     
==========================================
- Hits        12932    12874      -58     
- Misses       3521     3605      +84
Impacted Files Coverage Δ
airflow/bin/cli.py 64.48% <ø> (-0.55%) ⬇️
airflow/jobs.py 77.31% <100%> (-0.14%) ⬇️
airflow/operators/hive_to_mysql.py 0% <0%> (-100%) ⬇️
airflow/operators/hive_to_samba_operator.py 0% <0%> (-100%) ⬇️
airflow/operators/jdbc_operator.py 0% <0%> (-100%) ⬇️
airflow/hooks/samba_hook.py 0% <0%> (-38.89%) ⬇️
airflow/operators/bash_operator.py 91.37% <0%> (-1.61%) ⬇️
airflow/www_rbac/views.py 72.42% <0%> (-1.24%) ⬇️
airflow/utils/helpers.py 82.22% <0%> (-0.39%) ⬇️
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 11a36d1...2816e09. Read the comment docs.

kaxil
kaxil reviewed Dec 15, 2018
View reviewed changes
Comment thread airflow/bin/cli.py Outdated
Copy link
Copy Markdown
Member

@kaxil kaxil Dec 15, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove num_runs as well, then?

Copy link
Copy Markdown
Contributor Author

@Fokko Fokko Dec 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point Kaxil, I've created a ticket: https://jira.apache.org/jira/browse/AIRFLOW-3533

@kaxil kaxil mentioned this pull request Dec 15, 2018
Merged
6 tasks
[AIRFLOW-3515] Remove the run_duration option
4359aa5 
We should not use the `run_duration` option anymore.

This used to be for restarting the scheduler from time to time,
but right now the scheduler is getting more stable and therefore
using this setting is considered bad and might cause an inconsistent
state,
@Fokko Fokko force-pushed the fd-remove-run-duration branch from 2816e09 to 4359aa5 Compare January 7, 2019 21:57
@Fokko
Copy link
Copy Markdown
Contributor Author

Fokko commented Jan 7, 2019

I've rebased onto master and resolved the conflicts

@kaxil kaxil merged commit 327860f into apache:master Jan 8, 2019
@kaxil
Copy link
Copy Markdown
Member

kaxil commented Jan 8, 2019

Thanks @Fokko

wmorris75 pushed a commit to modmed-external/incubator-airflow that referenced this pull request Jul 29, 2019
@Fokko
[AIRFLOW-3515] Remove the run_duration option (apache#4320)
100585c
@shargal
Copy link
Copy Markdown

shargal commented Jun 28, 2020 •
edited
Loading

@Fokko Actually I was looking for this feature even with 1.10.10 as many times restarting the scheduler solved problems that came up. For example, tasks that stuck in queued state and not being invoked by KubernetesExecutor.

@Fokko Fokko deleted the fd-remove-run-duration branch July 3, 2020 07:48
@Fokko
Copy link
Copy Markdown
Contributor Author

Fokko commented Jul 3, 2020

@shargal Restarting the scheduler isn't really a proper solution, right? Feels a bit like; have you tried turning it off and on again. Would be cool if we could isolate the problem of why the tasks get stuck or add logic to check which tasks are stuck and retry them.

@Fokko
Copy link
Copy Markdown
Contributor Author

Fokko commented Aug 19, 2020

@shargal We've experienced similar issues at @godatadriven, this fixed itself by upgrading to the latest version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaxil kaxil kaxil approved these changes

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Fokko @codecov-io @kaxil @shargal